1. Introduction
-
The Webperf test is designed to provide a useful resource for
companies looking to outsource their web hosting as well as a
useful benchmark for the web hosting companies themselves. It
currently does this by assessing the real-world connectivity
of a mass-hosting ISP or Reseller in terms of site availability
and overall performance.
2. Glossary
-
SUT - Site Under Test. This refers to the web site of
the ISP or Reseller whose performance and availability is
being monitored.
Test host - This refers to the machine from which we do
the testing. It is provided on a voluntary basis by third
parties who in turn receive a banner advert on the web site.
In order to provide stability, controls and redundancy to the
testing, there will be multiple test hosts active at any one
time.
3. Network proximity
-
Dynamic network proximity testing prevents a SUT being
tested by a test host on the same LAN. All test hosts perform
this check before testing each SUT.
4. Availability
-
Availability is defined as the proportion of time per month for
which the web server is responsive to HTTP requests.
Each test host contacts all the SUTs and issues a simple HTTP
GET request. The availability results are calculated
as:
100 * successful attempts
% availability = -------------------------
total attempts
As with the performance testing, in order to get a reasonable
sample size, each machine is tested many times a day. We
currently test every SUT every 30 minutes from all test hosts
that are found not to be proximate to the SUT at the time of
test.
5. Performance
-
Network performance along a connection is largely based on two
factors:
- the latency of the channel, and
- the maximum data throughput of the channel.
Poor performance can be the result of many factors:
- limited bandwidth
- poor peering
- overloaded bandwidth, or
- overloaded hardware/software serving the data.
5.1 How is performance measured?
The HTTP performance levels of each SUT is measured with
ApacheBench. It is part of the standard Apache distribution, and
is based on a tool called Zeusbench which was orginally written
by Adam Twiss. This tool is used for the following reasons:
-
it is publicly available in source form, and
will compile on almost any Unix platform
-
it gives simple, reproducable, intelligible measurements of HTTP
throughput
As load and performance will vary at different times of day,
performance measurements are taken every hour by each test
host in order to get a balanced overall result. The
performance will vary according to the location of each test
host. This variance is merely an indication of connectivity
between a particular SUT and a particular test host. Multiple
test hosts are used to avoid any bias of results.
The testing will measure the data throughput (in bytes/second)
when repeatedly downloading a file within a specified size
range (4096 - 6144 bytes). This file must be a static object
(i.e. not dynamically generated by a CGI) and tends to be
an image.
6. Publishing Results
-
Results from the Webperf testing are automatically collated on
our web server and stored in a database. This information
from this database is then published in real-time on our web
site. It is possible to 'drill down' into the information to
extract more detailed figures for each SUT.
7. Future Enhancements
-
There are various ways in which we hope to extend the testing in the
future. If you have any further suggestion for improvement, please
contact Webperf with them.
-
7.1 CGI performance
One aspect of web site performance that could potential vary a lot is
the performance of CGI scripts. This will involve running
standard scripts on different machines. This would probably
have to be written in an interpreted language like Perl.
-
7.2 Approved test files
Currently, SUTs submit a static test object of suitable size.
However, this means that we are testing the different SUTs with
slightly different files, a fact which might have a small
affect on performance. To remove this inconsistency, we intend
to require all SUTs have a specified set of files to be tested.
-
7.3 Test ISP customer web sites
Many ISPs now host Resellers. If these Resellers also
submit themselves for testing, the ISP hosting site will be
tested multiple times, causing multiplicity of results for
overall performance. The ideal situatation would be for
Resellers to be tested separately with a new set of tests
that are tailored to reflect their performance as separate
from that of their ISP.
-
7.4 Provision of more detailed statistics
By making use of additional information provided by
ApacheBench, it is possible to break down the overall
performance results currently being published using a
separate test of performance co-ordinated with the existing
performance testing. This would take the form of repeated
downloads of a very small static object. When combined with
the existing testing, this would permit an average network
delay metric and an average web server performance metric to
be presented alongside the overall performance metric
currently being published.
