How fast is your broadband service?

That
depends on what you measure!
Data speed is a key customer service metric,
but understanding the measurement context is
important

Signalling the Future

There are many terms used when describing a metric for

the speed of a data connection. There are also many
different views of how to calculate a single given metric.
This can lead to apparent contradictions in
measurements of data speed between different
sources. Comparisons made are not always on a likefor-like basis. However, the results provided are still
valid when the context is understood.

Introduction
Broadband speed is a very important metric but there are significant differences in the tests used to
measure it and interpretation of results. There is also a certain ambiguity around terms such as throughput,
broadband speed, data bandwidth, link capacity, etc. Some commonly used metrics are as follows.

1. Capacity is a measure of the total traffic carrying capability of a link or path in a network. The
capacity is expressed as the amount of traffic the link can carry over a particular time interval. To

measure link or path capacity in a live network, all the traffic must be accounted for.
Therefore, for a live network this should be done on the network side via probes or network
counters with the link/path fully loaded.
2. Bulk transfer capacity1 (BTC) is a measure of the amount of data that can be transferred along a
network path with a congestion aware transport protocol like TCP. A critical distinction is that the
bulk transfer capacity is a measure of the data or payload throughput. Packet headers and data
packets that are retransmitted are not counted in the byte counts. Also most of the academic
literature and IETF RFCs define the bulk transfer capacity of a network as the achievable throughput
of a single transport connection. BTC is typically tested with a known file transfer and duration
measurement.
3. Available bandwidth is a measure of how much capacity is unused and available in a link or along a
path. Available bandwidth can be determined from the current traffic and the total capacity of the
link. Alternatively, the available bandwidth may be tested by loading the link/path with a test session
and measuring the throughput.
1 Defined by Internet Engineering Task Force RFC 3148: BTC = data sent / elapsed time

It is difficult to exactly correlate different throughput measurement systems with each of these metrics, but
most user tests carried out on a live network measure the available bandwidth. Even so, different test
methodologies will give different results.

Throughput Testing: File Transfer versus Throughput Sampling

Ooklas Speedtest.net is a widely used user throughput testing service. This service uses throughput
sampling and so some understanding is needed to appreciate what is measured and reported.
Speedtest.net uses multi-threading (see below) to try and use all available bandwidth where up to four
HTTP threads to saturate your connection and get an accurate measurement 2. The throughput is then
sampled at up to 30 times per second. Therefore, for a 10 second download there may be 300 throughput
samples. The fastest 10% and slowest 30% of the slices are then discarded and the throughput calculated as
the average of the remaining 60% of the samples. For upload tests on speedtest.net, the result is even more
selective as only the fastest half of the throughput samples are used in the averaging.
While this selection process does not result in the peak throughput, it is clearly higher than the session
average as three times as many slow samples are discarded compared to the fast samples in the download
test, and only the fastest 50% of samples are used in the upload result.
Alternatively, file transfer testing gives a throughput measurement based on the file size and the transfer
duration. This normally involves single-thread testing and so may underestimate the potential throughput of
some user applications such as peer-to-peer and web browsing, but is more representative of the user
experience for file upload and download and for potential data streaming applications.

Results Comparison
While NetIndex.com3 publishes throughput results from speedtest.net, YouTube provides a (limited)
breakdown of throughput4. Though not a direct comparison, monthly results for these can be compared for
a country as a whole. Table 1 below shows the summary download throughput for 1 month for Ireland.

Average Download Speeds (1 Month5)

Speedtest

YouTube

10.74 Mbps

8.11 Mbps

Table 1: Download Throughput Comparison for Speedtest and YouTube for Ireland

https://support.speedtest.net/entries/20862782-how-does-the-test-itself-work-how-is-the-result-calculated
http://www.netindex.com/download/2,49/Ireland/
4
http://www.youtube.com/my_speed
3

30 Sep 2012 to 29 Oct 2012

In this case the Speedtest result is 32% faster than the YouTube speed results. There are obviously many
factors that can account for the differences in the results including sampled users, mix of ISPs, times of use,
etc. A very significant difference is that Speedtest attempts to fully load the link in their throughput test,
whereas the YouTube throughput may in fact be less than the available bandwidth in many instances and so
not fully loaded.
A second illustration of throughput differences is available from the report Understanding Broadband
Speed Measurements by S. Bauer, D. Clark, W. Lehr [3]. This presents throughput comparisons for the US
and the Massachusetts area from a number of different throughput estimating applications. Again, we see
the speedtest.net results are significantly higher than the others. The report gives a detailed explanation of
the main factors affecting the differences in the throughput speeds and the interpretation of the results
based on the test methodology. In this case the significant factors include throughput sampling versus file
transfer and multi-threading versus single TCP connections.
Reported US
Reported MA
Download Speeds
Download Speeds
Speedtest/Ookla
7.93 Mbps
11.54 Mbps
Comscore
3 4 Mbps
n/a
Akamai
3.9 Mbps
4.9 Mbps
Youtube
4.21 Mbps
7.39 Mbps
Table 2: Download Speeds from differing test systems [3]
Reporting Site

Multi-Threading
Multi-threading is the establishment of multiple simultaneous TCP connections between the user application
and the target server. This is done as the combined throughput from all threads may be higher than the
throughput for a single thread on the same link or path.
The assumption underlying the single TCP for Bulk transfer capacity is that the transport protocol is well
tuned and capable of fully using all the network capacity if it is available. However, this is not always the
case, and the client PC configuration, especially TCP receive window (RWIN), can lead to throughput
restrictions. In other instances, there are network or server limitations6 on the throughput per connection
available and so, often, the throughput test cannot fully load the available bandwidth.
The Ookla/Speedtest tools employ multiple TCP connections to try and use all the available bandwidth. This
is key to avoiding the TCP receive window (RWIN) limitation if a test is done from a desktop computer, and
so this gives a better representation of the available bandwidth and not the configuration limited
performance of the client PC.
Testing with multiple simultaneous TCP connections is representative of many applications on the Internet
today: Firefox, Internet Explorer 8, and Googles Chrome all use multi-threading with up to six simultaneous
for example, to maximise the number of connections/users that the server can manage or due to hardware limitations
from legacy equipment
6

connections per domain. As many websites use two or more sub-domains the number of simultaneous TCP
connections can even exceed this. Peer-to-peer traffic also uses multiple simultaneous TCP connections.
However, file transfers7 and video streaming applications such as YouTube still only use single TCP
connections.
Microsoft implemented TCP Auto-Tuning in Windows Vista and Windows 7. With this, RWIN is dynamically
optimised for the bandwidth available and requested and so throughput is less prone to inefficient receive
window settings on the user side.

Conclusions
The significant differences in throughput testing results from speedtest.net and from file transfer testing can
be attributed mainly to multi-threading and sampling & filtering as employed by Speedtest.net. Ooklas goal
with Speedtest is to measure the total capacity of the broadband connection, something they call peak
sustainable throughput. This is done with a fill the pipe approach generating as much traffic as possible
and then aggressively measuring how much is actually getting through8.
The file transfer test measures the actual throughput for a file and includes factors such as throughput ramp
up at the start of the session or transfer, retransmission time due to errors, periods of excessively slow
throughputs, etc. that are not always captured by the speedtest.net method.
Both results are valid, choosing which test to use just depends on what you actually want to measure.

7
8

Some FTP servers now implement multi-threading to speed up file downloads

http://blog.ookla.com/2010/05/14/testing-speed-tests/

REFERENCES
[1]
Internet Engineering Task Force, RFC 3148, Bulk transfer capacity
http://tools.ietf.org/html/rfc3148
[2]
Ookla, Speedtest.net throughput calculations:
https://support.speedtest.net/entries/20862782-how-does-the-test-itself-work-how-is-the-result-calculated
[3]
S. Bauer, D. Clark, W. Lehr. Understanding Broadband Speed Measurements. Technical report.
2010.
http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurements.pdf

For further information contact:

Baldev Gill
baldev.gill@vilicom.com
+44-1483-243-591

Stephen Shannon
stephen.shannon@vilicom.com
+353-1-435-8420

Vilicom is an expert provider of

consultancy services with over ten
years of experience in the analysis,
design, test and implementation of
wireless networks. Vilicoms
strengths lie in technology strategy
consulting, the planning of cellular
networks, transmission network
design, implementation of
specialised in-building coverage
systems and network benchmarking
and testing. Vilicom has delivered its
services in over 20 countries for
network operators, network
equipment vendors, industry
regulators and investors. Vilicom
delivers value for its customers by
adopting a flexible, customerfocused approach, retaining cuttingedge expertise and maintaining its
independence.

2012 Vilicom Engineering Ltd.,

14 Joyce Way, Park West, Dublin 12.
vilicom.com
+353 1 435 8420