Measuring Mobile Data

A Vilicom White Paper

Author
Title
Date of Publication

Brendan McKenna, B.Sc. (Hons), C. Eng. MIEI

Senior Consulting Engineer
22nd February 2011

Copyright Vilicom Engineering Ltd.

All Rights Reserved.

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

Measuring Mobile Data

Mobile network operators and service providers have always needed to measure their own and their
competitors network performance. In the past couple of years mobile data usage has exploded and
is accelerating at a rate which can be considered phenomenal even by the mobile industrys exacting
standards for that term.
Such rapid growth means it is no longer sufficient to travel out once or twice a year and compare
your performance with your competitors. The rate of investment and change in mobile data
networks demands much more monitoring than that. But how can this be done in a meaningful way
reflective of actual customer experience? And anyway what exactly is it that needs to be measured?
In this paper, Vilicom seek to answer these and other questions about mobile data measurements.

The Mobile Data Experience

Fig. 1: What is Driving Mobile Data Takeup?

Mobile data take up rates in the last few years have been remarkable. During 2008 - 2010:

O2 UK saw data traffic increase 20-fold in 12 months, doubling every four months1
Vodafone UK saw data traffic increase 10 fold in 12 months1
AT&T in the US experienced mobile data volumes up 5,000% in the three years since they
introduced the iPhone in June 2007. The AT&T CEO is quoted as saying that he thinks that
this will increase a further 40 to 60 times over the next 5 years from July 2010s levels.
AT&T will spend $90bn this year alone on their network infrastructure.2
Mobile broadband subscribers and revenues in the top 5 European markets are forecast to
nearly double by 2011.3

A comparison with the launch of desktop internet applications such as Netscape and AOL is shown in
Figure 24. Apart from the phenomenon that is the iPhone (and its increasing number of
competitors), some of the other reasons for the amazing growth of mobile data include the
increasing use of mobile data cards (or USB modems), the exponential viral marketing that comes

Recognising the Promise of Mobile Broadband. The UMTS Forum 2010. Page 7.
th
Fortune.com C-Suite interview with Randall Stephenson, CEO of AT&T on August 5 , 2010.
3
Mobile Broadband in Europe, 4Q09, CCS Insight, January 2010.
4
Economy + Internet Trends, Mary Meeker, Morgan Stanley, Web 2.0 Summit, October 2009.
2

Page 2

22/02/11

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

with social networks like Facebook and
the increasing data capabilities of
Smartphones driven by Moores Law.
Another factor that has driven mobile
data uptake has been flat rate plans.
These were introduced by operators as a
result of a desire to compete with fixed
broadband providers. The extraordinary
demand has compensated for this poor
revenue model but expect the model to
change as the high end users are
identified and targeted.

Figure 2: Mobile internet outpaces desktop internet adoption. Source: Morgan Stanley.

Comparing the Mobile and Fixed Data Services

The fact that mobile data providers are competing with fixed broadband providers leads to many
comparisons between the two types of data and many issues for the mobile networks.

Speeds
The mobile data speeds quoted by mobile network operators appear to be often at variance with
those experienced by users. Such differences are best shown in some of the online board
discussions. Mobile network operators want to close the gap between reality and theory and one of
the biggest obstacles to doing this is the lack of verifiable and extensive mobile data measurements.

Quality
3G cells breathe and there are many variables
determining the perceived quality of service by an end
user including e.g. contention and RF signal level. The
end user sees an internet connection with intermittent
performance but the underlying reasons can be very hard
to detect.
They could include user error, PC
configuration, base station location and capacity,
backhaul capacity fibre deployment, core network
signalling and data capacity, and connections to the
Internet from the core. In contrast, with fixed line broadband it is relatively easy to ensure a steady
quality of service.

All this leads to the need to know what is actually being delivered by the network and that can only
be discovered by measurements.

Page 3

22/02/11

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

Measuring Mobile Data: Factors to Consider

Consumer Experience v. Network Capability
Mobile Network Operators (MNOs) want to know about consumer experience if they are
benchmarking themselves against their competitors. There can be a number of reasons for
measuring network capability, e.g. if they are assessing whether a vendor has met their specified
data rates in the newest rollouts or testing a change after a cell site optimisation.
Measuring network capability, while complex and expensive to do correctly, is a relatively straight
forward engineering task. However there can be many more soft as well as non obvious
engineering factors to consider when trying to measure consumer experience.

Mobile Broadband USB modems v. Smartphones

Currently equipment vendors normally incorporate the latest 3GPP standards for maximising data
throughput into USB modems before they are implemented in Smartphones. This is because of the
complexity of the latter compared to the former. So for up to date network performance, USB
modems are useful. It is also easier to set up tests on mobile broadband USB modems since there is
a full PC environment behind them on which to construct the test routines. In addition, mobile
broadband USB modems connected to laptops generate 22 times more traffic than Smartphones
(1.79GB compared to 79MB per month)5 so that they are a very significant part of the market and a
significant stress point for mobile data network capacity and quality.

Measurement Locations
Mobile Data Measurement locations of interest to Mobile
Network Operators include cities, interconnecting roads, points
of interest and rural areas. Measurement location categories
include in car, outdoor and indoor. Mobile broadband usage is
primarily indoors in residences and offices and at points of
interest such as shopping centres, restaurants, hotels, train
stations and airports.
In choosing the measurement locations there is a need to obtain as many measurements as are
required to ensure statistical significance. The locations chosen should be places where mobile
broadband is heavily used such as shopping centres, airports, train stations and built up office block
locations.
An in car measurement method can be made to simulate an indoor environment and enables
flexibility in location selection.

Measurement collection methods

There are three main methods of doing mobile data measurements: drive tests, static / nomadic
points of interest and walk tests6. Measurements of mobile broadband experience are best
5

Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update 2010 2015, February 2011.
For the purposes of this white paper, static tests are defined as those done using measurement equipment or
probes left in situ. Nomadic tests are carried out using a measurement tool transported from point to point.
Drive tests are tests continuously carried out while the vehicle carrying the test tool is moving at typical vehicle
speeds.
6

Page 4

22/02/11

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

undertaken in static locations. However mobile data measurement equipment and probes are quite
expensive so nomadic point of interest measurements can be used to add flexibility and to reduce
the initial capital outlay for the customer. Once proven and accepted by the customer they can then
move to unattended probe operation. Unattended, centrally managed probes will gradually take
over from nomadic points of interest at a majority of locations where they can be installed and
maintained safely and under a satisfactory agreement with landlords.

The Hardware
Consumer experience could be affected by processor (CPU), on board memory (RAM) and hard disk
drive speeds as well as the data USB modem and smartphone specifications. A PC based
measurement tool is the most flexible and cost effective solution and should be based on a mid
range CPU, RAM and hard drive specification.

The PC Operating System (OS) and other software residing on the PC.
Different OS could have different default settings that affect the mobile data speeds. Software can
clog up RAM and hard disk space and affect CPU speeds. A data measurement tool should use the
most commonly used operating system and browser applications (i.e., Windows XP7 and Microsoft
Internet Explorer version 88) pre-installed to match customer experience as closely as possible.

The USB modem management software.

All USB modems come with mobile service provider specific management software which is installed
automatically when the modem is connected. When this management software is being installed it
may change default settings on the PC for various parameters. For instance, Vodafones default
setting for its Mobile Connect USB modem management software is to change the TCP/IP window
size to 131072 bytes for HSPDA and set the number of retransmissions before aborting the
connection to 5. This software and other mobile network operators USB modem management
software also installs optimisation software from ByteMobile.
A dedicated PC for each operator is needed to keep the settings implemented by the operator
specific management software on each PC during tests if customer experience is to be replicated.

Typical consumer mobile data consumption patterns.

A survey conducted by Ciscos Internet Business Solutions Group (IBSG) indicates that the
percentage of time spent using mobile Internet at home is approximately 40 percent of total mobile
data use, on average. The amount of mobile data use that is on the move is approximately 35
percent, while the remaining 25 percent of mobile Internet use occurs at work.9
Normal user web browsing patterns include random delays between web page downloads. These
and other usage patterns need to be analysed and fed into the test tool and test cycle design.

In January 2011 Microsoft had nearly 90% of the global installed base of operating systems with XP at 55%
and Windows 7 at 22% (Vista has the remaining 12%). However Windows 7 share is doubling year on year and
is expected to take over from XP as the most popular operating system by early 2012. Source:
NetMarketShare.
8
Microsofts Internet Explorer (all versions) had 56% of the browser market on all devices in Jan 2011.
Internet Explorer version 8.0 had 34% market share. Source: NetMarketShare.
9
Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update 2010 2015, February 2011.

Page 5

22/02/11

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

What types of data should be measured?

Application
Data*

Data Communications

File Sharing

HTTP

Share of Traffic
26.39%

HTTPS

0.64%

VPN and Tunneling

0.57%

Admin

0.21%

Default Service

0.14%

Maps

0.02%

Other

0.00%

Email

0.23%

Instant Messaging

0.12%

Instant Messaging File Transfer

0.03%

P2P

24.85%

Web-Based and Other File Sharing

18.69%

Gaming

PC Gaming

0.65%

Gaming Consoles*

Xbox

0.12%

PlayStation

0.04%

Nintendo Wii

0.00%

Online Video*

Voice and Video

Communications*

Streaming Video

Tests involving web browsing (HTTP sessions

in parallel), HTTP download (includes web
based and a proportion of P2P file sharing)
and streaming video capture most consumer
experience based mobile data measurement
types currently (see table).
Predictions from Cisco point to a much
greater increase in video traffic (see chart
below)10 than other types in coming years
and this will have to be addressed in any
data measurement scenario. The file sizes
chosen for HTTP data transfers should be
typical of consumer experience using mobile
broadband.

10.52%

Flash Video

6.99%

Streaming Video via P2P

4.80%

Audio and Video over HTTP

3.56%

Video Downloads

0.28%

Other VoIP

0.64%

Skype

0.57%

Media Gateway Control Protocol (MGCP)

0.40%

Session Initiation Protocol (SIP)

0.05%

Voice and Video over Instant Messaging

0.04%

Phone VoIP

0.01%

H.323

0.00%

Poor latency on a network should also be

taken into account since it affects customer
experience. Ping tests can cover this aspect
of the data measurements.

Table 1. Broadband Traffic by Application Subcategory, 3QCY10

* These categories contain video.
Source: VNI Usage, 2010

10

Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update 2010 2015, February 2011.

Page 6

22/02/11

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

Vilicoms Data Evolution Watch (DEW)

Data Evolution Watch is a managed service from Vilicom that enables
ongoing competitive evaluation of data network performance
according to simple, agreed KPIs at customer premises and points of
interest as well as during walk and drive tests. The DEW Benchmarking
Tool has been fully developed in house by Vilicom and consists of off
the shelf PCs and software designed to emulate customer experience
as closely as possible. It uses USB modems to measure data download
speeds with a nomadic point to point test methodology.
There is an internet web browser
based view available of the results.
This gives unified and easy access to
the data gathered via a number of user
friendly web based screens. The data
can be updated on a daily, weekly or
monthly
basis
using
results
automatically processed from one or
more DEW Benchmarking tools in the
field.
Vilicom have striven to ensure that all the many factors to be considered in measuring mobile data
have been included in the design of DEW:

Consumer experience based

measurements
Mobile broadband USB modems
Suitable for point of interest indoor, in car
and outdoor measurements
Consumer experience based hardware and
third party software choices
Measurements of commonly used mobile
data application level protocols

A PROGRESS/COMPETITOR
Monitoring service
Built around a mobile data only
Vilicom Benchmark Tool
Focus is on Key User perceived
data metrics
Multiple deployments, rapid &
high resolution testing

Armed with DEW a mobile network operator can:

Monitor in (near) real time the evolving performance of your competitors Data Network
performance vs your own.
Monitor at a higher resolution and higher frequency than possible using traditional drivetest services.
In these times of rapid data network change, know immediately when and where your
competitor has recently upgraded their network, or is lagging.
Get instant benchmark data on the impact of your own network improvements.

Page 7

22/02/11

Copyright Vilicom Engineering Ltd.

Version 1.0

Mobile Data Measurements White Paper

Standards
STQ is the European Telecommunication Standard Institutes (ETSI) technical committee for
Speech and Multimedia Transmission Quality. It was formed in 1997 combining interests
from the former BTC2, TE4 and TM5 areas of ETSI. It was intended initially as a centre of
expertise on speech quality issues but has broadened its scope to handle more general
quality issues including measuring methods.
Any measuring tool needs to take into account technical specifications such as ETSI TS 102
250-2 and other report outputs from this committee in the design of mobile data
measurement tools.
The Third Generation Partnership Project (3GPP) currently manages all the UMTS/3G
Wideband CDMA technical standards. In particular the TSG Radio Access Network manages
the standards for mobile data transmission.

Page 8

22/02/11