WHITEPAPER

MANAGED TESTING
SERVICES
Focus on your core competencies by
test outsourcing

DR DANIEL SIMON
Senior Research Manager
daniel.simon@sqs.com
Daniel Simon studied Computer Science and has been working for SQS since
2005. His main responsibilities include conducting research projects for the
Group, evolving and innovating SQS service portfolio, and demonstrating its
thought leadership. His core competencies relate to outsourcing, technical
quality, KPI systems, and quality management. His range of experience covers
all sectors of the IT industry, and he is Programme Chair for the iqnite
conferences 2012 at Vienna and Geneva.

DR FRANK SIMON
Head of SQS Research
frank.simon@sqs.com
Frank Simon studied Computer Science and has been with SQS since 2001.
He is Head of SQS Research, Head of the BITKOM working stream Software
Development Processes and Tools, and a Member of the German Testing
Board. His responsibilities include thinking ahead, innovating SQS service
portfolio, evaluating hypes and trends, conducting research projects, developing services, and presenting SQS research in lectures.

Managed Testing Services

Page 2

1 MANAGEMENT SUMMARY
This whitepaper outlines the concept of Managed
Testing Services (MTS), an outsourcing technique
for testing-related tasks across one or more projects delivering one or more applications and
systems, spanning the life cycle of software and
system development and system operation.
The paper argues the idea of IT industrialisation
as the underlying motivation for using MTS, and
shows that MTS are an important step on the industrialisation road. It explains the different parameters that need to be taken into account when deciding on IT outsourcing, and gives an overview of
important prerequisites that have to be checked
before MTS benets can be leveraged.

tion has achieved a reasonable level of maturity

and is demanding changes towards process and
organisational modularisation, standardisation,
automation, and focusing on core business competencies. If the prerequisites are not yet met,
MTS in themselves will be worthwhile to develop
an organisation towards the higher maturity level.
Since MTS are a specic instantiation of the
outsourcing concept, the current status of this
market is analysed and an outlook is given on
expected future developments for MTS: Managed
Testing Services will mainly be delivered by independent testing companies, blurring the difference to the large full-service providers.

MTS are a logical consequence of IT industrialisation in modern IT environments. MTS promise

signicant savings and quality improvements due
to the effects of division of labour and specialisation, as well as exploiting location-independent
delivery of well-dened pieces of work. Nevertheless, MTS are most efcient when an organisa-

Based on these high expectations, this paper describes the steps towards setting up and operating
MTS, focusing on costs and benets, opportunities and risks in the context of these services.
Since MTS will often be applied to already existing
testing projects, a separate section explains the
transition step which is necessary to apply MTS.

2 THE CONCEPT OF MANAGED

TESTING SERVICES
This section gives a short overview of the fundamental basics of so-called Managed Testing Services
(MTS). Before a clear denition is provided, it will
be shown how both IT and innovations in IT have
evolved over the last decades, explaining that Managed Testing Services are a manifestation of a more
fundamental concept. This overall idea leads to
specialisation (i.e. organisations focus on their core
competencies), division of labour (i.e. outsourcing
in different variations) and (cross-country / crosscultural) supply chains (i.e. global delivery).

As Adam Smith outlined as early as 1776 in his

famous An Inquiry into the Nature and Causes
of the Wealth of Nations (1), progress and a
qualitative increase in productivity in society
lie in specialisation and industrialism, essentially represented by the concept of division of
labour. In Smiths work, the different approaches
to manufacture pins were investigated and compared, but the same principles apply to the IT industry today and reect the evolution seen over
the past decades.

Managed Testing Services

Page 3

As the IT industry has evolved over time, fundamental changes to businesses and their supporting IT organisations have occurred. Recent
years have seen the development from integrated full-service units towards specialisation
of departments and companies along the various functions required to run businesses and
their supporting IT. Additionally, cost efciency
considerations put high pressure on executives
who try to focus on their core competencies
and hand over non-core tasks to specialised
providers.

evolve according to the maturity of the business partners, offering increased predictability
of costs as well as exibility and improved overall resource utilisation. A more detailed view on
these dimensions can be obtained by means of
the global delivery cube explained in the previous whitepaper.

2.1 INDUSTRIALISATION: THE ESSENCE

BEHIND MANY INNOVATIONS

2.2 THE TERM MANAGED TESTING

SERVICES

Growing technical innovations in the IT industry

increasingly affect the way in which a service or
product is built, delivered, and consumed. The
new way IT understands itself is often called
industrialisation and is strongly related to the
car industry and its history: while the production-focused sectors (such as the automotive
industry) have already undergone signicant
changes from craftsmanship to an industrialised
approach, for IT this is starting right now.

Managed Testing Services are a combination of

managed services and testing activities. Both of
these are rened in terms of the innovation
YkX[(i[[ <_]kh['_dj^[\ebbem_d]ikXi[Yj_edi
X[\eh[WYecfb[j[Z[\_d_j_ede\CWdW][ZJ[ij_d]
Services is given.

There are different dimensions to consider when

looking into the details of the IT service industry.
The rst and most prominent dimension among
these would be the vertical range of manufacture, which refers to the various engagement
styles possible when liaising with businesses.
The second dimension addresses the service
delivery location and brings attention to the
fact that in todays IT industry various technical
means of collaboration across geographical limits
are ready for day-to-day use (e.g. delivery of the
service through the use of Internet technology,
available virtually everywhere in the developed
and developing world, or through the use of
concepts such as Cloud Computing). And as
a third dimension, the compensation models

Finally, Managed Testing Services also can be

directly linked to IT industrialisation since they
help customers focus on their core competency,
which rarely is in the testing area.

2.2.1 MANAGED SERVICES

The term managed services refers to a business situation where a customer and a service
provider interact in a particular set-up. The
customer will consider making use of a service
outside his core competencies if he needs it to
deliver an important (supporting) process for
his core business processes but neither wants to
manage the resources required for delivery nor
the associated risks. To this end, the entire service is dened and handed over to the service
provider, making him responsible for the delivery of the respective service in time, in budget,
and in quality.

Managed Testing Services

Page 4

BUSINESS PROCESS
OUTSOURCING
APPLICATION
OUTSOURCING
INFRASTRUCTURE
OUTSOURCING
BUSINESS
VALUE-BASED

OUTTASKING

TRANSACTION-BASED
BODY LEASING

FIXED PRICE
COST PLUS

DIY

RE
SH
O

FF

H
RS
EA
N

Figure ': Global delivery cube with its dimensions

RE
O

O
H
NS
O

NS

IT

RE

TIME & MATERIAL

Managed Testing Services

Managed services can be pinpointed by the following dimensions within the framework of the IT
service innovation cube:
Engagement styles: Managed services can be
applied for infrastructure outsourcing (e.g.
managed services for delivering databases),
application outsourcing (e.g. managed services
for delivering ofce suites), and even business
process outsourcing (see below, Managed Testing Services).
Geographical locations: There is no need
for a particular geographical favourite since
managed services focus on output. However,
in many cases nearshore and / or offshore
resources are utilised to be able to offer
competitive pricing, if legal and regulatory
constraints do not hinder it.
Compensation models: In view of the aims
of competitive pricing and the realisation of
scaling effects, and in order to account for
the investment into the set-up of a managed
service, the overall volume of the services
purchased must be reasonably sized. In most
cases, managed services focus on repetitive
tasks suggesting a transaction-based compensation model. If there is a linkage between the
delivered service and the business value, even
business value-based pricing might be possible.

2.2.2 TESTING ACTIVITIES

In the IT industry, testing refers to the process
consisting of all life cycle activities, both static
and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specied requirements, to demonstrate that
they are t for purpose and to detect defects. (()
Typically, the input to testing is the asset under
test and the corresponding specication of the
asset under test. The output of testing is an
analysis of deviations of the asset under test

Page 5

with regard to its corresponding specication.

The output will be used to remove errors from
the test asset and to assess the overall completion of development work. Before the go-live of
systems, test results can be used to evaluate the
systems readiness for use in productive environments and the risks associated with the golive. Testing nowadays is widely recognised as a
mandatory activity in the software development
life cycle and should be closely linked to enterprise risk management since testing identies
defects impeding specic risks if rolled out.
In software and systems development, testing
activities can start as early as requirements
elicitation and can span the entire life cycle
down to deployment and maintenance phases
of the software and systems. Recent surveys
conrm that testing activities are deemed most
effective when they are performed by an organisational unit that is independent of the development units. ())
There are international standards for testing
(e.g. those provided by the ISTQB), including
capability and qualication schemes. Testing is
a prime candidate for service outsourcing for
customers, not only in order to focus on core
competencies but also to exploit the advantage
of utilising independent resources. Moreover,
many organisations are already used to collaborating with testing teams and will have identied
and set up their testing capacities in a proper
organisational framework. Depending on the
maturity of the organisation, tests are executed
independently from the development tasks and
report to different lines to ensure their independence. This set-up is widely used and makes
it possible to integrate a specialised service provider with reasonable effort.

Managed Testing Services

2.2.3 THE DEFINITION OF MANAGED

TESTING SERVICES
In view of the preceding two subsections, a definition of Managed Testing Services might read
as follows:
Managed Testing Services (MTS) are managed services for testing-related tasks across
one or more projects delivering one or more
applications and systems, spanning the life
cycle of software and system development
and system operation. The resources for testing (staff, testing infrastructure, system under
test) are managed by and under the responsibility of the service provider, to support the
customers business processes. Based on
transaction- or business value-based prices,
the service provider ensures scalability of the
MTS and takes care of the resource management according to the utilisation scheme required to support the tasks.
Denition 1: Managed Testing Services (MTS)

Managed Testing Services according to this definition have the following objectives (cf. (*)):
Complexity reduction: Since MTS mean
outsourcing on a high level within the process
chain, they reduce the overall process and
organisation complexity on the customers
side by providing a black-box view of it.
Risk reduction: MTS providers usually take on
a high degree of risk to deliver MTS. This is
underpinned with corresponding Service Level
Agreements (SLAs) to shift testing risks to the
provider altogether.
Competitive pricing: This is typically realised
by a low onshore rate and by moving the
geographical focus of the service mainly to
nearshore and offshore locations.

Page 6

Output- or transaction-based compensation

model: This makes it possible to overcome
the time & material-based models and shift
enterprise resource risks to the provider.
Thus, it simplies budget planning and
generates more budget exibility.
Leveraging scaling effects on the MTS providers side: An MTS provider specialises in
testing and he processes many similar tasks
around testing which permit setting up the
best possible t-for-purpose infrastructure
that is supported by the best-tting tools integrated into an overall smooth toolchain. These
scaling effects allow for very quick testing
cycles (because usually the provider uses an
optimised tool suite for automation) and short
response times (because an MTS provider has
an established and matured process landscape
in place, suggesting dedicated processes for
incident management or reporting).
High quality of service: For the provider of
MTS, testing is a core competency. He has access to specialists, best practices, tools and infrastructures to deliver high-quality services.
In addition, many service providers are able
to use the advantage of being truly independent of the development disciplines, as the
provider is not integrated into the customers
organisation (cf. Section 3.3).
Innovation improvement: Using MTS allows
customers to refocus on their core competencies again. Thus, the services free up customer employees capacities, improve motivation,
and generate new innovation potential.
Increased range of the service-oriented
approach: The idea of thinking in serviceoriented structures and organisations is not
limited to testing but can be expanded to HR,
marketing, sales, etc. Using MTS supports this
general approach.

Managed Testing Services

2.3 PREREQUISITES AND BENEFITS

MTS have benets for both service providers
and customers. The benets for the service
providers are transferred to the consumer, e. g.
by higher cost efciency and more competitive prices compared to time & material-based
projects.
Even though the monetary and non-monetary
benets are widely recognised, it should be
noted that the exploitation of these benets
comes with prerequisites. If these prerequisites are not met on either the customers or
the service providers side, the risk of false
promises and failure to achieve the objectives
is high. However, a well-dened transition process like the one presented in Section 4 will
identify those gaps as early as possible and will
bridge them efciently.
2.3.1 PREREQUISITES
There are three different types of prerequisites
that should be fullled before leveraging MTS
benets:
Industrialised process chain: First of all, the
organisation and processes on the customers
side must be mature enough to support the
division of labour across organisational units.
This typical industrialisation state has at least
four prerequisites:
1 Modularisation: The overall business process
chain must be rened to a level where a
detailed process landscape can be identied.
This process landscape denes activities,
responsibilities, dependencies, and results.
2 Standardisation: Based on the modularisation, the process landscape as well as the
interfaces must utilise standards. Applying
those standards guarantees a specic level
of quality and makes establishment easier
due to an abundance of existing literature
and knowledge.

Page 7

3 Automation: For each activity within each

process step or even for complete process
steps, the possible level of automation has
to be considered. There will be activities
which require a lot of manual interaction, while others might be automated
completely.
4 Focus on core competencies: At this level,
it can be decided if MTS are possible or not.
The fundamental decision whether it makes
sense to utilise MTS can be taken if and
only if management have a conscious view
on what the core competencies of the business are. This decision can only be taken
for identied tasks (modularisation) using
standards (e.g. by the ISTQB) and automation to a certain degree.
The decision for or against MTS can only be
made if Steps 1 to 3 are successfully completed.
Otherwise, it is not clear (i) what testing means,
(ii) how testing works, and (iii) to what degree
testing can be automated. It is important to
point out that a company on its way to Level
4 can already realise signicant benets along
the way by improving its industrialisation level.
For example, standardisation has values of its
own long before MTS can nally be used. And
these values may be utilised even without taking the last step of using MTS.
MTS volume: Due to scaling effects and ramp-up
efforts, the degree to which MTS pay off depends
on the size of the testing effort and the number
of applications subject to Managed Testing Services. Typically, the benets of MTS are realised
in three- to ve-year engagements and are not
achieved in a single project-based liaison between customer and provider. Therefore, the customer should be prepared to make the strategic
investment with the respective support by senior
management. For MTS, a customer needs to be in
a position to commit to a three- to ve-year plan
(often known as mid-term planning). Otherwise,
the increased risks for the service provider will

Managed Testing Services

have to be compensated for by higher prices.

Openness for change: Setting up MTS in an
organisation implies the shift of labour from
internal resources (including body leasing or
outtasking) to external resources having their
own responsibilities. In pure MTS, the external
resources are even no longer visible to the
customers staff. Thus, for most customers
changes will occur that need to be managed
appropriately, and management must be
fully aware of the consequences for internal
resources and address concerns proactively.
Ideally, the customer will already have experience in the eld of supplier management and
can build upon this experience to collaborate
with the service provider.

2.3.2 BENEFITS FOR CUSTOMERS

Key benets for service consumers can be realised over the run-time of the MTS agreement,
most prominently the cost savings resulting from
transparent pricing models such as output-based
pricing, as well as delegated procurement of test
infrastructure and tooling. If set up properly, customers will see an increase in transparency of cost
with regard to their application life cycles. Note
that this does not necessarily imply lower pricing
but prices are made explicit rather than being hidden in various budgets. One additional benet is
provided by cost attening: usually, the ramp-up
phase needs more effort than the execution of
MTS. However, this peak effort is not directly invoiced to the customer, who instead pays a at fee
over the whole life cycle. This is a key enabler for
management as an engineering discipline.
As MTS will be delivered by service providers
having expertise in the eld of quality assurance
and testing, it can be assumed that these service
providers have highly skilled resources and modern technical environments. If the MTS providers
are independent of the development organisa-

Page 8

tion, the best possible organisational set-up for

running independent tests has been chosen. Both
of these aspects will have a positive impact on
the overall effectiveness of the testing process,
and will ultimately result in a higher quality of
the applications and systems being tested (cf.
Section 3.3).
In many cases, the volume of the MTS is high
enough to justify investments to leverage crossproject and system synergies in terms of test automation and systematic regression testing. Not
only does this increase the effectiveness of the
testing service but it also reduces response times
from the testing functions to the development organisation.
If the MTS provider can make use of offshore
testing in different time zones and the delivery
processes are set up properly, a follow-the-sun
approach of MTS delivery can be achieved which
will dramatically reduce the turnaround times of
testing. This means that the development and the
testing capabilities in a project can be used in a
24-hour schedule by following the time zones. On
the other hand, this approach requires mature
processes and a signicant amount of trust between the different parties to be successful.
From the perspective of the customer, a signicant risk and management effort concerning the
resources is delegated to the service providers.
Since the latter will have different customers with
different schedules, the scalability of resources in
the project life cycle is no longer an issue for the
customer and will be managed by the service provider. Therefore, the customers internal resources can be released from non-business-related
tasks and can focus on his business core activities. He will still need to contribute and provide input, but this can happen in a more systematically
structured way and thus will result in both higher
quality and less effort.

Managed Testing Services

Page 9

2.3.3 BENEFITS FOR PROVIDERS

Providing MTS to a customer has a number of
advantages over time & material projects for
the service providers. Typically, MTS are aligned
with an overall strategic sourcing decision of the
service customer, and a strategic partnership is
established for delivering MTS. Strategic partnership decisions usually cover a timeline of three
to ve years commitment and accordingly give
the service provider a security of revenue for his
own planning, along with an enhanced forecast

on stafng needs. The signicant sizing of MTS

allows the service provider to set up core teams
that facilitate an internal handover in case of staff
uctuations or peak workload. He also has the option to build up internal core expert teams which
will be cross-pollinated from testing expertise and
customer / domain expertise. At the same time,
resource demands from different projects (even
across different customers) can be managed on a
more exible and scalable basis.

3 MARKET CURRENT STATUS AND

OUTLOOK
3.1 OUTSOURCING AND CLOUD COMPUTING

These numbers clearly demonstrate the overall

trend of IT industrialisation: a growing number of
Outsourcing as the overall approach behind MTS companies use third parties to deliver IT services.
is already a well-known and established concept. And this trend is set to increase over the next
However, in the IT industry, maximum coverage few years. A study conducted by market research
has not yet been achieved. Typical trends in out- rm PAC expects an outsourcing ratio of about
sourcing and how they have evolved over the last 25 % by 2020 this means that one out of four
few years as a percentage of total IT budgets are euros in the IT market will be spent on outsourgiven in Figure (, based on a study published by cing. (,)
Computer Economics. (+)

2008

3.8 %

2009

6.1 %

2010

7.1 %

0%

4%

Figure (: Outsourcing ratios as a percentage of total IT budgets over the last three years

8%

12 %

Managed Testing Services

Today, outsourcing is strongly associated with the

usage of cloud services and cloud computing in
general. For the following, we will use Gartners
denition of Cloud Computing:
Public cloud computing [is a] style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service to
external customers using Internet technologies. Private cloud computing is dened as
a style of computing in which scalable and
elastic IT-enabled capabilities are delivered
as a service to internal customers using Internet technologies.
Denition 2: Cloud computing (-)

Page 10

Some studies refer 40 % to 50 % of the cost of

software development to testing (.), others
othersare
are
measuring a ratio of 1:1 for development to test
effort (/). InInaarecent
recentsurvey
surveyconducted
conductedon
onthe
the
occasion of the iqnite 2011 conference, 65 % of
the participants reported spending between 20 %
and 40 % of their overall project budget on quality management and testing activities. ())
However, most of the testing budget is spent on
system integrators which contradicts the bestpractice approach of separating product development from system testing. In a PAC study, 78 %
of the overall quality assurance budget was spent
on system integrators and only 22 % was spent
on dedicated and independent testers. ('&) This
is expected to change signicantly over the next
few years, as currently 57 % of companies still do
testing on the DIY level and will need to mature in
that respect.

Viewed from the outsourcing perspective, cloud

services are a particular type of outsourcing with
some additional constraints. Nevertheless, cloud
computings part of the outsourcing market will
increase from less than 20 % today to 70 % in 3.3 FULL-SERVICE PROVIDERS VS. TEST
2020. (,) A direct
directcomparison
comparisonbetween
betweenthe
thedefdefSERVICE SPECIALISTS
inition of MTS and the denition of cloud services
reveals a high similarity between MTS and cloud In general, Managed Testing Services can be procomputing because MTS are scalable and elastic vided by two different types of providers:
Test service specialists: Specialist providers
as well, most of the delivery of MTS is IT-enabled,
typically offer services in the subject matand the results can be provided using Internet
ter testing only. They do not offer forward
technologies.
development (implementation services), they
do not design software architectures, nor do
This similarity suggests that MTS will have a simithey have any particular interest in (or benet
lar rate of growth and can be delivered by means
from) any proprietary solution compromising
of cloud computing.
their independent approach. The advantage
of those players is that they are independent
3.2 TESTING SERVICES
of all systems under test. According to most
known testing literature, being independent is
The general overview of the IT market provided
one of the most important prerequisites for efin the previous section can be rened to focus on
MTS. Both academic research and industry expericient testing. Usually, test service specialists
ence report a range of gures on how much test
full this requirement by being an independent
effort is necessary and reasonable for a project.
test organisation (see Denition 3).

Managed Testing Services

An independent test organization is an organization, person, or company that tests

products, materials, software, etc. according
to agreed requirements. [] They are independent because they are not afliated with
the producer nor the user of the item being
tested: no commercial bias is present.
Denition 3: Independent test organisation ('')

The key advantages of using an independent

test organisation are the following:
Independence: Independent people are more
likely to question everything, provide an
objective second opinion, identify implicit
requirements, and objectively report any ndings. Different lines of reporting ensure that
no conicts of interest will interfere.
Scaling effects: Because independent test organisations are usually pure-play testers, they
test many different projects at the same time,
which allows them to set up highly sophisticated toolchains and utilise a large number of
synergies between different projects.
Resource capabilities: The skills provided by a
test service specialist are clearly focused on
one single topic, i.e. testing. This guarantees
a high, coherent skill level for all employees
and simplies the exchange of resources.
Full-service providers: These providers usually offer any services relating to IT systems,
starting from collecting requirements, designing system architectures, implementing the
entire system, testing the system, and nally
deploying the system into productive use.
Also administration of the systems and testing
services, including Managed Testing Services,
form part of the comprehensive service portfolio. In many cases the full-service providers
even market their own tool platforms, complementing and mixing their service ideas with
software license vending.

Page 11

The main advantages of using a full-service provider are:

Company size: A broader portfolio of services
and tools allows for a bigger market enabling
a bigger company size. Typically, full-service
providers are known globally and have a large
pool of employees at their disposal.
All-inclusive single sourcing: Using a fullservice provider simplies both the process
of selecting third parties and the negotiation
step. Having a contract with a full-service provider means getting everything through one
contract, with one key-account person it is a
one-stop shop.
Technical knowledge: Introducing an independent test organisation to a new IT system costs
more time and money than using one of the
service providers developers. The full-service
provider has deep knowledge of the technologies he develops and provides, which enables
him to start working without having a clear
knowledge-transfer step.

Both independent test organisations and full-service providers will try to move into the Managed
Testing Services market (see Section 2.3.3). However, the market will be further segmented: the
more mature an organisation is (e.g. in terms of
industrialisation), the more important attributes
like provider exchangeability, dependency reduction, and output-based pricing are going to be.
These considerations assume an explicit testing
awareness on the customers side. If it is given,
the application of MTS is a natural step, i.e. MTS
will successfully be provided by test service specialists.
Figure ) shows the total revenue generated by independent test organisations in 2010.

Managed Testing Services

Page 12

INDEPENDENT TEST ORGANISATIONS

163

SQS
Applabs

70

40

Tescom
28

DNV-IT
20

Tesnet

16

Plan IT

14

Thinksoft

13

Imbus

13

FHG / IESE
Amsphere
MTP

11

Acial

10

In 2011, Applabs was taken over by CSC Company.

12

ps-testware

Quality House

Maverick
QA Infotech

RTTS

7
7

Genilogix
6

Squerist

Experior

Assurity

KJR

Immune

m 0

20

40

60

160

Figure ) : Revenue of independent test organisations ('()

Currently, conservative estimates anticipate about

20 % to 25 % of all testing activities provided by
test service specialists to be realised through MTS
by 2015. For the independent test organisations
listed, this amounts to an overall business size of
about 121 million of revenue for MTS (25 % of
the total market size as listed above).
On the other hand, full-service providers usually
blend testing activities and package them into an
all-inclusive bundle. If the client lacks awareness
of what really is involved in testing, testing effort
and testing outputs, motivating him to change the

testing approach to MTS may prove difcult. In

other words, MTS are likely to remain a side product for full-service providers, and consequently
only about 5 % of all testing activities provided
by them will be delivered as MTS by 2015. Using
the revenue of the full-service providers listed in
the NelsonHall Report, this amounts to an overall
business size of up to 150 million of revenue for
MTS (5 % of the total market size as listed in ('()).
Consequently, the independent test organisations
will reach the same market share for MTS as the
full-service providers.

Managed Testing Services

Page 13

4 MANAGED TESTING SERVICES SET-UP

AND OPERATION
4.1 TRANSITION OVERVIEW
The MTS process is structured into three phases Pre-Engagement, MTS Framework Design, and
MTS Operation and laid out in Figure *.

1. PRE-ENGAGEMENT

2. MTS FRAMEWORK
DESIGN

3. MTS
OPERATION

Figure *: MTS process overview

The Pre-Engagement step covers the strategic

activities a client needs to perform in order to apply MTS. The MTS Framework Design step covers
the tactical activities within each clients project
to prepare MTS application. And the MTS Operation step leverages the benets associated with
MTS. Each of these steps is rened in the following subsections.
4.1.1 PRE-ENGAGEMENT PHASE
At the very beginning of the transition to MTS,
the customer needs to define and commit to
the vision of industrialising his IT environment
(see Figure +). On
Onthe
theone
onehand,
hand,management
management
need to identify the strategic services and core
competencies they want to operate on their
own behalf, and on the other hand, they need
to specify the supporting non-core services to
be outsourced.
The vision to industrialise IT and make use of service providers has to be supported by conscious
planning and budgeting decisions. A successful migration to an outsourcing model demands
proper preparation of the underlying business
case. Before MTS can be deployed into the organisation, the strategic sourcing decision must

be taken and at least mid-term planning perspectives have to account for handing over a complete process to a service provider. The strategic
decision is a sine qua non because of the sizing
of the engagement. Typically, using a managed
service requires the investment of signicant efforts into organisational maturity, and the corresponding changes have to be communicated and
managed appropriately.
The nal step of the Pre-Engagement phase is to
set up the relationship to potential service providers and selecting the best supplier. For MTS,
criteria for supplier selection start with matching the three dimensions from the IT innovation
cube (engagement style, geographical location,
and compensation model, cf. Section 2.1) to the
customers needs. Additional parameters to be
considered are the following:
Level of testing expertise
Independence from the development
organisation
Domain knowledge
Size of the provider
Commercial background
Track record and reputation of the service
provider

Managed Testing Services

VISION TO
INDUSTRIALISE IT

Page 14

STRATEGIC PLANNING
CONSIDERATIONS
(MID-TERM PLANNING)

PROVIDER
PRESELECTION

Figure +: Pre-Engagement activities

4.1.2 MTS FRAMEWORK DESIGN PHASE

One of the greatest challenges when setting up
MTS lies in the degree of IT industrialisation on
the consumers side. Clearly, without an explicit
organisational set-up and mature processes, the
realisation of benets from division of labour
and the consumption of specic processes /
services from the outside are rarely possible.
Therefore, the rst step in bringing MTS to life is
the assessment of the customers current organisation and its processes to evaluate their tness
for purpose (see Figure ,). The
Theassessment
assessmenttypictypically unveils a number of necessary improvements
to close the gaps and achieve the adequate level
of maturity. However, these improvements do not
only affect MTS readiness but essentially generate
added value even without using MTS.
In our experience, customers aiming at the use
of MTS (or other managed services) often overestimate their readiness for the consumption of
services. Typical unrecognised shortcomings are
the following:
Insufciently structured and dened processes, tasks, and supporting IT landscape as
well as undened or unclear ownership of the
process or IT components
Insufcient deployment of process and IT standards (e.g. non-existent or incomplete process
landscape, high number of variations in business
processes, unclear and undocumented interfaces between processes and departments)
High proportion of manually executed processes and technological and organisational
ruptures in process chains as well as deciencies in process interactions

Heterogeneous views on core business competencies and unclear strategic considerations

with regard to sourcing decisions

Depending on the results from the maturity assessment, different initiatives to industrialise and
professionalise IT will be launched as business
processes are set up, streamlined, and optimised.
Preparing the organisation for MTS increases the
necessity to outline the strategic vision of future
IT set-ups, and makes the corporate strategy visible to all employees.
Accordingly, the typical set-up for an IT improvement towards industrialisation consists of the
four steps mentioned in Section 2.3.1:
1 Modularise the entire value chain within the
organisation.
2 Standardise the process landscape, the processes themselves, and their interfaces.
3 Automate single processes, process interactions, or process chains, wherever possible.
4 Focus on core competencies and delegate peripheral services to external service providers
as managed services.

The benets of preparing for MTS are tangible for

customers when MTS are being used as a driver
for maturing the organisation. As a side effect
of the MTS Framework Design phase, customers
need to reconsider their internal processes and
organisational set-up. The maturity assessment
of processes and organisation can be leveraged
to foster clarity and transparency of core com-

Managed Testing Services

Page 15

ASSESSMENT OF
CURRENT
ORGANISATION
AND PROCESSES

PROCESS AND
RESOURCE
TUNING

FRAMEWORK
AGREEMENT
WITH PROVIDER

Figure ,: MTS Framework Design activities

CONTINUOUS IMPROVEMENT
VERIFICATION OF
TRANSFER
READINESS PER
APPLICATION

KNOWLEDGE
TRANSFER &
COLLABORATIVE
DELIVERY

SERVICE
OPERATION

SERVICE
COMPLETION

Figure -: MTS Operation processes

petencies and non-core supporting functions. It

is worth mentioning that through the course of
process improvement, the relationship with the
service provider can be established and the transformation can be used to build mutual trust.
Finally, when readiness has been accomplished,
the customer needs to design the framework
agreements with the service provider of choice.
Utilising Managed Testing Services requires methods to monitor and control service delivery. The
challenge lies in a proper denition of the SLA:
due to the black-box view of MTS, it should focus
on the compensation model rather than on internal MTS details. In order to link the KPIs to the
compensation model and evaluate them according to a reasonable schedule, a sound framework
needs to be dened. From the very beginning, the
denition of KPIs should take into account the
evolution of service level objectives over time to
establish appropriate productivity goals. For monitoring and controlling purposes, these KPIs are
utilised for an effective governance and communication structure and process, so as to be able to
react when the control limits are reached.

4.1.3 MTS OPERATION PHASE

After having set up the general MTS framework
for collaboration between the customer and service provider, the testing activities for the individual applications have to be transferred from
the customers responsibility to the aegis of the
MTS provider. For greeneld projects, this seems
a straightforward task whereas for existing projects there is potentially more substantial change
involved. In any case, the readiness for transfer
is ensured in the rst step, where potential gaps
are identied and closed accordingly (see Figure -). Secondly, the
theknowledge
knowledgetransfer
transfer(both
(both
from the development organisation to the MTS
provider, and from the test organisation to the
MTS provider) is initiated, and the rst iteration
of delivery is accomplished jointly. This collaborative delivery is a one-off undertaking designed
(i) to ensure a successful transfer of the testing
activities to the third step (Service Operation) by
passing on comprehensive knowledge to the MTS
provider, and (ii) to sort out any issues that might
arise during stand-alone delivery to the best possible extent.

Managed Testing Services

Page 16

Subsequently, the Service Operation step may be

executed as often as necessary over the lifetime
of the application, until this lifetime comes to a
close and both the application and its MTS are
decommissioned in the Service Completion step.
Since the Service Completion step is the reverse
of the transition step, planning for this step helps
limit dependency on a specic MTS provider. The
Service Completion step takes care of gathering
all information (including internal details like test
scripts, test data, or test procedures) necessary
to deliver MTS for a particular project. This allows
the customer to hand over the collateral information to any other MTS provider, or to insource the
testing activities at a later point in time, ensuring
a shorter introduction time.
The sequence of four processes is overarched by
the Continuous Improvement of the MTS Operation phase. During the Knowledge Transfer & Col-

laborative Delivery step, it helps determine and

ne-tune the agreed KPIs of the framework agreements and can be used to adjust the SLA. While
Service Operation is repeatedly executed, Continuous Improvement ensures that the KPIs are
properly monitored and adequate action is taken
to improve and control the delivery. The same
KPIs used to report back on service performance
should also be applied to manage and govern the
MTS Operation phase. Since the benets based
on improvements affect both consumer and MTS
provider, the sharing of the benets needs to be
clearly dened in the SLA.
Out of the four MTS phases, the MTS Operation
phase is the one covering the longest time span
it implies a shift from a project governance structure towards a programme governance structure.
A stereotypical organisational set-up is shown in
Figure ..

EXECUTIVE STEERING COMMITTEE

PROJECT GOVERNANCE

MANAGED SERVICE GOVERNANCE

Project
Management

Test Factory

MS Delivery
Management

Applications
Quality
Assurance

Functional
Sub-Projects

Technical
Sub-Projects

Testing
Services

Testing
Services

Test Team

Testing
Services

Test Team

Testing
Services

Test Team

SERVICE DELIVERY COMMITTEE

INNOVATION & TECHNOLOGY COMMITTEE
Figure .: Governance structures for the MTS Operation phase

Managed Testing Services

Project Governance rests entirely with the Project

Management (which then reports to the executive
level). The Project Management function supported by the Quality Assurance function performs Testing Services as per project by the same
hierarchical decomposition as the Functional and
Technical Sub-Projects. In the MTS setting, the
Testing Services liaise with the Quality Assurance
function and are managed as per application by
suppliers interfacing with the respective project
teams. The reporting lines to the executive level
are no longer purely bottom-up but complement
the internal reports with an independent view on
projects and applications from the suppliers perspective. This typically gives customer management deeper insight into the project status and
increases trust in the reports.
As a general principle, the MTS Operation phase
must be supported by an open dialogue between
customer and service provider relating to the
provision of the services throughout the term. In
order to keep up with technological changes over
the run-time of the overall agreement, details of
the agreement may need to be amended, whether
as a result of market trends, new technology,
evolution of new services, or otherwise, and must
be kept under constant review, control, and subject to proactive search for improvements in the
MTS Operation phase. To facilitate the dialogue,
both partner organisations typically appoint a Relationship Manager at senior management level
to cover all matters concerning the agreements.
The service provider appoints a Service Delivery
Manager as the point of contact for operational
matters affecting the services (e.g. adding new
applications to the portfolio; customer projects
ordering services).
Having dened the roles, a corresponding formal
and structured channel of escalation and communication has to be set up and maintained. The

Page 17

proven governance structure consists of the following decision bodies in a strategic and an operational layer:
STRATEGIC LAYER
Executive Steering Committee (ESC) Meetings are held quarterly (or as described in the
framework agreement) as senior management
meetings to monitor overall MTS performance,
to govern MTS from a long-term perspective,
and to de-escalate issues arising from the
operational layer.
Innovation & Technology Committee (ITC)
The ITC meets quarterly to share the latest
industry, innovation and technical trends, in
order to decide on major innovation projects
and jointly agree on business cases and funding, if required. Its members propose changes
to the agreements to the ESC and also consult
the ESC on strategic decision-making.

OPERATIONAL LAYER
Service Delivery Committee (SDC) The SDC
performs weekly reviews of the current performance and progress, focusing on the domain,
project and service levels. It decides on priorities, resolves escalation issues as best possible, and elaborates on how to mitigate risk
and issues within the scope of the services
operation. Issues outside the SDCs responsibility are referred to the ESC.
Operational Status Meetings These meetings
are scheduled weekly and aligned with core
testing processes in order to review progress,
focusing on the project or application level,
and to decide on priorities and work on risks
and issues, seeking for resolution wherever
possible or referring them to the SDC.

Managed Testing Services

4.2 MTS RISK MANAGEMENT

As with all other business activities, there are
risks associated with MTS. For risk, we use the
following denition, based on the concept of uncertainty:
Uncertainty is the lack of complete certainty,
that is, the existence of more than one possibility. [] Risk is a state of uncertainty where
some of the possibilities involve a loss, catastrophe, or other undesirable outcome.
Denition 4: Uncertainty and risk ('))

So each risk is an uncertainty but not vice versa.

Uncertainties can have both positive or negative
impacts, risks only have negative impacts on business. Please note that this understanding of risk
is more focused than the one used in ISO 31.000
since there even positive impacts are classied as
risk.
From a risk management perspective, the focus is
on negative risks because they require mitigation
actions to deal with their impacts.
For a customer, MTS involve two different types
of risks:
Ineffective Testing: Testing does not work
properly, i.e. the SLAs for the MTS or the strategic objectives to do MTS are not fullled.
Unfullled Collaterals: Testing works but other
collaterals are uncertain. This uncertainty can
cover the economic future of the service provider or might touch the cultural acceptance of
MTS within an organisation.

Both risks are rened in the following subsections.

Page 18

4.2.1 INEFFECTIVE TESTING

If testing is ineffective, the uncertainty about the
system to be rolled out increases because nobody
knows about possible hidden errors in production and potential impacts. Since risks are based
on uncertainties, the overall risk of an IT system
increases as well. This does not necessarily imply that the system fails in production but that
it lacks the necessary transparency and reliable
insights about production readiness. Therefore,
ineffective MTS for a business-critical application
might induce additional risk.
From an MTS perspective as described above, the
following countermeasures are to be taken:
Within the transition step, many prerequisites
for MTS are veried. The transition step will
identify most of the detailed hurdles for effective testing and eliminate the respective
reasons.
Before MTS enter the operational phase, the
collaborative delivery is an explicit mitigation
activity to ensure in a dry-run that testing can
be implemented as planned.
The MTS process incorporates the denition
of an overall control framework underpinned
by supporting KPIs to make the status of the
testing process transparent to all stakeholders. Therefore, uncertainties about the test
progress should be minimised.

Managed Testing Services

4.2.2 UNFULFILLED COLLATERALS

Outsourcing a well-dened service might stipulate uncertainty in some collateral areas. Most
of them are not solely related to MTS but to all
major changes in the context of service delivery.
In a study about risks of outsourcing (not limited
to MTS), a set of most common risks was identied by survey respondents. ('*) Most of them are
valid for MTS as well:
Pricing risks: In many cases, using MTS should
have a positive effect on costs. However, the
specic costs of MTS are not ultimately known
before the rst Service Operation step. The
MTS framework provides general guidance but
only the Collaborative Delivery will determine
the nal price. From a customer perspective, a mitigation action would be to select an
experienced MTS provider whose rst effort
estimation does not need much adjustment. If
there are price adjustments while the service
is delivered, the Service Completion step ensures that it is possible to change the service
provider to a cheaper one.
Planning risks: If some other constraints from
MTS are not met (e.g. missed deadlines, unclear planning), this might generate additional
risks as well. The MTS process guarantees to
identify deviations as early as possible. In most
cases, any deviations are followed by contractual penalties compensating negative effects
by planning risks.

Page 19

Communication risks: In general, MTS need

more mature communication channels (no
longer gossiping) when communicating
from customer to testing. In addition, a more
mature defect management is necessary since
the organisational boundaries discourage
on-the-oor communication. And last but not
least, cultural and language issues are a major
source of misunderstanding, in particular
if cross-national and offshore locations are
involved. MTS might generate risks in this
eld, but the industrialisation steps, that are
checked on the customers side in advance, as
well as the systematic transition step minimise
them.
Dependency risks: Outsourcing is often associated with outsourcing responsibilities, i.e. testing capabilities for MTS. The less knowledge is
retained on the customers side the more he
is dependent on MTS providers. Typical
examples of a high dependency are price
adjustments without any reason, disburdening
details of the negotiations, or low quality. In all
cases, MTS propose the well-dened Service
Completion step, ensuring that all customers can step back. Due to the detailed entry
criteria (industrialisation), the customer has a
very clear understanding of the dependency,
possible impact, and alternatives for delivery.

Managed Testing Services

Page 20

5 CONCLUSION AND OUTLOOK

Today, testing is established as an important discipline for risk management. For maximum efciency, the testing activities, the methods, the
practices, tools and skills should only be of interest to those who execute testing. From a business
perspective, testing should be seen as a black box
focusing testings output. So there is a great opportunity to outsource testing activities at the
rate of output-based pricing: Managed Testing
Services.
Applying MTS is the next step of IT industrialisation and needs a mature process landscape on
both the consumers side and the producers side.
Even if there is no need to apply MTS as a last
step on the industrialisation roadmap, the road
itself generates a considerable amount of added
value for improving testing efciency and effectiveness.

The market for MTS will increase dramatically

over the next few years. Even if both, full-service
providers and test service specialists, will try to
do business in that area, the test service specialists will dominate the market in a few years time
due to many advantages in their MTS delivery.
Their process to establish MTS on the consumers
side reects a well-dened current-state analysis
as well as a corresponding transition step for MTS
readiness. Based on a continuous improvement
loop, MTS execution itself can be managed effectively. A possible closing step at the end can
reect an applications end as well as the demand
to in-house all activities again.

Bibliographical References

Adam Smith. An Inquiry into the Nature and Causes of the Wealth of Nations. Oxford University Press, 2008.

International Software Testing Qualications Board, Glossary Working Party.

Standard Glossary of Terms Used in Software Testing. Homepage: ISTQB,
December 2007.

iqnite conference. iqnite interaktiv SMS voting. iqnite germany program.

[Online] 01/07/2011. [Cited: 28/07/2011.] http://www.iqnite-conferences.com/
Trial%20Wiki/iqnite_interaktiv_sms-voting.aspx.

Dr
DrWolfgang
WolfgangBalze,
Balze,Dr
DrWolfgang
WolfgangRebel
Rebeland
andPeter
PeterSchuck.
Schuck.Outsourcing
Outsourcingund
und
arbeitsrechtliche Restrukturierung von Unternehmen. Verlagsgruppe Hthig
Jehle Rehm, 2007.

Computer
ComputerEconomics.
Economics.ITITOutsourcing
OutsourcingStatistics
Statistics2010
2010/ /2011:
2011:Outsourcing
Outsourcing
and Offshoring Trends, Cost/Service Level Experiences, and Analysis for 11
Outsourced IT Functions. California, Irvine: Computer Economics, 2010.

Karsten
KarstenLeclerque.
Leclerque.Outsourcing
Outsourcingim
imJahr
Jahr2020.
2020.www.cio.de.
www.cio.de.[Online]
[Online]
09/09/2010. [Cited: 19/07/2011.] http://www.cio.de/knowledgecenter/outsourcing/2247905/index.html.

- Daryl C. Plummer, David Mitchell Smith, Thomas J. Bittman, David W. Cear - ley, David J. Cappuccio, Donna Scott, Rakesh Kumar and Bruce Robertson.
Five Rening Attributes of Public and Private Cloud Computing. Gartner,
May 2009.
. Praveen Ranjan Srivastava. Estimation of Software Testing Effor t:t:An
Anintelintelligent approach. Birla Institute of Technology and Science, Pilani, Rajasthan,
India: Computer Science and Information System Group, 2009.
/ Michael A. Cusamano and Richard W. Selby. Microsoft Secrets. New York:
Simon and Schuster, 1995.
'& PAC. Compliance and Software Testing. Compliance Magazine. [Online]
04/04/2007. [Cited: 19/07/2011.] http://www.compliancemagazin.de/markt/
studien/pac040407.html.
''

Wikipedia.
Wikipedia.Independent
Independenttest
testorganization.
organization.Wikipedia,
Wikipedia,the
thefree
freeencyclopedia.
encyclopedia.
[Online] 27/07/2011. [Cited: 27/07/2011.] http://en.wikipedia.org/wiki/Independent_test_organization.

Page 21

Bibliographical References

'(

NelsonHall Report. Independent Testing Providers vs. System Integrators.

Cologne:SQS,
SQS,April
April2010.
2010.
Cologne:

'3

Douglas Hubbard. The Failure of Risk Management: Why its broken and how
tox
xit.
it.John
JohnWiley
Wiley&&Sons,
Sons,2009.
2009.
to

'* Diamond Management & Technology Consultants. 2006 Global IT OutsourcingStudy.

Study.Chicago:
Chicago:Suite
Suite3000
3000John
JohnHancock
HancockCenter,
Center,2006.
2006.
cing

Page 22

SQS Software Quality Systems AG

Phone: +49 (0)2203 9154-0
Stollwerckstrasse 11
51149 Cologne / Germany
www.sqs.com