Testing As A Service

- Models
Discussion Paper

Jonathon Wright
June 2015
Is This Book Right For Me?

This eBook is intended for

ALL AUDIENCE LEVELS

INTRODUCTORY

Introductory content is for software testing professionals who are

relatively new to the subject matter of the ebook. Introductory
ebooks typically feature an overview of an aspect of testing or
guidance on understanding the fundamentals of the subject at hand.

INTERMEDIATE

Intermediate ebooks are for software testers who are already

familiar with the subject matter of the eBook but have limited hands-
on experience in this area. Intermediate level usually focuses on
working examples of a concept or relaying experiences of applying
the approach in a particular context.

ADVANCED

Advanced ebook content is for software testers who are, or intend

to become, experts on the subject matter. These ebooks look at
more advanced features of an aspect of software testing and help
the reader to develop a more complete understanding of the subject.
 The traditional focus of testing in many organisations has been on quality assurance
Abstract and return on investment over relatively long baselines. Testing as a Function
is increasingly being seen by many organisations as a costly, barely justifiable
overhead, which cannot make a sufficient impact on quality or return on investment
rapidly enough even within modern enterprise delivery environments. Perhaps an
alternative approach is long overdue?

The current economic climate is making companies review their approach to IT

even more closely. The same vision could also extend to IT services and particularly
testing. Existing technologies of service network virtualisation, business process
modelling, Cloud based test automation tools and rapid and easy internet access
allow for the delivery approaches that allow companies to consume Testing as a
Service and pay only for what they use. There is no need to spend large sums on test
environments, data masking, test tool selection and maintenance. The use of Cloud
based services mean you can select the right level of service at a time when you need
it and at the volumes you need it whether it is one or one thousand testers on one or
many environments, located locally or around the world.

Copyright 2015 Jonathon Wright

All rights reserved.

Biography Jonathon Wright
onathon Wright has over 15 years of international
automation experience with a number of global
organizations; including Deutsche Bank, Lehman
Brothers, Hitachi Consulting, Thomson Reuters,
Xerox, New Zealand Lotteries Commission, Unisys and
Siemens. Hes a serial blogger on Test Automation as a
Service (TaaaS.net). Hes on twitter at
@Jonathon_Wright

Jonathon also contributed to the best-selling book

Experiences of Test Automation: Case Studies of
Software Test Automation, Dorothy Graham &
Mark Fewster, and a number of eBooks on Testing
as a Service models (epistemic & systemic entropy),
Advanced UFT 12 for Test Engineers Cookbook
(testing-store.com) and API testing in the cloud (service
& network virtualisation).

He is the founder of Automation Development Services

(Automation.org.uk) as well as presenting at various
international testing conferences, such as Gartner
(London), STARWest (California), STAREast (Orlando),
Fusion (Sydney), ANZTB (Melbourne), EuroSTAR
(Gothenburg and Dublin), HP Discover (Las Vegas and
Barcelona), ADM (Prague and Istanbul), BCS SIGIST and
Unicom (London) further detail about Jonathon can be
found at www.linkedin.com/in/automation
 1. Why as a Service?
Table of contents
2. Testing as a Service (TaaS)
Business Lifecycle Management
Software as a Service
Solution Delivery Lifecycle integration (SDLCi)
Critical Service Components
o Compare Testing
o TestAdvisor
o Testing Alliance
o Testing Marketplace

3. Test Platform as a Service (TPaaS)

Testing Community
Agile Portfolio Management
Your Test Cloud
API Testing in the Cloud
QA in the Cloud
4. Test Automation as a Service
AMMi Assessment Model
Return on Effort
Open Source
Fragile Practices
wAgile
Design patterns
1. Why as a Service?
The testing landscape is changing, the traditional approach to providing business value through testing is
constantly being challenged. So businesses need to constantly re-examine the real value of testing
services as an integral part of their overall delivery capabilities.

The tendency has been to rely on complex hybrid resourcing models made up of internal resourcing
and/or external resourcing (near shore, mid shore and off shore), to strive for Testing Centres of
Excellence (TCOE).

The question is: are traditional Testing as a Function (TaaF) as part of the Software Development
Lifecycle (SDLC) model still valid? or whether Testing as an Activity (TaaA) needs to develop a new
Solution Delivery Lifecycle Integration (SDLCi) model supporting Business on a Page (BoaP) to
provide business visibility through powerful Model-Based Design (MBD)1 techniques?

Traditionally, with the Software Development Lifecycle (SLDC) (e.g. Waterfall) key Testing as a
Function (TaaF) resources were retained in-house, in order to benefit from their extensive domain
expertise and increased understanding of business requirements.

Modern approaches such as organisation-wide / enterprise agile (OEA) adopt more Testing as an
Activity (TaaA) approaches by utilising cross-functional teams that are re-useable, re-deployable and
have dynamic resource allocation. Such teams are no longer specific to purely Testing as a Function
(TaaF) but contribute to and integrate with other business deliverables as part of, say, the proposed
SLDC model.

This aligns with last years EuroSTAR2 (2013) topic from conference chair Michael Boltons theme of
Challenging Testing? This explored the fundamental question as to who should be providing testing
services?

Businesses should focus on business, and not try to be TCOE3.

Similar questions have also been raised as to whether businesses should be retaining highly skilled
testing professionals specialising in areas such as performance and security when that is not their core
business offering? (e.g. a large supermarket chain spending millions each year on building Testware
Automation Frameworks (TAF)).

Performance and security testing disciplines are good candidates for Testing as a
Service models that can be provided at a fixed usage or output-based price4.

At a recent Gartner5 presentation, I had discussions on this topic with Gartners researchers. It was
apparent that the increased focus on business delivery is not just about chasing the Hype Curve and
the Magic Quadrants promoting the maturity of Solution Integrators (SI) but also:

How to drive true delivery innovation powered by the Cloud?

How to gain Actionable Business Insight (ABI) through Big Data?
How to embrace next generation Digital Enterprise, Internet of Things or Quantum Computing?
How to understand the fundamental difference between Testing > Quality > Assurance?

1
Paul Gerrard, 2014, New Model for Testing discussion paper published, 6th August 2014
2
Jonathon Wright, 2013, Testing as a Service abstract for EuroSTAR, Gothenburg, Sweden, 13th Feb 2013
3
Jonathon Wright, 2011, Testing as a Service presented at BCS SIGIST, London, 5th December 2011
4
Sogeti, HP and Capgemini, 2014, World Quality Report (www.uk.sogeti.com/WQR2014-15) 2014-2015, UK, 16th October 2014
5
Jonathon Wright, 2014, Ha(API)y testing in the hybrid-Cloud & beyond presented at Gartner ADM, London, May 19th 2014
How these approaches can empower businesses to improve the understanding of their core business
through abstraction or encapsulation of business processes by adopting the Business on a Page
(BoaP) model.

Over the last 5 years I have been actively exploring Testing as a Service (TaaS) models. The key focus
has been on the core businesses differentiators such as agility, visibility and flexibility as achieved by:
Real adoption of Cloud platforms (Public, Private or Hybrid) and services (Infrastructure as a
Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS))
Improving the Time to Market through Continuous, Integration, Build & Delivery (CIBD),
Shift Left (Test-Driven Delivery) or Shift Right (Service Delivery i.e. DevOps), Shift Up
(Business Acceptance Testing) and Shift Down (User Acceptance Testing)
Providing dynamic Delivery Solutions (Products or Services) through pluggable Service
Oriented Architectures (SOA), Web Services and Application Programming Interfaces (API)
Managing Portfolios of these Services (API, SOA) over bespoke design of Applications for a
specific Customer or Client; no requirement6 for User Interface (UI) based apps in the future.
Focusing on the All-Channel Customer Experience4 through the Digital Enterprise journey for
Transformation or Convergence supported through Social Intelligence or Enterprise
Gamification, rather than building platform specific User Interfaces (UI) or native mobile apps.
Preparing for Testing in the Wild (TinW) with mobile devices7 exceeding the 7.2 billion people
on the planet and Machine to Machine (M2M) now in the excess of 250million (growing at a
faster rate than mobile) Near Field Communication (NFC) and Internet of Things (IoT).
Leveraging powerful Service / Network Virtualisation platforms that support Test First Delivery
(TFD) methodologies as the Solution delivery enabler not the delivery mechanism i.e. which
Technology or Platform
Migrating the impact & risk around Digital Disruption when it comes to Big Data and the
Netflix model for consuming Database as a Service (DBaaS) with real-time data masking
embracing moment in time (Small Data) to empower Business Intelligence (BI)
Integrate As A Service Solution Integrators (SI) that provide the end to end Business
Delivery process linked to quality deliverables directly enforceable Service Level Agreements
(SLA) with defined Business Acceptance Criteria (BAC), measurable by Key Performance
Indicators (KPI) as well as complying with the businesses appetite for Risk Mitigation /
Exposure

The above topics are too diverse to cover in detail here; so this discussion paper will focus on how to
integrate As A Service models within Testing as an Activity (TaaA) in order to add clear value to the
Business Delivery process through the following models:
Testing as a Service (TaaS) +
Test Platform as a Service (TPaaS) +
Test Infrastructure as a Service (TIaaS)
Test Automation as a Service (TAaaS) +
Functional Testing as a Service (FTaaS)
Performance Testing as Service (PTaaS)
Mobile Testing as a Service (MTaaS)
Security Testing as a Service (STaaS)

In particular, the topics shown with a plus sign will be directly challenging the current industry perception
of what the As A Service models mean in terms of unlocking truly consumable testing services aided
by the creation of a Global Testing Marketplace (GTM).

6
James Whittaker, 2014, Beyond the Web and Apps: The Domestication of Knowledge Agile Development Conference West, Las Vegas, 1st June 2014
7
GSMA Intelligence, 2014 http://www.cnet.com/news/there-are-now-more-gadgets-on-earth-than-people/, CNET, 6th October 2014
 What is the future of testing? The possibilities associated with Cloud computing provide instant
2. Testing as a Service
scalability, flexibility and availability for testing on demand with no upfront investment. This provides the
industry with a perfect opportunity to develop powerful testing delivery solutions.

My keynote at the BCS SIGIST 5 (2011) on Testing as a Service (TaaS) explored the core principles of
this model:

Support for the Global Testing Marketplace (not just regional hubs)
o Instant Scalability, Flexibility and Availability
o Improved Communication, Collaboration and Mobility
Models

Support end to end Business Lifecycle Management (BLM)

o Portfolio Lifecycle Management (PLM)
o Solution Lifecycle Management (SLM)
o Application Lifecycle Management (ALM)
Business Delivery Management integration (BDMi)
o Software Development Lifecycle (SDLC)
Waterfall
o Solution Delivery Lifecycle integration (SDLCi)
Organisation-wide / enterprise agile (OEA)
Scaled Agile Framework (SAFe)
Test First Delivery approaches (i.e. ATDD, BDD, TDD & TDDi)
Business Process Mapping
o Business Process Modelling (BPMn v2.2)
o Business Process Intelligence (Actionable Business Insight)
o Business Process Design (Business Rules)
o Business Process Validation (Business Validation)
o Business Process Scenarios (Business Workflows)
o Business Process Data (Business Sanitised Data)
o Business Process Components (Business Transactions)
o Business Process Testing (Jigsaw Pieces)
Cloud-based test environments
o Test Platform as a Service (TPaaS) (on-demand)
o Test Infrastructure as a Service (TIaaS) (on-demand)
Cloud-based test resources
o Social Enterprise Collaboration (SEC) ready (portability/accessibility)
o Flexible/scalable access to global resources pools (on-demand)
Cloud-based test assets (Test Definition Layer)
o Test Specific Languages (TSL)
o Business Specific Languages (BSL)
o Validated against Business Process Rules/Workflows/Data
Consumer/customer freedom
o Pluggable service delivery layers (Data In > Data Out (DIDO))

Business Lifecycle Management

The traditional approach to Application Lifecycle Management (ALM) (referring to a single application
instance) is no longer relevant as Solution Lifecycle Management (SLM) relates to the logical
groupings of applications that make up the solution landscape. This in turn has now been supplemented
by Portfolio Lifecycle Management (PLM) in which multiple solutions are represented by logical
groupings as part of a Business Portfolio or split by Business Domain or Workstream. Business
Lifecycle Management (BLM) encapsulates all previous delivery lifecycles whilst providing support for
Solution Delivery Lifecycle integration (SDLCi) across a number of different businesses (i.e.
manufacturing through to distribution).

Testing has often been considered to represent only a small part of the overall Software Development
Lifecycle (SDLC) along with analysis, development and deployment. Historically, there has been a
tendency for each function to blame the other and adopt a throw it over the fence approach to
Application Delivery Management (ADM). This has resulted in silos of As A Function delivery
functions which are highly inefficient and also the creation of Domain Tools and Delivery Resources
specific only to that function.

The concept behind Shift Left is the ability to start Testing as an Activity (TaaA) as far back as the
origin of the Business Definition (Problem, Idea or Challenge), Shift Up enables the continuous
visibility to Business Stakeholders (CIO / COO / CTO) throughout the Business Lifecycle
Management (BLM) from Problem Definition through to Solution Decommission. However, Shift
Left focuses on such activities as Analysis, Architecture and Development with greater emphasis
(and effectiveness) than the Shift Right activities from Go Live through to Decommission. and Shift
Down is the constant customer feedback loop (i.e. targeted end-users).

Thankfully the industry has already started to embrace these changes under such As A Activity
headings as Developer in Test (Shift Left) and DevOps (Shift Right) (The latter should not be
confused with the testing industry embracing Test in Dev or TestOps activities).
The model below, presented at BCS SIGIST 3 (2011), shows the logical split between the Business
Layer and the Service Layer and how the Shift Left and Shift Right is the cross-communication
between domains through domain standards within each delivery phase.

Domain Standards such as CMMI, UML and ITIL are industry standards that wrap process around the
Communication layer between such delivery phases. The Language layer used has to be
unambiguous (e.g. Lojban which eliminates ambiguity unlike English which is great for poetry but
ambiguous when used for requirement definition) so must be in predicate logic (as used in maths and
physics (e.g. quantum mechanics or even quantum teleportation).

Domain Tools support the Solution Implementation and help provide integration throughout each
Delivery Implementation phase. For example Business Stories should not only be visible within
Business Story Method (BSM) domain tools but also within the Service Management (SM) domain
tools that are used within functions such as service delivery or production support. This enables all
Delivery Resources pools to operate within the same teams (as in SCRUM teams).

Businesses should focus on their core business and maintain competitive

advantage within their marketplace and not strive to be the best in the industry at
testing competencies. 3

The Business Solution Delivery Quality will always be driven by cost, benefit, time to market (cheap,
good, fast - the eternal conundrum8 or the extended Delivery Pentagon cost, benefit, time, risk,
quality) which will determine the overall quality of the delivered product or service. However, when we
explore LEAN processes (such as Toyotas factory models wherein the objectives are waste reduction
and to continuously strive to improve quality) we find it is in stark contrast in many of the practices within
the IT sector. For example, developing multiple similar products and services (such as foreign exchange
market (FX) platforms for each geo-based region) encourages waste by duplication of time and effort.
Another example is the lack of enforceable quality, not everyone in the organisation can pull the stop
cord at any time if they believe a product or service may be at jeopardy. Only layers upon layers of
middle management instead of everyone in the SDLCi being responsible for quality.

How does this contrast to the IT as a Service (ITaaS) approach? After all, the core principle behind say
Cloud is to reduce the waste between the Logical / Physical service layers. For example:

Hardware as a Service (HaaS) +

8
Chris Amber, 2014, Good, Fast, Cheap The Eternal Conundrum presented at the TMF, London, 29th October 2014
 Infrastructure as a Service (IaaS) +
Platform as a Service (PaaS) +
Software as a Service (SaaS) +
Testing as a Service (TaaS):
o Test Infrastructure as a Service (TIaaS)
o Test Platform as a Service (TPaaS)
o Test Automation as a Service (TAaaS)
o Functional Testing as a Service (FTaaS)
o Performance Testing as a Service (PTaaS)
o Mobile Testing as a Service (MTaaS)
o Security Testing as a Service (STaaS)

Real world examples of the models shown with a plus sign are covered in detail in the full eBook version
that can be downloaded from www.leanpub.com/taas within the appendix section on Testing as a
Service (TaaS). These case studies give context to the complete end to end flow between logical /
physical server layers.

Software as a Service (SaaS)

Consider the SalesForce.com model which is the benchmark success story for the Cloud. It provides
Software as a Service (SaaS) solutions to businesses to enable them to build their business platform
around pluggable/interchangeable products and services. This begs the question: even if the
SalesForce.com are providing Business Platform as a Service (BPaaS), who is the best person to
test the target end solution?

Traditionally, businesses are involved not only with the initial business requirements and design phases
(i.e. Business Process Validation (BPV) of the target business processes/models) but also during
testing cycles such as User Acceptance Testing (UAT) but not always across the entire Solution
Delivery Lifecycle (SDLC) (development, deployment and support phases).

Dont assume the ecosystems (commercial or community) and other fulfilment

systems to be tested in anything but isolation. - Julie Gardiner 9 (2012)

Business Acceptance Testing (BAT) is essentially always going to be a business activity (Shift Up) as
they best placed to wrap Business Acceptance Criteria (BAC) around the Solution Delivery as they
best understand the Business Domain. However, the introduction of complex Cloud ecosystems such
as Public, Private, Hybrid or Community presents new challenges and opportunities for the
connectivity and integration points for a number of logical / physical service layers:

Software as a Service (SaaS)

Hardware as a Service (HaaS)
Platform as a Service (PaaS)
Infrastructure as a Service (IaaS)

The above topics are the reason why traditional enterprise businesses be starting to think about hiring
hardware engineers as well as software engineers given that their target product or service may run on
All-Channel Customer Experience platforms such as the Internet of Things (IOT) or Bring Your Own
Device (BYOD) that rely heavily on physical hardware such as System on Chip (SoC) or Near Field
Communications (NFC) components requiring Testing in the Wild (TinW) using Test Drones (TD)?

9
Julie Gardiner, 2012, STARWest, California, USA, 4th October 2012
Solution Delivery Lifecycle integration (SDLCi)

The answer instead is we need to strive for Solution Delivery Lifecycle integration (SDLCi) with
Solution Integrators (SI) who can provide Testing as a Service (TaaS) around each of the logical /
physical service layers. The major advantage would be that the quality gate points providing assurance
to the business by reducing the risk of throwing it over the fence by defined transparent Business
Impact or Risk10 analysis around any changes to any software (logical) or hardware (physical) layers
which affect the overall Delivery Solution. Compare this approach to the use of legacy release notes
which documented new features and fixes in terms of minor bug fixes for the Application Under Test
(AUT) or hardware release notes such as ZQ calibration of the System on Chip (SoC). These provided
no information about the Business Impact or Risk11 whereas if you directly integrate the Solution
Integrators (SI) target product or service into your Continuous, Integration, Build and Delivery
(CIBD) platform then you can assess the Test Impact Analysis (TIA) of any changes/fixes to
functionality.

Critical Service Components

Clearly there are a number of critical components which are of fundamental importance to the effective
provision of any service. One of these is having the right people in the right place at the right time i.e.
Just in Time (JiT) delivery. The challenges/opportunities of this component will now be discussed in the
context of the Solution Delivery Lifecycle integration (SDLCi) model.

In terms of locating the right people, assessing their capabilities and organising them, a number of
options are available:

COMPARETESTING.COM

This is very different to the current traditional testing supplier model provided by Solution Integrators
(SI) who are not always up to speed with the rate of change within the industry12 (as evidenced by the
Digital Enterprise Convergence (DEC), All-Channel Customer Experience, Big Data (BD), Internet
of Things (IOT), 3D Bioprinting, Quantum Computing (QC) and Artificial Intelligence (AI)).

For a long time, Solution Integrators (SI) have wrapped managed services around testing resourcing
models e.g. body shops. But in a highly competitive testing market, their added business value is
becoming less apparent - especially when they try to undercut other Solution Integrators (SI) by using
blended rates for resourcing pools without necessarily having the range of skill sets needed to follow
through on the promised deliverables.

10
RBS, 2014, RBS fined 56m over 2012 computer glitch, http://www.bbc.co.uk/news/business-30125728, London, 20th November 2014
11
RBS, 2014, RBS fined 56m over 2012 computer glitch, http://www.bbc.co.uk/news/business-30125728, London, 20th November 2014
12
Gartner, 2014, Building your Digital Enterprise Gartner Event, Poland, 15th May 2014
Similarly, external recruitment agencies operating as Virtual Testing Specialists or even Virtual Test
Consultancies, often barely justify their daily mark-up rate (around 1/5th of the daily resourcing costs
models in the UK) to place appropriately skilled short term resourcing pools. The need for a comparison
platform for independent testing suppliers or vendors as mentioned at this years13 at STARWest keynote.

TESTADVISOR.NET

Social enterprise platforms such as LinkedIn allow direct access to skilled individuals without involving
third parties or preferred testing suppliers. The latter all too frequently do little to ensure the quality of the
resources they provide. They generally lack enforceable Service Level Agreements (SLA) or have
failing service & support models. Consequently they cannot always justify their inflated daily rates
notwithstanding the investment needed to establish the relationship with the business or to gain
acceptance on preferred supplier lists / vendor supplier frameworks.

On the other hand, a key advantage of the Testing as a Service (TaaS) model is the ability to provide
global dynamic resource allocation through a Global Testing Marketplace (GTM) rather than relying on
regional supplier hubs. This model is based on common access to global resource pools which act in the
style of TripAdvisor (called TestAdvisor) for rating potential resources based on instant feedback
from clients on delivered performance - rather than trying to put square pegs into round holes. But what
skill sets and what metrics apply in this context?

Many of the best testers are technically minded and can program but a testers
real skill is applying testing knowledge to generate reusable test assets. 3

TESTING-ALLIANCE.COM

At Fusion14 (2012) Elisabeth Hendrickson referred to resources being like a broken comb reflecting
individual strengths and weakness in a DNA-like combined skill matrix for the target delivery unit (or
scrum team). Given the duration of the fixed, short term work units (in minutes, hours, days, weeks) the
blended resource allocation is designed to complement each resources proven experience and skills
sets to provide the agility to deliver the target solution.

In turn does this mean we are no longer looking just for Testing as a Function (TaaF) resources (i.e.
anyone that describes themselves as a tester or developer) or Testing as an Activity (TaaA) resources
(i.e. the perfect agile resource profile of the jack of all trades, master of none). But instead no-one that
has pigeonholing themselves into a category or label but instead who understand universal Languages
such as maths and physics with experience of Quantum Mechanics (QM) theoretic terms to express the

13
Julie Gardiner, 2014, Avoiding Vendor Driven Delivery keynote presentation at STARWest, California, USA, 15th October 2014
14
Elisabeth Hendrickson, 2012, Acceptance Test Driven Development at Fusion, Christchurch, New Zealand, 10th September 2012
information theoretic concepts () = (()) 15 required in Solution Delivery Lifecycle
integration (SDLCi).

People who can also apply theories like Quantum Teleportation16 (QT) to solve the hard maths required
for Business on a Page (BoaP) or even may have hands on experience using Quantum Computing
(QC) devices such as the D-Wave's17 Quantum Annealing (QA) to business domain challenges like
quantifiable real options.

TESTING-MARKET.COM

A persons contribution to a project can also be tracked in the same way that other Social Collaboration
platforms have been doing for decades (such as Xbox Live / PSN Gamer IDs, crowdsourcing platforms
such as uTest18 below or the above Game Dev Tycoon). These score an individuals skills in specific
areas to produce an accurate resource skills matrix together with quantitative measures of their expected
delivery capabilities.

15
Huw Price & Llyr Jones, 2014, Grid-Tools.com, https://twitter.com/GridTools/status/487264150578790400, 10th July 2014
16
Flix Bussires, Nature Photonics http://www.nature.com/nphoton/journal/v8/n10/full/nphoton.2014.215.html, 21st September 2014
17
T. Lanting, Entanglement in QA, http://journals.aps.org/prx/abstract/10.1103/PhysRevX.4.021041, 29th May 2014
18
uTest, http://blog.utest.com/2014/11/04/utest-platform-preview-new-dashboard-for-testers-on-paid-projects, 4th November 2014
This could be a game changer for the IT sector, as poorly skilled resource pools or lack of delivery focus
(due to being billed to multiple clients or projects) would be visible to the client. Consequently near-
shore/off-shore resources with a low blended rate but a low delivery capability would have to stand
comparison with multi-shore/on-shore, high delivery capability resources albeit at a higher rate but with
the capability of being flexibly redeployed onto other work streams.

The Global Testing Marketplace (GTM) operates across all time zones providing 24/7 access to the
Testing Delivery Services platform in the Cloud (Public, Private, Hybrid or Community) allowing
Solution Integrators (SI) to provide the right capabilities at the right place and at the right time through
the As A Service models and provides:

Improved Communication, Collaboration and Mobility to both the customer by providing

Business Delivery Management (BDM) through Solution Command Centres (SCC) as part of
the Solution Delivery Lifecycle integration (SDLCi) process.
Target resourcing pools with the Test Platform as a Service (TPaaS) and access to the Test
Infrastructure as a Service (TIaaS) required to access the Solution Under Test (SUT) and
report back to the Testing Command Centre (TCC).

In summary, these models offer Instant Scalability, Flexibility and Availability of consumable Testing
Delivery Services on a Pay as you Use (PAYU) basis. The next chapter will explain how to leverage
Test Platforms as a Service (TPaaS) as the Social Enterprise Collaboration (SEC) platform to create
the foundations for the Testing as a Service (TaaS) model.
3. Test Platform as a Service Practically every area of the commercial software industry is being continuously revolutionised by new
concepts and technologies.

Weve all heard the claims that Cloud computing will, without any up-front
investment, provide instant scalability, flexibility, and availability for testing-on-
demand. - But how well does this work in practice? 19

TESTING-COMMUNITY.COM

Following on from the previous chapter, there is a need for a global Social Enterprise Collaboration
(SEC) platform to enable resources to collaborate with other specialists to:

Produce high-quality reusable business assets together

Capture specialist technical and cross-domain knowledge that is no longer limited to a single
Application Lifecycle Management (ALM) instance but spans the entire Business Lifecycle
Management (BLM) space from Business Requirements Definition through to Decommission
whilst encompassing Analysis, Architectural, Design and Implementation compliance
safeguards
Produce design patterns that allow business rules to be codified in business-specific meta-
languages.

Existing technologies of virtualisation, business on a page . that allow

companies to order testing as a service and pay only for what they use (PAYU). 20

This aligns with my EuroSTAR21 (2014) topic and is also in alignment with conference chair Paul
Gerrards theme on Diversity and how leveraging Social Enterprise Collaboration (SEC) as part of
providing a Business Lifecycle Management (BLM) platform by enhancing business agility by
unlocking the full potential of the Cloud to drive innovation through instant scalability, collaboration and
accessibility:

How to build a Business Lifecycle Management (BLM) platform that works for you (and the
importance of becoming an effective Data Scientist (DS) and Data Ambassador (DA) by
leveraging Social Enterprise Collaboration (SEC) through Business Process Modelling
(BPM) / Model Based Design (MBD)
How to build an Testing Platforms in the Cloud (by embracing Test Platform as a Service
(TPaaS) and Test Infrastructure as a Service (TIaaS) models whilst protecting valuable
Business Assets / Artefacts through the introduction of the Business / Test Abstraction layers
combined with minimum testing standards (such as ISO 29119)
Implementation of Social Enterprise Collaboration (SEC) using industry standards by
introducing BPMn v2.2 (Business Process Modelling notation) and xPDL
Effectively communicating Actionable Business Insight (ABI) to all business echelons through
Funnel Virtualisation linked up as far as the Business on a Page (BoaP) model
Workflow management/solutions/platforms utilising XAML (eXtensible Application Mark-up)
Addressing challenges faced with Agile Portfolio Management (APM) across All-Channel
Customer Experience Platforms such as bring your own devices (BYOD) or mobile devices.

19
Jonathon Wright, 2012, Test Automation as a Service presented at STARWest, California, 4th October 2012
20
Jonathon Wright, 2014, Solution Lifecycle Management in the Cloud presented at Unicom, London, 27th February 2014
21
Jonathon Wright, 2014, Test Innovation in the Cloud abstract for EuroSTAR, Dublin, Ireland, 20th February 2014
The traditional enterprise, workforce and or processes are rapidly changing and with the introduction of
Big Data we are all becoming Data Scientists (DS) and Data Ambassadors (DA). Directly impacting
the Quality of Data (QoD) and driving the need for Actionable Business Insight (ABI) to be available
anywhere on Bring Your Own Device (BYOD) to enable productivity and the speed of change.

AGILE-PORTFOLIO.COM

As discussed at Unicom22 (2014) around Agile Portfolio Management (APM) is the effective
information exchange for the communication horizontally (i.e. Solution Delivery Lifecycle integration
(SDLCi)) and vertically (upstream executives level and downstream solutions, projects or applications).
Therefore, Organisational-wide / Enterprise Agile (OEA) environments are highly dependent on
Business Process Intelligence (BPI) techniques (i.e. Funnel Virtualisation) to provide real-time
Actionable Business Insight (ABI) through23:

Delivering business value, not just driving process efficiency

Integrating the SDLCi with cross-functional Domain Tools
Extending lifecycles to include delivery (ALM/SLM/PLM/BLM)
Supporting individual, team, organisational and enterprise collaboration
o Integration with the Global Testing Marketplace (GTM) platform
o Produce high-quality reusable business assets in the cloud
o Capture specialist technical and cross-domain knowledge (i.e. DIDO)
o Game mechanics to encourage collaboration (i.e. Enterprise Gamification (E.G))
o Produce design patterns that allow business rules to be codified in
business-specific meta-languages (i.e. BPMNv2.2, xPDL, XAML)
Using reporting and dashboards to provide instant Actionable Business Insight (ABI)
Managing software that gets deployed everywhere (All-Channel Customer Experience)

Support for the entire cross-delivery phases requires a new breed of intelligent Agile Portfolio Management
(APM) platform that can include both Change Management (CM) and Service Management (SM) platforms (see
Portfolio Lifecycle Management model23 below).

22
Jonathon Wright, 2014, Agile Portfolio Management in the Cloud presented at Unicom, London, 20th March 2014
23
Forrester, 2012, The Forrester Wave Application Lifecycle Management, Q4 2012
YourTestCloud.com

The current understanding around Test Platform as a Service (TPaaS) actually relates to Test
Infrastructure as a Service (TIaaS) that is the foundation on providing the test infrastructure to create
and manage test environments. However, the fundamental differentiator is providing the Social
Enterprise Collaboration (SEC) platform component to support distributed agile resources Enterprise
Crowdsourcing (E.C) pools by providing access to the necessary domain tools.

The ability to provide Test Infrastructure as a Service (TIaaS) is still extremely important especially
when powered by the Cloud - as discussed at Unicom24 (2011). This is not a simple lift and shift
migration from the current Delivery Platforms or Solution Under Test (SUT) to the cloud ecosystem.

However, the migration from traditional test environment / lab setup to Test Infrastructure as a Service
(TIaaS) lays the foundations for:

Flexible Test Services:

o Testing as a Service (TaaS)
o Test Automation as a Service (TAaaS)
o Functional Testing as a Services (FTaaS)
o Performance Testing as a Service (PTaaS)
o Mobile Testing as a Service (MTaaS)
o Security Testing as a Service (STaaS)
Environments On-demand:
o Live pause/rewind/playback
o Repeatable/Actionable Defects
o Geo-based Performance Testing as a Service
o Geo-based Mobile Testing as a Service
Instant Scalability
o Flexible Test Resourcing
o Flexible Test Assets
o Flexible Test Execution
Instant Accessibility
o Social Enterprise Ready
Instant Portability
o Domain tools on-demands
No upfront investment

The concept that all cloud services are pay as you use (PAYU) models means delivery cost is easier to
manage also the no upfront investment means that you can directly manage the direct business value of
what is being delivered.

Do not invest heavily upfront in something that may fail. 3

Most importantly, you only pay for services that are consumed so you have the flexibility to turn
expensive test environments / labs on or off when they are not been utilised through support for
Persistence Test Data (PTD) (i.e. offline storage).

24
Jonathon Wright, 2011, Test Automation as a Service presented at Unicom, London, 18th May 2011
A further example the added value of Test Platform as a Service (TPaaS) is the Solution Integrators
(SI) ability to highly extendable SOA / API architectures the functionality (i.e. add the special sauce) to
customise the Testing as a Service (TaaS) offerings. The Test Infrastructure as a Service (TIaaS)
can also be managed through the Monitoring / Operation Management portals / dashboards to enable
Solution Integrators (SI) to improve the underlining Delivery Platform.

TESTABOVE.COM

A much overlooked and critical element of the modern day Cloud ecosystems is the interlocking web of
dependencies often hidden and unknown to its varied clients and customers. For example, Google
location services, or the Amazon eCommerce and fulfilment infrastructures are present in a myriad of
services often at a level far removed from the core functionality. These services individually can be
provided within the Test Infrastructure as a Service (TIaaS) but at what additional cost?

Is the cloud a match made in heaven? or new challenges & opportunities? 24

The modern Cloud aware, Service Oriented Architectures (SOA) are also uniquely vulnerable to
interlocking failure modes, caused by simple outages of services not perceived to be core or even
present. The concept of public Cloud computing has added an extra level of complexity to the Solution
Under Test (SUT) ecosystem without solving any of the existing infrastructure challenges. Even normal
operations of an interlocking web services can be perceived by an end user or client as an error.

What happens when these service FAIL?

Alternative service providers?

Unique Services (such as GPS)
Systemic Failure (DNS or Network)
API Testing in the Cloud

As discussed at Gartner 5 (2014) the introduction of Service/Network Virtualisation in the cloud is the
Shift Left (upstream) Test First Delivery (TFD) approach for testing complex cloud ecosystems at the
messaging and database layers.

The Shift Left, Look Right25 approach to seek out reusability throughout the Solution Delivery
Lifecycle integration (SDLCi) is the challenges around the management of multiple Cloud endpoints
that span numerous Test Infrastructure as a Service (TIaaS) providers.

Support for powerful Test Automation as a Service (TAaaS) platforms

o UI Testing that is Scalable and Cost-Efficient
o API Testing that is Repeatable, Reliable and Fast
Heterogeneous Test Infrastructure as a Service (TIaaS) that are open, flexible and extensible
Continuous Build, Integration & Delivery (CBID) in YourTestCloud
Use Cloud Maps to Learn, Discover and Model endpoint(s)
Create Bridges (Vnet to Vnet) If you build it, they will test (IYBITWT)
Enable Day-zero Performance, Penetration & Security Testing as a Service
Continuous Build, Integration & Delivery in YourTestCloud
Embrace Community Test Clouds (Collaborate/Share Recipes)

QACloud.net

This requires an intelligent cloud deployment platform that can build complex ecosystems by simplified
IFTTT recipes:

The idea behind this approach which has already been utilised within the Internet of Things (IOT) to
connect things that are complicated together provides a collaboration platform that support sharing
recipes like the 15 million recipes used every day by IFTTT26 collaborators.

Basic IFTTT recipe to create a cloud environment for a solution under test:

If <Solution Under Test> Then <Build Cloud 2.0>

Advanced IFTTT recipe to capture physical endpoints of the solution under test:

If <solution under test> exists Then <Learn Physical Endpoints>

If that fails then the platform utilises Service/Network Virtualisation to mock/stub out endpoints:

Else If <solution under test> does not exist Then <Define Virtual Endpoints>

The concept behind all the model patterns in this chapter is targeting a LEAN process models, reducing
waste through lack of the complete Solution Delivery Lifecycle integration (SDLCi) to enable the
Continuous, Integration, Build and Delivery (CIBD) solution.

The next chapter covers the historical challenges around proving the business value of the Test
Automation within the industry and how As A Service has changed the dynamic for implementation &
delivery agnostic solutions providing transparent business value.
25
Niell McCarthy, 2014, Shift Left, Look Right presented at HP, London, 1st October 2014
26
2011, IFTTT, https://iftt.com, 1st September 2011
4. Test Automation as a Service
Traditional approach to test automation is destined to become less and less relevant and cost-effective,
whilst lagging behind changes in technology. We will then outline alternative model patterns that focus
on emerging Delivery Solutions.

Estimates vary as to the value of the global testing domain tools market, a study conducted by Ovum27
projects the market to be worth an estimated 56 billion US dollars in 2013 growing at an annual rate of
9.5% faster than most other (information technology services). This estimated 56 billion US dollar testing
services market of which the test automation industry represents an estimated 6.3 billion dollars costing
the industry an estimated 3.8 billion in failed automation projects.

32% Failure
60% 40% Working
ROI
8%

The majority of in-house automation projects fail to deliver any measurable benefit. Figures released by
MI indicated that over 60% of automation projects fail straight off and out of the remaining 40% only
20% actually meet any return on investment. Resulting in 92% of implementations fail to meet target ROI
mainly due to resourcing and/or process failures. Successful automation over time will save money failed
projects are inevitably expensive.

AMMi Assessment Model

This large failure rate was typically down to the lack of defined automation methodology (such as AMMi
assessment model or no proven automation strategy). Leading to automation not been treated as a
legitimate project with the necessary associated planning and resources.

55% of enterprises in 2014 had challenges around the inability to apply test
automation at appropriate levels 4

Adopting a methodology such as the AMMi Assessment Model will provide you with the current
automation maturity level and the target maturity level to work towards:

AMMi Phase Target Criteria

Level 0 Accelerating Automated Test Lifecycle Methodology (ATLM)

Level 1 Traditional Framework Driven (Gen 1-5)

27
Ovum, 2009, Source: http://www.ovum.com/ (Internet News Article: http://phys.org/news155992141.html), 11th March 2009
 Processes are planned, performed, measured, and
Level 2 Managed
controlled
Automation process defined and validated against
Level 3 Sustaining
international standards (ISO-29119)

Level 4 Quantified Predictability of automation process performance

Automation process variation and statistical

Level 5 Optimising
predictability

However, the model above is rarely adopted within the industry and tends to apply the same approach
and level of maturity to automated checking for each approach which is primary failure as no one
solution, approach or methodology to rule them all.

Sometimes half the battle just starting companies on their automation journey by
taking them one step closer to becoming ready for automation. 28 AMMI

Return on Investment (ROI) vs. Return on Effort (ROE)

It is easy to imagine that a firm who spends thousands of pounds on Automated Checking that could
deliver a product and have got absolutely zero value out of the activity. Companies often buy in
expensive tools that they have no idea how to use within their process and are sold a potential silver
bullet that in fact is a dead weight.

The Cloud helps reduce ownership & maintenance costs, whilst moving away
from expensive fixed-seat licensing models and un-utilised maintenance
agreements. 19

Equally analysis of what can be automated is time consuming i.e. spending analysis time evaluating
what is possible to automate and what is not is time consuming. This time is entirely wasted if the
automation never happens.

Traditionally, automation initial investment costs are high - wouldnt it be great if

they could be lowered? 19

Well managed use of these resources would obviously deliver more value, but this needs better process.
Highly skilled testers with a process-driven toolset will deliver more value. Furthermore, a tool that can
make use of the same test resources for a variety of testing tasks such as performance and penetration
testing will save money as opposed to separately developing them.

No such thing as a free meal

Whether something is expensive depends on the value you derive from it. Spending 1 on a tool that
delivers no or negative value is expensive. Within testing the total cost of ownership of tools is high
compared to other enterprise level software. This usually drives companies to target tooling that are
supposedly free such as open source tools e.g. Selenium Webdriver, Cucumber or Watin.
28
Jonathon Wright (Founder), 2004, AMMi Assessment Model, www.AMMi.org.uk, 26th January 2004
However, lower maturity tooling that may have been labelled free are usually based on the costly and
labour intensive due to the capability of the tool and the associated maturity level. Not to mention
continual script maintenance by skilled personnel can become an expensive and error prone task.

Being able to more accurately see the clear value to cost figures upfront would
be a real advantage. 22

Proficiency and certification training is often seen as essential for test personnel. In reflection Developers
are usually recruited for the skills they have and can demonstrate, training is not usually offered or
considered. Within test, a policy of on the job training and certification is more accepted.

The standard approach from the test community (both vendors and practitioners) and their involvement
in the Software Development Lifecycle (SDLC) has not changed much in past decade. The training
materials, qualifications and domain tools available on the market still reflect this out-dated view.

FrAgile Practices

Often - and almost exclusively, Automated Checking function becomes involved in the process at the
end of the cycle, this is partially due by design for domain tool vendors not embracing the Agile
development methods such as ATDD, BDD and TDD which are all now seen as the antidote to all
challenges of Automated Checking?

However, the real pitfall is focusing on the traditional approach to Automated Checking i.e. through the
User Interface (UI) when the logical evolution to Automated Checking is through the service layer such
as API testing that supersedes the previous test industry focus on building elaborate Testware
Automated Frameworks that are both ineffective and based on frAgile (UI) practices for example
complex techniques to recognise object models:

Optical Character Recognition (OCR) / Image Recognition

Object Models (i.e. DOM)
XPATH
Regular Expressions
Descriptive Programming (DP)
Fuzzy Logic
Social Intelligence
Artificial Intelligence (AI)

Due to this businesses still base their processes around the Software Development Lifecycle (SDLC)
waterfall model waiting for the frAgile (UI) to be developed and often end up relying on manual methods
more heavily because the near-complete UI solution delivery late in the development cycle. This means
that automating the regression testing of the Solution Under Test (SUT) becomes a monolithic task that
runs in parallel with the real business of manual functional testing and therefore is always a second
priority.

wAgile.net

The agile methodology also emphasises on the immediate use of test automation asset reuse for unit
tests, acceptance tests, performance tests, security tests and UI tests, right at the start of the
development cycle? It also promotes the idea that that team members should be multi-skilled and able to
do multiple tasks rather than stick exclusively to one role.
These factors have the effect that the modern resources should be more technical and therefore more
capable of making the most of complex automation domain tools. In many industries, the trend is
towards having a low quantity of highly skilled labour rather than a high quantity of low-skilled labour.
Testing is no different but should it be?

wAgile promotes resources to be a Jack of all trades, Jedi master of none.

Agile also demands that testing begins early, sometimes before a line of code has been written. Testing
can begin as soon as the first high-level stories has been written by identifying ambiguity and
contradiction within that Business Level Story (BLK) i.e. Epic Theme(s). Resources should also assist
in formalising Business Acceptance Criteria (BAC) for those stories. Agile domain tools should allow
this to happen also with automation domain tools. Resources need the ability to start coding their
automated tests without having the software available. As with agile in general, we want to leave the final
decision for the Delivery or Solution Implementation until it is really needed. Ideally we want to start
with the Business Level Questions (BLQ) and link the lower-level Business User Stories (BUS) to
them both with Business Acceptance Criteria (BAC) then create all the Business Process Tests
(BPT) before a single line of code has even been written.

Design Patterns

To make this possible the follow target design patterns should be used within either Test Automation as
a Service (TAaaS) or Functional Testing as Service (FTaaS) with the foundations provided by the
Test Platform as a Service (TPaaS) throughout the Solution Delivery Lifecycle integration (SDLCi).

This is demonstrated within the full eBook version www.leanpub.com/taas around Test Automation as a
Service (TaaaS.net) providing additional detail for the following advanced design patterns:

Business Abstraction Layer (BAL)

o Business Definition Language (BDL)
o Model Based Design (MBD)
Business Process Behaviours (BPB)
Business Acceptance Criteria (BAC)
Test Abstraction Layer (TAL)
o Test Definition Language (TDL)
Self-validating
Self-maintaining
Truly human readable
Implementation Agnostic (i.e. Logistics)
o Dynamic Solution Adapters
o Dynamic Delivery Adapters
Support for Solution Delivery Lifecycle integration (SDLCi)
o Test as a Service (TaaS)
Test First Delivery (TFD)
o Test Platform as a Service (TPaaS)
Legacy Testing
o Test Infrastructure as a Service (TIaaS)
Continuous, Integration, Build and Delivery (CIBD) in the Cloud
Passive Event based testing
Trigger Event based testing
o Performance Testing as a Service (PTaaS)
Metrics
 Globalisation
o Mobile Testing as a Service (MTaaS)
Accessibility/Usability/UXD/WAI
Cross Platform and Compatibility
o Security Testing as a Service (STaaS)

AutomationByExample.com

The challenge with any Test Automation implementation is the Domain Specific Languages (DSL)
whether it is used to describe the creation of test cloud environments or the execution of test automation
assets.

The idea behind Automation by Example (AbE) is that any test assets that is created can be
transparently traced back to the high-level Business Level Questions (BLQ) and they are truly human
readable, self-validating and self-maintaining. Other characteristics include Test Type agnostic (i.e. they
should be able to be reused for say performance, security or penetration testing), implementation
agnostic (i.e. technology platform VR (Oculus) or Web 2.0) and be First Day Testing (FDT) ready.

They also require an abstraction layer such as a Test Definition Layer (TDL) that include both
Business Specific Language (BSL) and the Test Specific Language (TSL) these are usually
combined to make a Domain Specific Language (DSL) that had limited reuse outside of that Business
Implementation.

BUSINESS ABSTRACTION LAYER

The Business Abstraction Layer (BAL) contains the high level Business Level Stories (BLS) in a
form that represents the Business Level Questions (BLQ):

As a <Credit Manager> I need to report daily on <Credit Scores> so that I can

present them to senior management.

The Business Level Keywords (BLK) represent the Business Level Answers (BLA) that need to be
tested through the Test Definition Layer (TDL) to provide evidence and build confidence for the
Solution Under Test (SUT):

Process.Accounting.CreditManager.CreditScores.Daily.Request
Process.Accounting.CreditManager.CreditScores.Daily.Generation
Process.Accounting.CreditManager.CreditScores.Daily.Display
First Day Automated Testing

My chapter in the Experiences of Test Automation29 book detailed in SECTION 29.3.3.1 not only how to
define the above Business Level Keywords (BLK) but how to utilise Business Process Modelling
(BPM) techniques for the decomposition of Business Process Scenarios (BPS) (i.e. Business
Workflows or paths through the SUT) into high-level models that represent the Business Process
Components (BPC) high-level Jigsaw pieces that make up the Solution Under Test (SUT).

BUSINESS PROCESS MODELLING (BPM)

The recent focus for enterprise businesses that are striving to encapsulate the Business on a Page
(BoaP) through abstraction techniques such Business Process Modelling (BPM) to empower business
agility, visibility and flexibility of business processes.

These models can be used as the underlining roadmap for the Solution Under Test (SUT) and can
either statically generated from a modelling tool or dynamic source like cloud maps that learn, discover
and model physical and logical endpoint(s) or a test asset loader that discovers the Solution Under Test
(SUT) at runtime or as part of each Continuous, Integration, Build & Deployment (CIBD).

The Business Process Model (BPM) below was automatically generated from the Solution Under
Test (SUT) code base which was using dynamic forms such as XAML that contained the Business
Process Design (BPD) which contained all the logic behind the Business Workflows & Rules.

BUSINESS PROCESS SCENARIOS

The business value of Business Process Scenarios (BPT) is the ability to relate the above Business
Level Questions (BLQ) with Business Level Answers (BLA) which requires the logical execution of
Business Process Components (BPC) represented Business Process Tests (BPT) and associated
Business Process Data (BPD).

29
Dorothy Graham, Mark Fewster, 2012, Experiences of Test Automation ISBN-10: 0-321-75406-9, 9th January 2012
 Request.VM

A1 C3 C4
C1 D1 C2
D2

Login.Process A2 Access.VM Logout.Process

D3

D4 E1
A3 E2
E3 E4

Manage.VM

The problem with anything that is static, such as manual or automated tests, is that they only represent a
Moment in Time (MiT) as soon as anything changes within the Solution Under Test (SUT) then they
become invalid or legacy. The same rules apply with test data as by the very nature it of data it is stateful
which is why test data preparation can be so time consuming.

Another problem is the complexity of testing or checking each physical or logical Business Process
Component (BPC) is that it could have hundreds or thousands of valid Business Process Tests (BPT)
associated with it. The above Business Process Model (BPM) contains only five BPCs but results in
over 20 different paths between each BPC and a huge number of possible BPTs.

Now what happens to the complexity when the Business introduces a fundamental change to the
Solution Under Test (SUT) for the following Business Level Stories (BLS):

As a <System User> I need to accept <Terms & Conditions> so that I comply with new
security EU data security policies.

This fundamental change to the SUT provides a conditional stateful BPC that is also geo-based /
regional (i.e. only for users accessing the solution from within the EU).

B1 Request.VM

C3 C4
C1 D1 C2
D2

Login.Process A1 Accept.Terms B2 Access.VM Logout.Process

D3

D4 E1
E2
E3 E4

B3
Manage.VM

This fundamental change could be associated with both Business Acceptance Criteria (BAC) (i.e.
ATDD) as well as Behaviours-Driven (i.e. BDD) defining Business Process Behaviours (BPB) that
contains Workflow and Rules and associated Business Process Data (BPD) that represents the
various Business Process Scenarios (BPS) that have now been impacted by this change (i.e. the path
from A1>C3>E1>D4>E3>C4 is now A1>B1>C3>E1>D4>E3>C4).
This is instead of attempting to create custom Behavioural or Executable Specifications to express
the endless possibilities impacted by this change (such as the difference between existing users, specific
users, new users, users logged in, users that have not logged in since a certain date) due to the rigid
Given When Then But ubiquitous language or syntax (taken directly from Domain Driven Design
(DDD)) to describe already highly complex ecosystems this approach uses Model-driven Design (MDD)
to visualise complexity and overlap business processes (such as business logic, workflows and rules).

The challenge with anything that is static and not dynamic is that it is instantly out of date as soon as
someone writes it and then requires maintenance to be fixed / updated, usually to support the latest
release (Build 2), and then overwriting the previous (Build 1). The same applies for static test scripts
Liner Process Mapping (LPM) or even executable specifications that contain any business logic within
the Test Abstraction Layer (TAL) and has not been externalised to the Business Abstraction Layer
(BAL) that dynamically creates the verbs and nouns for the Business Definition Language (BDL)
automatically which can then be interpreted by the Business Definition Layer (BDL).

BUSINESS PROCESS DATA

The same applies for Business Process Data (BPD) the traditional approach of using statically
generated test data that has been sanitised by the business using data masking techniques is stateful
and once consumed needs to be regenerated.

42% of companies in 2014 lack the right tools to create reusable test sets. 4

BUSINESS PROCESS TESTS

The greatest challenge is around creating self-aware emerging test assets that are both self-maintaining
and self-validating against dynamic sources such as a Test Asset DB (TA.db) or Test Asset Cube
(TA.c).

Whilst supporting the Business Process Intelligence (BPI) used describe the Moment in Time (MiT)
for the combinations of Solution, Application / Component, Version, Client, Workstream, Location
or Language that impacts the dynamic creation of Business Process Data (BPD) required by the
Business Process Tests (BPT) whilst still been truly human readable (i.e. supporting Legacy Testing).
 Test Automation as a Service (TAaaS)

Benefit Analysis Story Board Risk Dashboard

(Presentation Layer) (Presentation Layer) (Presentation Layer)

Business Questions Business Stories Business Risk

Business Specific Language Business Rules Project Risk

Test Specific Language Business Workflow Technical Risk

Solution Delivery Lifecycle Integration (SDLi)

Business Epic Business Process Business Process

Themes (BET) Management (BPM) Modelling (BPMN)
(HP Executive Scorecard (SaaS Platform)) (HP Application Lifecycle Management 12 (ALM)) (Business Process Modelling Notation v2.2 (xPDL))

Business User Business Process Business Process

Stories (BPC) Intelligence (BPI) Scenarios (BPS)
(HP Agile Manager (SaaS Platform)) (HP Application Lifecycle intelligence (ALI)) (extendable Application Markup Language (XAML))

Business Process Business Process Business Process

Design (BPD) Data (BP.db) Tests (BPT)
(Dynamic Data Source (xPDL)) (Business Data Cube (LINQ statements)) (Dynamic Data Source (xPDL))

Business Definition Layer (BDL)

Dynamic Delivery Adapters Dynamic Solution Adapters

Test Definition Layer (TDL)

Solution Under Test (SUT) Layer

The Test Definition Layer (TDL) encapsulates all the complexity and provides simplification of the high
level Testing Level Keywords (TLK) (i.e. Process.Login or Process.Logoff).

This is extremely important because it does not matter what Dynamic Delivery Adapters (DDA) we use
to communicate with the target Technology Delivery Adapters (TDA) (i.e. SQL(Azure), ORACLE(12c),
NoSQL(MongoDB)) or which Dynamic Solution Adapters (DSA) we use as the Test Solution
Adapters (TSA) (i.e. CodedUI, CodedFT (Alpha), MobileCloud C# (Beta), Ranorex, Telerik or
even the Selenium WebDriver) because that is all Logistics1 and no longer matters.

TEST ABSTRACTION LAYER

Implementation Agnostic (Logstics)

o Dynamic Delivery Adapters (Technology/Architecture/Application)
o Dynamic Solution Adapters (Version/Client/Workstream/Language)
 Support Test First Delivery (TFD)
Support Test-Driven Delivery integration (TDDi)
Support Legacy Testing (Manumation/Manual Execution)

Test First Delivery

This opens up the ability to use Test First Delivery (TFD) also known as First Day Testing (FDT). In
addition, the ability to have a direct traceability between stories, acceptance criteria, behaviours, test
scenarios and the underlying test steps is very valuable. It allows direct interrogation of test coverage or
test impact analysis (TIA) and enforces Solution Delivery Lifecycle integration (SDLCi) process.

The traditional approaches within Test-Driven Development (TDD) referred directly to development
activities using TDD, BDD and ATDD approaches for example in ATDD the direct mapping Acceptance
Criteria to Automated Checks or for BDD mapping Behaviours to Automated Checks.

Test-Driven Delivery integration

Modern day approaches for Test-Driven Delivery Integration (TDDi) supports First Day or Day Zero
techniques which supports Automated Checking of the conceptual Solutions Under Test (SUT without
a single line of code been written from the day one or during Sprint 0 when the solution has not yet even
decided on the Technologies or Platforms that will be under test easily be through using such domain
tools as Service/Network Virtualisation.

Legacy Testing

One of the biggest challenges with the traditional Testware Automation Frameworks (TAF) was what
happened when the automation broke or stopped working? Typically the Test Assets where unreadable
by manual testers who would have to understand the Domain Specific Language (DSL)
implementation of both the Test Specific Language (TSL) and Business Specific Language (BSL)
and interpret how that linked back to the Business Requirements.

One of the core design patterns within Test Automation as a Service (TAaaS) is that you only pay for
Business Process Scenarios (BPS) that have executed and provided results so if a Business
Process Component (BPC) breaks then not only can the Business Process Test (BPT) that failed can
be identified but you can visualise the Test Impact Analysis (TIA) on the Business Process
Management (BPM) domain tool over the complete end to end Solution Delivery process.
MANUMATION

Then execution can be resumed when either the identified broken piece of the Jigsaw puzzle has been
fixed or choose to execute the Business Process Components (BPC) manually. At which point not
only does it translate the Business Definition Language (BDL) but the specific Moment in Time (MiT)
of the Test Definition Layer (TDL) of the Solution Implementation such as Version, Client,
Workstream or Language as part of a Test Asset Journal (TA.j) that can be executed manually.

Manumation is an approach to automated testing that leverages the manual

effort involved in traditional testing to encapsulate both user and system under
test interactions. 5

Taking advantage of the Manumation approach the manually executed Business Process
Components (BPC) can then be used to resolve the issue of the broken piece of the Jigsaw puzzle
enabling the testing of the Solution Implementation to be executed from the Test Automation as a
Service (TAaaS) platform.

Design Goals

In summary the target Design Goals30 for Test Automation as a Service (TAaaS) platform:

Relevant Clear traceability of the business value of Automation through the

visualisation of the tests via Business Process Modelling (BPMn v2.2
compliant)
Effective Self-validating test assets achieved using natural language with context
sensitive validation against business and testing rules, workflows and data

Maintainable Self-maintaining test asset loader/scraper

Efficient Real-time reports on Solution Under Test (SUT) health including ratings such
as percentage availability since build/release, reported errors over time and
traffic to error ratio
Manageable Unified platform which non-domain experts can use a natural language to
represent business processes, user stories and acceptance criteria

Portable Technology agonistic - Platform, client, component, browser, version &

language
Test type agnostic smoke, regression, integration & performance
Reliable Fault tolerance is built in to report and continue on different levels of fuzzy
matching combined with technology agnostic test abstraction layer

Diagnosable Actionable defects provided by Test Infrastructure as a Service (TIaaS)

supporting live pause-playback and Dynamic Data Adapters (DDA) for
accelerated defect investigation and resolution

30
Alan Page, 2012, The Big Picture of Test Automation: Test Trustworthiness Fusion, Sydney, Australia, 8th September 2012
9. Further Reading

Check out Jonathon Wrights webinar on

Testing as a Service models and join the
discussion on TESTHuddle.com

Join the Discussion

 Click Here To View
The Forum For This eBook

www.eurostarconferences.com
Join us online at the links below.