Agile Test Strategy

Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
@paul_gerrard
Agile Test Strategy
Overview
What is Agile Test Strategy?

Project Profiling
(Test Strategy as) Agile Interventions
Test Automation
Whats Left?
Summary
Q&A
Intelligent Definition and Assurance
Slide 2
What is Agile Test

Strategy?
Agile Strategy an
oxymoron?
Agile Test Strategy

What do we mean by this?
1. AGILE Test Strategy how to create a test
strategy in an Agile way?
2. AGILE Test Strategy a test strategy for
an Agile project?
Well look at how we created an Agile

approach to strategy, but well spend
more time on strategy for an Agile
project.
Slide 4
Google Agile Test Strategy

There are plenty of recipes out there
Most offer a selection of techniques but dont
provide much guidance on how to blend them
We need to know how to make choices, not
just know what choices exist
Strategy is a thought process, not a
document
Although you might document the ideas for reuse,
as a reminder or for the record.
Slide 5
Agile governance
Governanceis the act of governing. It
relates to decisions that defineexpectations,
grantpower, or verifyperformance
Wikipedia
Defineexpectations DEFINITION of need
Grantpower DELEGATION to a project
team
Verifyperformance ASSURANCE of
solution.
Slide 6
Strategy helps you decide what to

do
A. The strategy presents some decisions that can be
made ahead of time
B. Defines the process or method or information that
will allow decisions to be made (in project)
C. Sets out the principles (or process) to follow for
uncertain situations or unplanned events
. In a structured/waterfall environment, most questions
answered off-the-shelf A-style, ready for anything
. In an Agile environment might have some readymade policies but we manage scope and adapt
(mostly C?)
Slide 7
Contexts of Test Strategy

Axioms
Risks
Goals
Test
Strategy
Culture
Human
resour
ce
Environme
nt
Early Communicati
on
Testing
DeDuplicati
on
Constraints
Contra
ct
Opportunities
Automatio
n
Skills
Process
(lack of?) Timescales
User
involveme
nt
Artefacts
Traditional v Agile test

strategy
Traditional structured, goal/risk-driven
Identify stakeholders; what are their goals?
Product risk analysis
Allocate risks/goals to test stages
Formulate test stage definitions (entry/exit criteria,
environments, tools etc. etc.
Agile interventionist, consensus-driven

Project profiling to set the testing theme
Identify testing interventions (perhaps better,
contributions) in the Agile process
Test policy overlays the process; catches exceptions.
Slide 9
Project Profiling
Select a profile for your
project first, then choose the
aspects of test strategy that
suite your project
Template-driven? Bah!
So this is just a template copy and edit process?
Wont you always end up with the same document?
Profiling doesnt need to be prescriptive
No need to write a document if you dont need to
But if company policy or common sense dictates certain
approaches, save yourself some time
Create a set of deeper, more detailed questions to be
answered (Pocketbook)
Profilers are really just checklists: heuristic

guidelines designed to help you make choices and
trade-offs.
Slide 11
Using a Project Profiler to

Derive a Test Strategy and
Project Plan
(A government client
example)
Project
Profiler
Consultation
Project Manager
Incident Mgt.
Tools
Project Plan
Story
Guideline
Test
Test
Plan
Items
Risk Profiler
Produc
t Risks
Environment
Cerise
Orange
Green
Waterfall
Test
Strategy
SCRUM/Agi
le
Test
Strategy
Assurance
Business, Project Team

and Boards
Unknowns
The Project Profiler (with Test Assurance)
helps Project Managers to:
Blue
Select a project style that fits (Waterfall
or Agile)
Identify the product risks that need
testing
Intelligent Definition
12include in
Identifyand
test activities to
Assurance project plans
Project profiling process

Task
1
2
Have the Information you need to hand

Project Profiler (section 3):
Select the statements that best match your project context. The Blue
column indicates that you need more information consult your
stakeholders, team or relevant Board(s).
Generic Risk Profiler (section 4):

Consider the generic project risks which are significant for your
project? Can testing help?
Product Risk Profiler (Section 5):

Consider the functional and non-functional risks that most concern your
stakeholders should they be tested?
Actions and Test Activities (Section 6):

Consider the actions that prompt your next steps and the test
activities that should be incorporated into your project plan.
6
7
Create your Test Strategy from the Test Strategy

Framework Template
Incorporate the activities from stage 5 and identified in
6 into your Project
Plan.
Intelligent
Definition and
13
Assurance
Project Profiler
(part of section 3)
Project Aspect
Responsibility for
Acceptance
Cerise
Users will take
responsibility for UAT; they
have UAT experience
Orange
Users will be responsible
for UAT but have no test
experience
Green
Users will take part in UAT
or witness tests at critical
periods, and will review
the outcome
Requirements (Sources
of Knowledge)
New system replaces a

well-understood existing
system; users have clear
vision of system goals and
prefer to document their
requirements up-front
Users want to collaborate

to jointly define
requirements and meet
them incrementally or
iteratively
Users put the onus of

requirements elicitation on
the project; requirements
and the solution will evolve
Requirements Stability
New system is a functional

replacement of an existing
system or a well-defined
process (requirements can
be fixed early on)
New system replaces an

existing system with
enhancements or an
established (but not
necessarily documented
process)
New system supports a

new business need;
business process exists
but will change/evolve;
users have experience of
requirements
New system supports a

new business need;
business process is not
yet known; users have
no experience or
requirements
Visibility, Formality
High visibility/risk to
general public; formal
progress reporting
required at board level;
fixed scope and
deliverables; formal
approvals and sign-offs
High visibility/risk to
business; formal progress
reporting required; some
defined deliverables, some
deliverables will
emerge/evolve; some
approvals and sign-offs
Relatively low businessrisk; informal progress

reporting is acceptable;
partial solution may
suffice,
incremental/iterative
delivery
Potentially, high
visibility, high risk
project, uncertain impact
on the business
External Dependencies
More than one or new

external suppliers
responsible for
development (and supplier
testing)
Etc.
Single, known supplier

responsible for
development (and supplier
testing)
In-house development, no
external dependencies
Etc.
Etc.
Dependencies on
external suppliers, their
responsibilities or
competence not yet
known
Etc.
Etc.
Intelligent Definition and

Assurance
14
Blue
Users are
unwilling/unable to take
part in UAT; reluctant to
make the acceptance
decision or not known
Inexperienced users
who are unable or
unwilling to collaborate
with requirements
gathering
Project types - examples

Cerise
Orang
e
Green
Blue
Structured, waterfall style of project (and

includes COTS projects)
Iterative/prototyping style of project using
SCRUM in a formal way and having
dedicated resources for the Business
Analyst and Tester roles.
A project using SCRUM in a less formal way
but not having dedicated resources for the
Business Analyst and/or the Tester roles.
Blue column statements describe where
there is insufficient information available to
identify the style of project and the
recommendation must be that some further
Intelligent
Definition
and
15
investigation
is
required.
Assurance
(Test Strategy as)

Agile Interventions
Im using Scrum/Sprint
terminology, but you dont
have to of course
Interventions
On the
following
slides, we
highlight 8
interventions
Some are test
phases, but
some arent
(government client example)
No.
1
2
The
se
acti
vitie
s
are
repe
ated
for
eac
h
Spri
nt
iter
atio
n
7
8
Activity
Story
Challenge
Story Definition
3 Daily Stand-Up
4 Story
Refinement
5 Developer
Testing
6 Integration
(and
incremental
System)
Testing
System Testing
User
Acceptance
Testing
9
Non-functional
When?
As stories are added to
the Product Backlog
As stories are added to
a Sprint Backlog
Once per day during
the Sprint
Occurs throughout the
Sprint as new
information emerges
Occurs throughout the
Sprint as the developer
codes the stories
During and at the end
of each sprint,
including the final
sprint
At the end of each
sprint, including the
final sprint
At the end of each
sprint, including the
final 17
sprint
Slide
Expected to take place
1. Story
Challenge
Suggest what-ifs
to challenge new
stories and define
story headlines
2. Story
Definition
Introduce
scenarios to
enhance the
Acceptance
Criteria
Project Level Test Activities

(This diagram shows three sprints, but there could be
more or fewer)
Sprint
Backlog
Sprint 1
Developed
Stories
Sprint
Backlog
Sprint
Backlog
Sprint 2
Sprint 3
Developed
Stories
Developed
Stories
New Code
6.
Integration into
Existing Code base Integration
Automated testing Test
Increasin
g Scope
of Sys.
Test and
UAT
6.
Integration
Test
6.
Integration
Test
7. System Test
8. User Test
Increasing Scope of
Integration, System and Users
Testing
Complete Tests
after Final Sprint
1. Story
Challenge
Suggest what-ifs
to challenge new
stories and define
story headlines
2. Story
Definition
Introduce
scenarios to
enhance the
Acceptance
Criteria

more or fewer)
Sprint
Backlog
Sprint 1
Developed
Stories
Sprint
Backlog
Sprint
Backlog
Sprint 2
Sprint 3
Developed
Stories
Developed
Stories
New Code
6.
Integration into
Increasin
g Scope
of Sys.
Test and
UAT
6.
Integration
Test
6.
Integration
Test
7. System Test
8. User Test
Increasing Scope of
Testing
Complete Tests
after Final Sprint
1. Story
Challenge
Suggest what-ifs
to challenge new
stories and define
story headlines
2. Story
Definition
Introduce
scenarios to
enhance the
Acceptance
Criteria

more or fewer)
Sprint
Backlog
Sprint 1
Developed
Stories
Sprint
Backlog
Sprint
Backlog
Sprint 2
Sprint 3
Developed
Stories
Developed
Stories
New Code
6.
Integration into
Increasin
g Scope
of Sys.
Test and
UAT
6.
Integration
Test
6.
Integration
Test
7. System Test
8. User Test
Increasing Scope of
Testing
Complete Tests
after Final Sprint
1. Story
Challenge
Suggest what-ifs
to challenge new
stories and define
story headlines
2. Story
Definition
Introduce
scenarios to
enhance the
Acceptance
Criteria

more or fewer)
Sprint
Backlog
Sprint 1
Developed
Stories
Sprint
Backlog
Sprint
Backlog
Sprint 2
Sprint 3
Developed
Stories
Developed
Stories
New Code
6.
6.
6.
Integration into
Integration
Integration
Test
Test
Test
Automated testing
Increasi
ng
Scope
7. System Test
of Int.
Sys.
8. User Test
and UAT
Increasing Scope of
Complete Tests
after Final Sprint
Testing
1. Story
Challenge
Suggest what-ifs
to challenge new
stories and define
story headlines
2. Story
Definition
Introduce
scenarios to
enhance the
Acceptance
Criteria

more or fewer)
Sprint
Backlog
Sprint 1
Developed
Stories
Sprint
Backlog
Sprint
Backlog
Sprint 2
Sprint 3
Developed
Stories
Developed
Stories
New Code
6.
6.
6.
Integration into
Integration
Integration
Test
Test
Test
Automated testing
Increasi
ng
Scope
7. System Test
of Int.
Sys.
8. User Test
and UAT
Increasing Scope of
Complete Tests
after Final Sprint
Testing
Test Activities in the Sprint

3. Daily Stand-Up
Report anomalies
found, stories tested,
amended, created
Daily Scrum
Stand-Up
Meeting
4. Story
Refinement
Refine scenarios to
enhance story
definition, create
system tests as
stories, as required
Sprint
Backlog
24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Product backlog
As prioritised by Product Owner
Assurance
5) Developer Testing
Private ad-hoc tests and
build/run automated
unit tests
6)
Integration/System
Testing
Incorporate automated
unit tests into the CI
regime.
On weekly basis and at
end of Sprint, deploy to
System test
environment and tester
runs system tests.
Potentially Shippable
Product increment
23

3. Daily Stand-Up
Report anomalies
amended, created
Daily Scrum
Stand-Up
Meeting
4. Story
Refinement
Refine scenarios to
enhance story
definition, create
system tests as
Sprint
Backlog
24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Product backlog
Assurance
build/run automated
unit tests
6)
Integration/System
Testing
regime.
System test
runs system tests.
Product increment
24

3. Daily Stand-Up
Report anomalies
amended, created
Daily Scrum
Stand-Up
Meeting
4. Story
Refinement
Refine scenarios to
enhance story
definition, create
system tests as
Sprint
Backlog
24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Product backlog
Assurance
build/run automated
unit tests
6)
Integration/System
Testing
regime.
System test
runs system tests.
Product increment
25

3. Daily Stand-Up
Report anomalies
amended, created
Daily Scrum
Stand-Up
Meeting
4. Story
Refinement
Refine scenarios to
enhance story
definition, create
system tests as
Sprint
Backlog
24
Hours
2-4 Weeks
Backlog tasks
expanded
by team
Product backlog
Assurance
build/run automated
unit tests
6)
Integration/System
Testing
regime.
System test
runs system tests.
Product increment
26
4. Story Refinement
(example
definition)
Objectives
Whats being
tested?
Deliverables
Responsibilities
(Orange)
Refined story definitions with defined acceptance criteria and scenarios,

where appropriate
System tests
Tester challenges stories by suggesting potential scenarios, new stories,
story merges and splits; performs ad-hoc testing with/on behalf of
developers; assures completeness of stories.
Developers considers stories, evaluates impact on development
Product Owner or Analyst collates feedback and decisions on stories
Product Owner approves changes to stories, accepts completeness of
stories
Scrum Master monitors progress; evaluates impact on resources and
schedules
Not performed in Green projects
Story Guideline (reference 3)

On commencement of the Sprint
Definition
and
27
WhenIntelligent
all stories within
a Sprint
are considered complete
Assurance
Queries,
anomalies, discrepancies and inconsistencies have been
Responsibilities
(Green)
Baseline
Entry Criteria
Exit Criteria
To define acceptance criteria for all stories that are included in a Sprint as
they are worked on by development
To define scenarios that describe the tests and expected behaviours of
the System
Improve understanding of the requirement and communicate anomalies
to developers
To identify System Tests that exercise functionality of multiple stories that
can be system tested in this sprint
To assure the completeness for stories in the current Sprint
Stories to be included in the current Sprint
Test Automation
Could you create an Agile Test
Strategy without automation?
Brian Maricks Testing

quadrants

Assurance
29
Test Automation Pyramid

Lisa Crispins version
(Google others)
Pyramid reflects the

relative numbers of
tests
Focus on
unit/component
Acceptance of
Services
GUI are end-to-end
Manual checking the
exception?

Assurance
30
Where do you automate?

3.
NonTechnical testers
write scripts
Tools Experts
write interface
Stories/
Scenarios
Test Code
GUI Test
Framework
2.
Technical
Testers code
scripts directly
Unit Test
Framework
GUI Test Tool

Browser
HTTP Driver
HTTP/S
HTTP/S
Inter/Intrane
t
HTTP/S
4.
Programmers
write low level
HTTP GET/POST
calls through a
driver that
simulates a
browser
1.
Programmers
write unit tests
or execute
embedded unit
tests using a
unit test
framework to
test components
Test Code
Web Server
App. Server
DB Server
API
Unit Test
Framework
Distributed testing
Use business stories and
scenarios/acceptance criteria to validate
requirements
Reuse those stories to feed AcceptanceDriven Development BDD/TDD process
Automated tests are an Anti-Regression
tactic
Automated tests dont replicate manual
tests; think of them as a change trip-wire
that triggers an alarm, if tripped.
Assurance
32
Deriving scenarios to test
To understand feature scope? Story

Challenge
To get stakeholder to accept? Story
Refinement
To validate the requirement? Story
Definition
To estimate the work to build this
Iteration
Planning
feature?
System Testing
To system test this feature?
Developer
Testing
To unit test this feature?
cenarios are created to meet several goals
Assurance
33
Whats Left?
Other aspects of test policy
Definitions (of done etc.)

Incident management
Test automation
Story format e.g. Gherkin
Environment request and
management
Regression testing (at what levels)
Test deliverables and documentation.
Slide 35
The Three Amigos

Business Analyst
Liaises and manages stakeholders and their needs
Transforms business requirements into
specification (at multiple levels)
Developer
Scopes, designs, builds, tests and delivers
features
Tester
Challenges the thinking on the project
Performs Assurance in the small.
Slide 36
The testers contribution to

testing
_________ feature/story acceptance criteria

_________ the developers to unit test (auto)
_________ feature/story acceptance
_________ acceptance test
_________ service/UI level automation
Scope: from low-level detail to system integration
Liaison with integration testers and feedback
Fill in the blanks yourself; negotiate with your

team.
Slide 37
Summary
Close
Agile test strategy has its place and many
aspects of test can be pre-defined
Importantly, we use a principled approach to
deal with the unexpected
Project profiling can help
Testing as interventions, rather than test phases
The testing role is split at least three ways the
tester doesnt own testing think TESTMASTER
Test automation in the context of Specification
by Example, requirements validation, BDD, TDD.
Slide 39
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
@paul_gerrard
Agile Test Strategy

Agile Test Strategy

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Agile Test Strategy

Uploaded by

Copyright:

Available Formats

Paul Gerrard