You are on page 1of 40

@paul_gerrard

Agile Test Strategy


Paul Gerrard
paul@gerrardconsulting.com

gerrardconsulting.com
Overview
What is Agile Test Strategy?
Project Profiling
(Test Strategy as) Agile Interventions
Test Automation
Whats Left?
Summary
Q&A

Intelligent Definition and Assurance Slide 2


What is Agile Test
Strategy?

Agile Strategy an oxymoron?


Agile Test Strategy
What do we mean by this?
1. AGILE Test Strategy how to create a test
strategy in an Agile way?
2. AGILE Test Strategy a test strategy for an
Agile project?

Well look at how we created an Agile


approach to strategy, but well spend more
time on strategy for an Agile project.

Intelligent Definition and Assurance Slide 4


Google Agile Test Strategy
There are plenty of recipes out there
Most offer a selection of techniques but dont
provide much guidance on how to blend them
We need to know how to make choices, not
just know what choices exist

Strategy is a thought process, not a document


Although you might document the ideas for reuse, as
a reminder or for the record.

Intelligent Definition and Assurance Slide 5


Agile governance
Governance is the act of governing. It relates
to decisions that define expectations,
grant power, or verify performance
Wikipedia
Define expectations DEFINITION of need
Grant power DELEGATION to a project
team
Verify performance ASSURANCE of solution.

Intelligent Definition and Assurance Slide 6


Strategy helps you decide what to do
A. The strategy presents some decisions that can be
made ahead of time
B. Defines the process or method or information that
will allow decisions to be made (in project)
C. Sets out the principles (or process) to follow for
uncertain situations or unplanned events

In a structured/waterfall environment, most questions


answered off-the-shelf A-style, ready for anything
In an Agile environment might have some ready-made
policies but we manage scope and adapt (mostly C?)
Intelligent Definition and Assurance Slide 7
Contexts of Test Strategy
Axioms Early Communication
Testing De-
Risks
Duplication

Test Opportunities
Goals
Strategy

Culture Automation
Contract User
Human involvement
resource Constraints
Skills
Artefacts
Environment Process
(lack of?) Timescales
Traditional v Agile test strategy
Traditional structured, goal/risk-driven
Identify stakeholders; what are their goals?
Product risk analysis
Allocate risks/goals to test stages
Formulate test stage definitions (entry/exit criteria,
environments, tools etc. etc.
Agile interventionist, consensus-driven
Project profiling to set the testing theme
Identify testing interventions (perhaps better,
contributions) in the Agile process
Test policy overlays the process; catches exceptions.

Intelligent Definition and Assurance Slide 9


Project Profiling

Select a profile for your project first,


then choose the aspects of test
strategy that suite your project
Template-driven? Bah!
So this is just a template copy and edit process?
Wont you always end up with the same document?

Profiling doesnt need to be prescriptive


No need to write a document if you dont need to
But if company policy or common sense dictates certain
approaches, save yourself some time
Create a set of deeper, more detailed questions to be
answered (Pocketbook)
Profilers are really just checklists: heuristic guidelines
designed to help you make choices and trade-offs.

Intelligent Definition and Assurance Slide 11


Incident Mgt.

Using a Project Profiler to Derive Project Plan Tools

a Test Strategy and Project Plan Story Guideline

(A government client example) Test


Environment

Project Profiler
Cerise Waterfall
Test Plan Test Strategy
Project Manager Items
Orange
Risk Profiler
SCRUM/Agile
Consultation

Product Test Strategy


Green
Risks

Assurance

Unknowns

The Project Profiler (with Test Assurance) helps


Blue Project Managers to:
Select a project style that fits (Waterfall or Agile)
Identify the product risks that need testing
Identify test activities to include in project plans
Business, Project Team
Carefully define the scope of the project
and Boards
Intelligent Definition and Assurance 12
Project profiling process
Task
1 Have the Information you need to hand
2 Project Profiler (section 3):
Select the statements that best match your project context. The Blue column indicates
that you need more information consult your stakeholders, team or relevant Board(s).
3 Generic Risk Profiler (section 4):
Consider the generic project risks which are significant for your project? Can testing
help?
4 Product Risk Profiler (Section 5):
Consider the functional and non-functional risks that most concern your stakeholders
should they be tested?
5 Actions and Test Activities (Section 6):
Consider the actions that prompt your next steps and the test activities that should be
incorporated into your project plan.
6 Create your Test Strategy from the Test Strategy Framework Template
7 Incorporate the activities from stage 5 and identified in 6 into your Project
Plan.
Intelligent Definition and Assurance 13
Project Profiler (part of section 3)
Project Aspect Cerise Orange Green Blue
Responsibility for Users will take Users will be responsible Users will take part in UAT Users are unwilling/unable
Acceptance responsibility for UAT; for UAT but have no test or witness tests at critical to take part in UAT;
they have UAT experience experience periods, and will review reluctant to make the
the outcome acceptance decision or not
known
Requirements (Sources New system replaces a Users want to collaborate Users put the onus of Inexperienced users who
of Knowledge) well-understood existing to jointly define requirements elicitation on are unable or unwilling to
system; users have clear requirements and meet the project; requirements collaborate with
vision of system goals and them incrementally or and the solution will evolve requirements gathering
prefer to document their iteratively
requirements up-front
Requirements Stability New system is a functional New system replaces an New system supports a New system supports a
replacement of an existing existing system with new business need; new business need;
system or a well-defined enhancements or an business process exists business process is not
process (requirements can established (but not but will change/evolve; yet known; users have no
be fixed early on) necessarily documented users have experience of experience or
process) requirements requirements
Visibility, Formality High visibility/risk to High visibility/risk to Relatively low business- Potentially, high visibility,
general public; formal business; formal progress risk; informal progress high risk project, uncertain
progress reporting reporting required; some reporting is acceptable; impact on the business
required at board level; defined deliverables, some partial solution may
fixed scope and deliverables will suffice,
deliverables; formal emerge/evolve; some incremental/iterative
approvals and sign-offs approvals and sign-offs delivery
External Dependencies More than one or new Single, known supplier In-house development, no Dependencies on external
external suppliers responsible for external dependencies suppliers, their
responsible for development (and supplier responsibilities or
development (and supplier testing) competence not yet known
testing)
Etc. Etc. Etc. Etc. Etc.

Intelligent Definition and Assurance 14


Project types - examples
Cerise Structured, waterfall style of project (and includes
COTS projects)
Orange Iterative/prototyping style of project using SCRUM in
a formal way and having dedicated resources for the
Business Analyst and Tester roles.
Green A project using SCRUM in a less formal way but not
having dedicated resources for the Business Analyst
and/or the Tester roles.
Blue Blue column statements describe where there is
insufficient information available to identify the style
of project and the recommendation must be that
some further investigation is required.
Intelligent Definition and Assurance 15
(Test Strategy as)
Agile Interventions

Im using Scrum/Sprint terminology,


but you dont have to of course
Interventions (government client example)
No. Activity When?

On the following 1 Story Challenge As stories are added to the


Product Backlog

slides, we highlight
2 Story Definition As stories are added to a
Sprint Backlog
3 Daily Stand-Up Once per day during the
8 interventions

These activities are repeated


Sprint

for each Sprint iteration


4 Story Refinement Occurs throughout the Sprint

Some are test


as new information emerges
5 Developer Testing Occurs throughout the Sprint
as the developer codes the
phases, but some 6 Integration (and
stories
During and at the end of

arent incremental
System) Testing
each sprint, including the
final sprint
7 System Testing At the end of each sprint,
including the final sprint
8 User Acceptance At the end of each sprint,
Testing including the final sprint
9 Non-functional Expected to take place on an
Testing and Pre- as-needs basis.
Production Testing

Intelligent Definition and Assurance Slide 17


1. Story Challenge
Suggest what-ifs to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3


Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test


Existing Code base
Automated testing

Increasing
Scope of 7. System Test
Sys. Test
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after


System and Users Testing Final Sprint
1. Story Challenge
Suggest what-ifs to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3


Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test


Existing Code base
Automated testing

Increasing
Scope of 7. System Test
Sys. Test
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after


System and Users Testing Final Sprint
1. Story Challenge
Suggest what-ifs to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3


Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test


Existing Code base
Automated testing

Increasing
Scope of 7. System Test
Sys. Test
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after


System and Users Testing Final Sprint
1. Story Challenge
Suggest what-ifs to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3


Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test


Existing Code base
Automated testing

Increasing
Scope of
7. System Test
Int. Sys.
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after


System and Users Testing Final Sprint
1. Story Challenge
Suggest what-ifs to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3


Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test


Existing Code base
Automated testing

Increasing
Scope of
7. System Test
Int. Sys.
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after


System and Users Testing Final Sprint
Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 23


Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 24


Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 25


Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 26


4. Story Refinement (example definition)
Objectives To define acceptance criteria for all stories that are included in a Sprint as they are worked
on by development
To define scenarios that describe the tests and expected behaviours of the System
Improve understanding of the requirement and communicate anomalies to developers
To identify System Tests that exercise functionality of multiple stories that can be system
tested in this sprint
To assure the completeness for stories in the current Sprint
Whats being tested? Stories to be included in the current Sprint
Deliverables Refined story definitions with defined acceptance criteria and scenarios, where appropriate
System tests
Responsibilities (Orange) Tester challenges stories by suggesting potential scenarios, new stories, story merges and
splits; performs ad-hoc testing with/on behalf of developers; assures completeness of
stories.
Developers considers stories, evaluates impact on development
Product Owner or Analyst collates feedback and decisions on stories
Product Owner approves changes to stories, accepts completeness of stories
Scrum Master monitors progress; evaluates impact on resources and schedules
Responsibilities (Green) Not performed in Green projects
Baseline Story Guideline (reference 3)
Entry Criteria On commencement of the Sprint
Exit Criteria When all stories within a Sprint are considered complete
Queries, anomalies, discrepancies and inconsistencies have been eliminated or explained
System Tests appropriate to the Sprint have been defined
Definition of acceptance is agreed with Product Owner

Intelligent Definition and Assurance 27


Test Automation

Could you create an Agile Test


Strategy without automation?
Brian Maricks Testing quadrants

Intelligent Definition and Assurance 29


Test Automation Pyramid Lisa
Crispins version (Google others)
Pyramid reflects the
relative numbers of tests
Focus on
unit/component
Acceptance of Services
GUI are end-to-end
Manual checking the
exception?

Intelligent Definition and Assurance 30


Where do you automate?
3. Non-Technical
testers write scripts Stories/
Scenarios
Tools Experts write
interface Test Code 4. Programmers
write low level HTTP
GUI Test Framework GET/POST calls
through a driver that
simulates a browser
GUI Test Tool Unit Test Framework
2. Technical Testers
code scripts directly
Browser HTTP Driver 1. Programmers
write unit tests or
execute embedded
HTTP/S HTTP/S unit tests using a
unit test framework
to test components
Inter/Intranet

HTTP/S
Test Code
Web Server

App. Server Unit Test Framework


API
DB Server
Distributed testing
Use business stories and scenarios/acceptance
criteria to validate requirements
Reuse those stories to feed Acceptance-
Driven Development BDD/TDD process
Automated tests are an Anti-Regression tactic
Automated tests dont replicate manual tests;
think of them as a change trip-wire that
triggers an alarm, if tripped.

Intelligent Definition and Assurance 32


Deriving scenarios to test
To understand feature scope? Story Challenge

To get stakeholder to accept? Story Refinement

To validate the requirement? Story Definition

To estimate the work to build this feature?


Iteration Planning

To system test this feature? System Testing

To unit test this feature? Developer Testing

Scenarios are created to meet several goals


Intelligent Definition and Assurance 33
Whats Left?
Other aspects of test policy
Definitions (of done etc.)
Incident management
Test automation
Story format e.g. Gherkin
Environment request and management
Regression testing (at what levels)
Test deliverables and documentation.

Intelligent Definition and Assurance Slide 35


The Three Amigos
Business Analyst
Liaises and manages stakeholders and their needs
Transforms business requirements into
specification (at multiple levels)
Developer
Scopes, designs, builds, tests and delivers features
Tester
Challenges the thinking on the project
Performs Assurance in the small.

Intelligent Definition and Assurance Slide 36


The testers contribution to testing
_________ feature/story acceptance criteria
_________ the developers to unit test (auto)
_________ feature/story acceptance
_________ acceptance test
_________ service/UI level automation
Scope: from low-level detail to system integration
Liaison with integration testers and feedback

Fill in the blanks yourself; negotiate with your


team.
Intelligent Definition and Assurance Slide 37
Summary
Close
Agile test strategy has its place and many aspects
of test can be pre-defined
Importantly, we use a principled approach to deal
with the unexpected
Project profiling can help
Testing as interventions, rather than test phases
The testing role is split at least three ways the
tester doesnt own testing think TESTMASTER
Test automation in the context of Specification by
Example, requirements validation, BDD, TDD.

Intelligent Definition and Assurance Slide 39


@paul_gerrard
Agile Test Strategy
Paul Gerrard
paul@gerrardconsulting.com

gerrardconsulting.com