EBOOK

LogiGear

Learn more about Agile Testing

At LogiGear, we believe that good test design is the main driver of automation success. For twenty years we have
successfully helped a wide range of clients implement and make the transition to agile automation methodology.

This eBook outlines ten ways to improve your automation, and includes some of our own experiences, as well as some
projects that we’ve tackled for clients over the years. We know that there are still a wide number of organizations that
are still struggling with Agile; whether you are new to automation, or an experienced practitioner, we hope that you
use this eBook as a resource.

Meet the authors

MICHAEL HACKETT | Senior Vice President
Michael is a co-founder of LogiGear Corporation, and has over two decades of
experience in software engineering in banking, securities, healthcare and
consumer electronics.
Michael is a Certified Scrum Master and has co-authored two books on software
testing. Testing Applications on the Web: Test Planning for Mobile and
Internet-Based Systems (Wiley, 2nd ed. 2003), available in English, Chinese and
Japanese, and Global Software Test Automation (Happy About Publishing,
2006). He is a founding member of the Board of Advisors at the University of
California Berkeley Extension and has taught for the Certificate in Software
Quality Engineering and Management at the University of California Santa Cruz
Extension. As a member of IEEE, his training courses have brought Silicon Valley
testing expertise to over 16 countries.

Michael holds a Bachelor of Science in Engineering from Carnegie Mellon

University.

HANS BUWALDA | Chief Technology Officer

Hans leads LogiGear’s research and development of test automation
solutions, and the delivery of advanced test automation consulting and
engineering services. He is the original architect of the keyword framework for
software testing organizations, and he assists clients in strategic
implementation of Action Based Testing™ throughout their testing
organizations. Hans is also the lead developer of LogiGear’s TestArchitect™,
the action-based toolset for software test design, automation and
management.

In his prior role as project director at CMG (now CGI) in the Netherlands, Hans
assisted clients across Europe in developing and deploying software testing
solutions. He is an internationally recognized expert on test automation, test
development and testing technology management. He is co-author of
Integrated Test Design and Automation (Addison Wesley, 2001), and speaks
frequently at international testing conferences.

Hans holds a Master of Science in Computer Science from Free University,

Amsterdam.

Learn more about Agile Testing 02

With Agile transformation, carefully fitting testing in with
other Agile practices will improve both system
development and test development, and lead to the
successful deployment of full stack test automation.
Some teams have succeeded, while others are still
struggling with implementing the practices. Fast, easy,
and low-maintenance automation, while essential, is
particularly troublesome for many agile test teams.
A majority of typical practices in the Agile
methodology come from Extreme Programming. These
include continuous user involvement and an emphasis
on the testing role (‘Users’ may be the actual users of
the system you are creating, customers, or business
analysts who provide the requirements on behalf of the
end-users). Testing is positioned in Agile system
development as a driver for system development, with
tests often created before the functionality they are
verifying.
One of the defining characteristics of Agile is the
speed of delivery. From traditional development
methods, the time to develop and deploy product has
greatly shortened. This is a positive movement thanks
to evolving user stories and requirements, collaboration
Learn more about Agile Testing
on design and implementation, and rapid iterations.
Yet each of these practices has crunched testing time
and made traditional automation implementation
more difficult.
At the same time, one of the benefits of the Agile
approach to projects is their friendliness towards
testing. The testing activities, and the testers with it, are
integrated into the team; testing and quality are now
defined as team responsibilities. Automation nowadays
is a must-have that needs to be addressed. So the
question is: How can we be Agile and create scalable,
low-maintenance automation?
New Agile thinking on automation focuses on test
design and the engineering of those tests being
separate activities. Test automation itself must also be
an Agile, iterative process. Meaning that you can
design and author a test early and interactively even
when you don’t have the actual product. For
example, you work with user stories and design the
tests just as a programmer would work with user stories
to design code. The test team would later engineer the
automation of the tests just as the programmers would
later engineer the code design into the product.
03
 TestArchitect™. Our clients have gained great values
Full stack automation happens on multiple levels in a
with these methods, especially as they transition to
system and occurs in stages, starting with unit tests. Here,
Continuous Testing and Continuous Integration.
we’ll focus on automated functional tests at the UI level.
Both of these methods employ a simple domain
At LogiGear, we value employing pragmatic test
specific language to facilitate the authoring of
development methods to ensure testing and test
automated tests. For detailed information on BDD
automation fit well with Agile development. The two
practices, go to https://dannorth.net/introducing-bdd/.
methods we promote and support are Dan North’s
behavior-driven development (BDD) method with Open In this eBook, we will primarily discuss how ABT drives
source automation technology, such as Selenium or Agile UI test automation and give some perspective on
Appium, and our own Action Based testing (ABT) how this can be best achieved.
method with our proprietary automation technology,

Learn more about Agile Testing 04

 1. Design and Organize Tests and Automation Architecture Into
Separate Phases
Align 2. Pair Test Development Phases to Product Development Phases
3. Have a Complete Detailed Modular Test Design

4. Build the Base Early

Build 5. Rethink What Is Automation
6. Have a Strategy Regarding Base Data

7. Make Automation Resilient

8. Follow the Development Sprints
Automate 9. Address Testability of the System Under Test
10. Test and Update the Automation

ABT is a modular-design and action-driven test
method that provides a systematic approach to
increase the success of automated testing.
Modular-design addresses the challenges of test
planning and test case management through
efficient test organization.
Action-driven test development eliminates the
majority of the programming work required to
automate and maintain tests long-term. ABT
consists of three parts:
1. TOP-DOWN, MODULAR TEST PLANNING
2. TEST MODULE DEVELOPMENT
3. ACTION-DRIVEN TEST AUTHORING
In ABT the focus is on development of test assets
called “test modules”, an activity that can be
independent from system development. It has no
specific preference on when in the Agile process
those test modules are developed. That makes it
easy to integrate the methodology into Agile. The
term often used for test automation in Agile
projects that best describes what is needed is
“just-in-time automation.” When ABT is applied,
the term changes to “just in time test
development.” Independent to that, a high level
of automation can be an invaluable contribution
in improving the productivity and speed in sprints.

What is a test module?

Test Modules are containers for tests of user stories or
software requirements. Every test module has a clear
and well-differentiated scope from every other test
module to avoid redundant activities and checks. This
makes tests less fragile and easier to maintain. Giving
each Test Module a descriptive name makes it easy to
identify what the tests in the module cover.

Learn more about Agile Testing 05

There are four sections to a Test Module:
Objectives Initial
Each Test Module is given a descriptive name to
make it easy to identify what the tests in the module
cover.
Test design
Making automated testing successful (scalable and
maintainable for the long-term) is often regarded as a
technical challenge, when in reality, it is a test design
challenge:
● Technical problems can take precedence
sometimes, but once resolved they normally go off the
critical path.
● On the other hand, poorly organized tests can be a
continuous burden for their automation.
Learn more about Agile Testing
Test Case Final
The objectives section lists every objective
associated with the module’s test cases and defines
the scope of the test module. Objectives allow
readers to understand why test cases are designed
the way they are, and give an auditor quick insight
into the correctness and completeness of a test.
The initial section contains the action lines required
for initialization of the test. For example, actions for
launching the application under test would be
written here. This avoids repetitive steps when
initialization is required for multiple test cases in the
module.
The test case section is where test cases are created.
Each test case has a representative number,
accompanied by title and one or more test
objective lines. The title and objective explain the
test case purpose.
The final section is for any cleanup operation upon
test completion, such as closing the application
under test.
How can automated functional testing fit into Agile
projects? That is a question we encounter a lot
nowadays. Functional testing often remains a manual
process because during Agile iterations/sprints, there is
simply not enough time to automate it. This is dissimilar
to unit testing, which is routinely automated effectively.
The organization of tests in meaningful modules, the
separation of test design from automation design, and
the low-maintenance action/keyword foundation all
contribute to making ABT the sure path to success for
Agile Automation. These are proven results.
06
 TOP 10 WAYS TO MAKE
AGILE TEST AUTOMATION

1. Design and Organize Test and Automation Architecture into Separate

Phases
With ABT and TestArchitect the main focus is on test design as the driver to automation success. The main
goals for test design are:
● Simplicity, in particular for business testers.
● Efficiency, to reach the maximum effect with a minimal amount of tests and actions.
● Manageability, to be able to “see the forest through the trees”.
● Maintainability, to be able to adapt to changing applications quickly.
● Depth and coverage, in other words, being good tests.

A good organization of tests can benefit a project in the following ways:

● Better tests: make tests more comprehensive and deeper.
● Better automation: know and address the impact of changes in the application under test.

Test design, while essential, is only one of the many activities of test development. Other activities include
test execution, test result follow up, and test maintenance. Automation architecture and implementation
are the technical part of a test automation project. Both test design and automation architecture need to
be thoughtfully organized with product development phases in mind.

2. Pair Test Development to Product Development Phases

Pairing test development to system development means sticking to what most Agile life cycles prescribe.
The tests, in the ABT test modules, are created together with users or product owners, whenever the life
cycle needs it. The advantage is that this is highly recognizable as an Agile approach, and it allows for
better interaction between users/POs and developers, with tests serving as a means of communication.
Early focus is on collaboration and gaining knowledge. Be knowledgeable on topics that are relevant for a
sprint, like subject matter, software testing, automation, etc. Try to get necessary automation technology
available ahead of a sprint needing them. A sprint is usually not the place to deal with technical puzzles like
custom controls, multi-media, etc. And the specialized engineers solving such puzzles will usually not be in
the teams.

Learn more about Agile Testing 07

 TOP 10 WAYS TO MAKE
A SUCCESS

Agile Testing Cycle

Sprint Sprint

Team Test Definition & Development Test Definition & Development

Automation Automation

Facility Additional Test Development

Additional Automation
Additional Test Development
Additional Automation

3. Have a Complete Detailed Modular Test Design

Establish an overall test design. Outline the folders and test modules within them. Even though in Agile
projects test development may not be top-down, an overall test design can work as a framework to help
keep tests organized, which benefits their maintainability and manageability.
For each test module: do an intake, make sure to understand what needs to be tested, and who needs to
be involved. Plan the module, in particular the test objectives and test cases. Testing techniques like
decision tables and state transition diagrams can sometimes help in this stage to identify test cases.
Create the test cases using actions. What actions to use depend on the scope of the test module. When
the test modules will actually be developed is a project decision, based on priorities, and on how projects
are organized. In the case of an Agile method, scrum teams play a large role in deciding what test
modules to develop. However, given the long lasting value potential of tests as assets or products, an
organization can decide to plan their production explicitly as outcomes of sprints, in addition to the
software products the teams will produce. It depends on your vision for the tests:
● Are tests supporting relatively straightforward artifacts just meant to help build and maintain the main
product, or
● Are tests products with business value, that reflect a deep understanding of the functions and workings
of the applications under test, and the business it supports?

What is an action?
Actions are the building blocks of your test
cases, telling the system which operations to
perform. Actions typically interface with the
application under test, but may have other
functions as well. These include documentary
actions, actions that interact with the host file
system, and various support actions.

Learn more about Agile Testing 08

 TOP 10 WAYS TO MAKE
A SUCCESS

An action may be something elementary, such as

report or click, two actions which are built into
TestArchitect.
System-level actions
(i.e., “click” or “enter”)
{Built-in or
User-Scripted Actions} Application-level
actions
Created by combining
pre-existing
System-level Action
and Application-level
Action (i.e., “order
entry”)
Other pre-existing
Application-level {User-Defined Actions}
actions

Agree on the approach, both the method in general and various relevant practices. Discussions on
fundamentals can slow a team down. To ease the process we like to do “sit down meetings”: at the end
of the day come together, relax (“sit down”), and discuss experiences and practices, if possible with an
experienced person in the room. It works better than retrospectives if a new trade needs to be learned
and understood.

Embrace the “sit-down”

For our own division that develops

TestArchitect we have daily “sit down”
meetings. The main investment was
some tea, and maybe 20-30 minutes
of the time of some key people. It was
surprising to see that after only a
couple of sessions, there was a good
agreement on the approach and
relevant practices. After that, the
teams were able to create and
automate action based tests within
the regular scrum project approach.

Learn more about Agile Testing 09

 TOP 10 WAYS TO MAKE
A SUCCESS

4. Build the Base Early

Build the framework. Use an iterative approach. Start high level and build down.
Save design and maintenance time by using a just-in-time, JIT, approach.
Hybrid approach: business-level tests are developed early; detailed, typically UI-specific tests are created
in parallel with the development sprints.
Test and automation artifacts are products in their own right, with their own life cycles that may or may not
be embedded in sprints. For example a test module can be developed early independent of sprints, but
also early within a sprint. An interaction test module will normally be developed later in a sprint, when the
interaction functionality has been defined and/or developed and is stable. Actions might be implemented
long after (or even before) their use in tests.

5. Rethink What is Automation. Actions and Interfaces are Different

than Test Design and Framework.
Automation focuses solely on the action keywords, interpreting actions, matching user or non-user
interfaces, researching technology challenges, etc.
In this step, you implement the action keywords, capturing the UI definitions, matching user or non-user
interfaces, researching and unblocking technology-related problems, etc.
Working with Action Definition—at some point actions have to be defined and created.
The action definition provides the details required for the activity and verification. In this example, using
ABTL, a domain specific language to implement the action “make reservation”, the action also calls other
actions (“search restaurant” and “set reservation details”) that would each have their own action
definitions with details.

Learn more about Agile Testing 10

 TOP 10 WAYS TO MAKE
A SUCCESS

6. Have a Strategy Regarding Base Data

This is often one of the most difficult puzzles in test design.

● Designing automated tests that deal with data can be particularly
challenging. Tests need certain base data to be available and in a
predictable state when they run.
● Virtual machines, containers, and cloud options are increasingly providing
solutions to plan and manage environments that can include base data.

7. Make Automation Resilient

Automated tests can be very disruptive if they’re too sensitive to changes of the application under test or
its environment. Much of what we discuss in Action Based Testing is aimed in achieving resilience:
● The ability of tests to still work even if there are changes.
● If a change in the tests or their automation is needed, it should be localized, in a single or few actions or
interface mappings.
The automation engineer cannot do this alone; it is a team responsibility, for example:
● The tester needs to produce a test design that is good for automation: specify detailed steps and
checks if and only if needed.
● The developers should play their role by providing insight in what they produce, and by baking testability
into their products.
● Product owners should help come up with “interesting” cases: business situations that they know are
hard for the application to handle. Without their help tests often get bland (and low level).

Share responsibility
One of our financial services clients was a few months into a project when
automation problems seeded and started to get out of control. They got
help. We paused the project. We then re-structured, reinforced and began
building again. We began by following the XP practice of coding
standards, common data formats, common naming suggestions, and most
of all, common test design methods. Team members were taught that
actions are shared and tests are shared—by using this practice,
maintenance became easier, and also became a shared responsibility.
Once we rebuilt the foundation, most of the tests that were created during
the initial phase were thrown out but after that the team’s test creation
velocity took off! More and better testing was happening. Even from the
start, maintenance costs were cut just from sprint to sprint.

Learn more about Agile Testing 11

 TOP 10 WAYS TO MAKE
A SUCCESS

8. Follow the Development Sprints

In an Agile project, testing falls Automating in the same sprint has become
into three timelines: an obstacle for most teams.
Avoid delaying test automation, since such
delays can add to technical debt.
1. Testing in 2. Test development
prior to development Avoid delaying automation to the next sprint
regular system as cleaning up from last sprint rarely meets
development sprints.
the definition of done and is not effective for
sprints. the scrum team to have testers focused on
a finished sprint.
3. Testing after Keep the QA members in sync with the rest
development of the team. Either get testing finished and
has finished. automated before a sprint is done, or
consider outsourcing what testing and
automation efforts are left, so the QA can
move on (but can still own the testing).

Focus on your strengths

One of our clients in the education market
also happened to provide Software services.
Their QA team had subject matter expert
(SME) testers and a few technical testers.
They outsourced building and maintaining
the framework for their automation and had
the internal team focus on test case design
and data development. By focusing on their
strength and outsourcing the rest, the team
designed better tests, got significant
automation results and cut delivery time on
their first release after jump-starting an
automation program.

Learn more about Agile Testing 12

 TOP 10 WAYS TO MAKE
A SUCCESS

9. Address Testability of the System Under Test

● This should be a primary requirement for any project.
● Collaboration with programmers is essential and often welcome.

10. Test and Update the Automation

If there are run problems, don’t debug tests first. If they don’t run, make sure:
● Lower level tests run smoothly first.
● Actions and interface definitions are properly tested themselves

Summary
Efficient Agile automation is the new normal! Shifting paradigms, planning ahead, and great collaboration
are key attributes to success. It also takes hard work: by remembering some basic Agile practices and
iterating them, you can make your automation better over time. Adopting low maintenance and scalable
methods like ABT with TestArchitect is essential to building fast resilient automation. Another key is to use
retrospectives to make the process and communication better, fix roadblocks quicker and share lessons
learned.

For more information on Agile, visit our

knowledge library. You can also read the
article “Get Automated Testing Done” written
by Hans for our own LogiGear Magazine. To
learn more about how LogiGear’s automated
testing solutions can work for you, visit
Logigear.com. To get a free trial of
TestArchitect, visit TestArchitect.com now.

ABOUT LOGIGEAR
Since 1994, LogiGear has been providing consulting, manual and automated software testing, and
application development and maintenance services that enable customers to shorten their time to
market, reduce costs, and improve product quality. We’ve served hundreds of clients around the globe,
ranging from early-stage startups to Fortune 500 companies across a broad spectrum of industries.

Learn more about Agile Testing 13

 { Test Automation Pre-Checklist for
Continuous Testing and DevOps }
Before You Start

Know that Automation does not ensure quality-what automation does is

measure consistency.
Understand that your Test Automation, skill, expertise, and staff size should
increase to satisfy the need.
Your current automation cycle needs to be “same sprint” automation”.
Building up technical debt on test automation will lead to failure in DevOps.

Audit your Existing Suite i

Make sure that your suites are lean and mean. Automated tests get old.
Automation does have a shelf life. Have a good methodology and
process to remove old tests as well as maintain and add new ones.
The automated test suites need to be fast, efficient, and effective.
Examine your tests and test design-be critical and honest about value.
A team will need to redesign, optimize, or even cut the size of their
automated suites for continuous testing.

Understand how Automation Fits into Continuous Testing and DevOps

The goal of DevOps is to give fast immediate feedback on the system

health with rapid delivery of new functionality. Product stability and all
the component parts and services on various environments.
You need sophisticated, thoughtful test automation for Continuous
Testing. Simple, mindless automation will not benefit anyone. This is
especially true when your product quickly changing Service Oriented
Architecture (SOA)
A low maintenance framework is not a hope-it is the foundation of
DevOps.
You may need multiple suites for different purposes.

© LogiGear Corporation 2017. All Rights Reserved. www.logigear.com 14