Testing within an agile environment

Chris Gollop & Beyza Sakir

Overview

History
Changing Testing Concerns
Adaptations
Future
Conclusion
Further Reading
History what we do

Release every 2 weeks
Trade $billions/day
Average 0.4ms execution time for MTF members
10,000 orders second
1 tester per 5-6 developers
Very low rate of production issues

History - company

Conceived as an Agile company
Agile has matured but were still adapting
Agile testing:
Acceptance Tests
Integration Tests
Exploratory Testing

History - company

This has allowed us to experiment and adapt:
Time freed up
Quick feedback on changes
Historic data available to analyse
History test team

4 test specialists with different strengths complement each other as a 'team
Rotate through all teams
No Test Manager
Part of development team though test interests have a separate iteration
meeting
Testing is responsibility of everyone
History team structure
l
Team One Team Three

Team Two
Test Specialists
Business Specialists
Development Specialists
History - innovate

Keep changing, even if something appears to work there may be
something better

If I had asked people what they wanted, they would have said faster
horses. I gave them a car (Henry Ford)
History measure and feedback

Measure
To know if it is effective
Many monitoring and measuring tools

Feedback
Tying feedback to ideas
Quicker this is the faster you can confidently adapt
Can be numerical or social
Changing Testing Concerns Over Time

Traditional Testing Maturity Models start at ad-hoc and move through
structured testing before finishing at Continuous Improvement

Some of our test concerns (over time)
Decrease Manual Testing & increase automated Acceptance Tests
Performance
DSL
Speed of Feedback
Increasing number of Edge Cases as system becomes more complex
Changing Testing Concerns Over Time

9,000 Behavioural Tests
8,000 Acceptance Tests (UI, API)
1,000 Integration Tests (subsets of services using internal interfaces)

Acceptance Test would test a rejection
Integration Tests would test the different rejection types
Changing Testing Concerns Over Time

Was right at the time as needed Acceptance Tests to get good coverage
quickly, trade-off is they are slower to run repeatedly.

Focus is now quicker feedback.
Adaptations Fitness Landscape
Adaptations Fitness Landscape

Each square is similar to it's neighbour
Higher it is the better the solution valleys are bad, mountaintops are good

You don't fully know where you are on the landscape
You feel your way by trying out new ideas and recognising failure
You may be half-way up the mountain rather than at the top
And the mountains move!
Some peaks move faster than others:
Google v McDonalds

Evolution isn't just biological it's everywhere

Adaptations Palchinsky Principles

Peter Palchinsky was a Russian engineer who advised on (against) Lenin Dam and
Magnitogorsk large projects which essentially failed due to being too big and
unable to adapt.

Palchinsky Principles are shaped to encourage stronger innovation, better leadership
and more effective policies:
Variation - seek out new ideas and try new ideas
Survivability - when trying something new do it on a scale where failure is
survivable
Selection - seek out feedback and learn from mistakes as you go along, avoid
an instinctive reaction of denial

Adaptations Recognise failure quickly

Being able to recognize a failure means you can re-cast it into something more likely
to succeed.

Accept something is failing, dont deny or ignore it.

"Hard to see what changes would improve current process when theyre currently
sweeping all problems under the our context rug." (Anthony Green)

The essential first step is being willing to fail.

Adaptations - Google v Apple

Both considered as innovative companies.

Apple innovates in things that are core to its business
Google innovates on things that indirectly contribute to its business

Apple releases products that are polished and well-scoped
Google releases many different products (Buzz, Wave, Docs, AppEngine) early in
their development and lets the community develop and adapt them.

Apple create new mountains on the fitness landscape
Google is surfing along on, or near, the moving fitness landscape mountain-top by
adapting its strategy and offerings.


Adaptations a brief history of what we have tried
Adaptations what worked

Production Data
Testing in Live
Big Feedback
Intermittent Tests
Tester Pairing
Adaptations Sanitised Production Data

Raw data (database)
Patterns of use
Data profile patterns

Migration (automated)
Real world data
Larger than we can create
Many edge cases
Try and reincorporate into our Acceptance Tests
Performance
Staging

Progression in what and how we use the data

Adaptations Testing in Live

Uses same hardware and code as it is just an isolated venue
Runs a subset of the Acceptance Tests

When we expanded our Exchange by adding a multi-tenanted concept
known as a venue we took advantage for testing

Verifies that the various components of the exchange are hooked up and
configured correctly and that the deployment process worked correctly

Runs automatically throughout the day as one of our monitoring systems
Adaptations Big Feedback
Adaptations Big Feedback

Important CI pipeline builds at the top
Red (failed) items rise to the top
Continuously and easily expanded
Useful features survive
Failure ownership/failures at a glance - AutoTrish
Intermittent tests - Magic Eight Ball
Auto Embargo
Performance metrics
Build sheriff to keep check-in embargo

Adaptations Intermittent Tests

Tests that can fail even if feature is working
Have 1 or 2 intermittent tests per run (out of 8,200)
Each about 1 test run in 20

Hides genuine failures
Have previously missed a release as couldn't tell if genuine

Can result from
Asynchronous messaging
Tests changing global settings/lack of isolation
End of day transitions
Date/time


Adaptations Big Feedback
anon
anon
Adaptations Intermittent Tests

All Failures run twice
Fail twice = failure
Fail once, pass once = intermittent
Accepted as something we need to deal with

Work to remove
DSL
Fix a genuine issue (race condition)
End of iteration is given over to removing intermittency
Build Sheriff also works on this
Adaptations Intermittent Tests
Adaptations Tester Pairing
Each tester has their own strengths

Noticed that at rotation handover the new tester generally found a bug
Issue with the testers (denial)
Something we can harness (acceptance)

Already do development pairing (Dev & Dev and Dev & Tester)

When do we pair?
At the right time for the story
Initial brainstorming when creating a mindmap
When we feel we need to (pull)
When someone has time (push)

Adaptations mindmap


Adaptations what didn't work

Not many absolute failures

Although many reasons it can fail

Even if something fails you learn something

Natural selection will continue

You can't stay still, unless you are improving you are losing ground as
others will innovate or adapt

Adaptations facts

Of the top 25 industrial corporations in the United States in 1900, only two
remained on that list at the start of the 1960s. And of the top 25 companies
on the Fortune 500 in 1961, only six remain there today (IBM)

Average life expectancy of a Fortune 500 company has declined from
around 75 years half a century ago to less than 15 years today, and heading
towards 5 years (Deloitte)

Future
Expanding the way we test

Speed up feedback
Increasing integration test coverage
Use PhantomJS for UI Acceptance Tests

Improving feedback/communication
Test team standups

Documentation
What is just enough
How to get people up to speed in a complex system
Future

Is about the people

Team make-up is varied and we recruit to keep it this way

Testing is a strictly formal process that by its nature should not vary - this is
a typical belief by people who know nothing about testing. (James Bach).

Conclusion

Need to experiment and adapt to stay ahead

Even if something works it does not mean there isn't something better

Make changes on a scale where failure can be survived

Do not be afraid of failure but know how to recognise it quickly
Further Reading

Tim Harford
Richard Wiseman
Malcolm Gladwell
Daniel Kahneman
Dan Ariely
Richard Thaler

LMAX Staff blogs - http://blogs.lmax.com/staff-blogs/