Model-based Testing:

Taking BDD/ATDD to the Next Level

Robert V. Binder
System Verification Associates
rvbinder@sysverif.com

#MoreModelsLessTests

Chicago Quality Assurance Association

February 25, 2014
The Incredible Eaten Alive by the Tell Me
Vastness of Testing Backblob A Story
Testing

More Models,
Model- More
Less Tests
Based Models,
Testing Less
Tests

Model Driven Development: Taking BDD/ATDD to the Next Level 1

The Incredible
Vastness of Testing
 Saved by Revealing Sub-domains
• Grains of sand to fill the • Trivial Program
Grand Canyon – Data space: 21024
~ 1.68 × 1025 – Sequences: 21024
~ 10 septillion – Execution space: 22048
– 1.34 × 10616

Large subsets of any program’s execution space are revealing: if a bug exists in any
member of that space and it is executed, an observable failure results.

But, we rarely know exactly which sub-domains are revealing until we test.
Model Driven Development: Taking BDD/ATDD to the Next Level 3
It’s release time.
Do you know where your bugs are?

0.7 Other Agile Average bugs per

Aerospace 0.8
thousand
1.9 equivalent source
Medical 1.8
code lines,
5.0 post-release
IT 4.4
6.0
Mobile 6.0

Best In Class 4.7

Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .

Model Driven Development: Taking BDD/ATDD to the Next Level 4

It’s release time.
Do you know where your bugs are?

Other Agile
325
Aerospace 375
Predicted total
925 latent bugs,
Medical 875 500 KLOC Java

2511
IT 2202

Best In Class 2350

Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP

Model Driven Development: Taking BDD/ATDD to the Next Level 5

 Driven to Distraction
• Test Driven Development • What is it?
– TDD
• How does it work?
• Behavior Driven Development
– BDD • What can it do for
me?
• Acceptance Test Driven Development
– ATDD

• Model-based Testing
– MBT
Model Driven Development: Taking BDD/ATDD to the Next Level 6
 Eaten Alive
by the
Testing
Backblob
EATEN ALIVE BY THE TESTING
BACKBLOB
 Where does the time go?
Total Sprint 1 Sprint 2 Sprint 3 Sprint 4
Number
of Test
Cases
} Tests/testing for
new features

Existing manual
Available Test Cases not
Dev/Test executed
Time
Existing automated
Test Cases not
maintained

Model Driven Development: Taking BDD/ATDD to the Next Level 8

The Attack of the BackBlob

http://www.youtube.com/watch?v=GODDLgM1gKo
Tell Me a Story

TELL ME A STORY
 The Chat Server
• Logon
• List active users
• Post a message
• Broadcast message
• Logoff
!!!
Client Client Chat
Endpoint Endpoint Server

Model Driven Development: Taking BDD/ATDD to the Next Level 11

 TDD – BDD – ATDD
• Test Driven • Programming strategies
Development
– Test suites are scaffolding
• Behavior Driven – Test each method/feature at
Development least once

• Acceptance Test Driven – Rerun after any add/change

Development
• Not testing strategies

Model Driven Development: Taking BDD/ATDD to the Next Level 12

 The TDD Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only Logon
Logon

GetStatus
GetStatus

Logoff
Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level 13

 The Red-Green Cycle
• Imagine a class and its
responsibilities
TestLogon
• Code its API only  Logon

• Write and run a simple test – it

should fail GetStatus

Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level 14

 The Red-Green Cycle
• Imagine a class and its
responsibilities
TestLogon
• Code its API only  Logon

• Write and run a simple test – it

should fail GetStatus
• Code implementation
• Rerun all tests – expect pass Logoff

Model Driven Development: Taking BDD/ATDD to the Next Level 15

 The Red-Green Cycle
• Imagine a class and its
responsibilities
TestLogon
• Code its API only  Logon

• Write and run a simple test – it

should fail TestGetStatus

 GetStatus
• Code implementation
• Rerun all tests – expect pass Logoff

• Repeat until feature complete

Model Driven Development: Taking BDD/ATDD to the Next Level 16

 The Red-Green Cycle
• Imagine a class and its
responsibilities
TestLogon
• Code its API only  Logon

• Write and run a simple test – it

should fail TestGetStatus
 GetStatus
• Code implementation
• Rerun all tests – expect pass TestLogoff Logoff


• Repeat until feature complete

Model Driven Development: Taking BDD/ATDD to the Next Level 17

 TDD Limitations
Process, • No explicit linkage for
Activities
Components, user/customer needs,
Test Objects
requirements, or
stories
• Gaps
– UI automation
– Feature testing

Model Driven Development: Taking BDD/ATDD to the Next Level 18

 Behavior Driven Development
Components,
Process, Test Objects
Activities
Features,
Scenarios
Feature Files  Scenario A

*DD Tool  Scenario B

 Scenario C
Test Objects

*Unit

Components

Model Driven Development: Taking BDD/ATDD to the Next Level 19

Acceptance Test Driven Development
Process, UI Test
Activities Cases
Features,
Scenarios
Feature Files  Scenario A

*DD Tool  Scenario B

 Scenario C
Test Objects
Test Scripts

Test*Unit
Tooling

System
Under Test

Model Driven Development: Taking BDD/ATDD to the Next Level 20

 Features and Scenarios
Feature: story
Scenario: instance of a story
Feature: Logon
Steps: In order to use the chat server,
As a validated chat client,
I want to submit my credentials and
Given: pre-condition establish a session

When: trigger Scenario: Submit Good Credentials

Given A connection to
Then: verification condition When I send the Login
Then I should see the
the server is open
message
"Hi Chatter" message

Model Driven Development: Taking BDD/ATDD to the Next Level 21

 Overview
• Enter features, scenarios
Feature Files  Scenario A • Generates skeleton test
SpecFlow  Scenario B code for each step
 Scenario C
Test Objects
– Extracts step names
– Can run immediately
Nunit
Visual Studio • Developer adds step code
Components • Test run reports
http://www.specflow.org
• CI integration, more
Model Driven Development: Taking BDD/ATDD to the Next Level 22
 1. Create a
Feature File
• SpecFlow
generates a stub
file
• Edit as necessary
• One feature
• 1:N scenarios

Model Driven Development: Taking BDD/ATDD to the Next Level 23

 2. Enter Scenario
Steps
• Given
– Precondition
– Data State
• When
– Trigger
– API call, GUI action
• Then
– Verification
conditions
(expected results)

Model Driven Development: Taking BDD/ATDD to the Next Level 24

 3. Inspect Skeleton
Test Code

• SpecFlow parses
feature file
• Generates
skeleton test
code for each
scenario and step

Model Driven Development: Taking BDD/ATDD to the Next Level 25

 3. Add Test Case
Code

• User adds more

code as needed
to implement
each test case

Model Driven Development: Taking BDD/ATDD to the Next Level 26

 4. Run Test Suite
• The generated
test code can be
run immediately
• Supports the red-
green cycle

Model Driven Development: Taking BDD/ATDD to the Next Level 27

 SpecFlow
BDD for Visual Studio • Feature File contains statements in
• Link Feature Files to test code Gherkin
• Generates partial test code – Keywords + natural language
• 1:N Scenarios per Feature File
Feature • 1:N Steps per Scenario
• Generates test code skeleton from
steps
Feature
• Steps may be parameterized
File
– Variables and variable bindings in Rows
– Test code generation using regular
expression patterns
Scenarios Steps Rows
• Integration with TFS, CI, …

Model Driven Development: Taking BDD/ATDD to the Next Level 28

Model-
based
Testing

THE VERY MODEL OF A MODERN …

 What is a Model?
• Focus
• Smaller, simpler
• In IT systems
– Behavior
– Structure
– Content
Model Driven Development: Taking BDD/ATDD to the Next Level 30
 IT Models
Process,
Behavior, Technology
Activities
Structure,
Content

Model Driven Development: Taking BDD/ATDD to the Next Level 31

 IT Models
• Notional (design)
• As built (documentation)
• Test Models
– Similar to notional and as-built
– Unique aspects
 Test Models
• Focus on aspects that
support testing goals
• Behavior
– Sequence and time
• Transformation
– Input, output function
• Verification (oracle)
• Coverage
• Risk Management
Model Driven Development: Taking BDD/ATDD to the Next Level 33
 Chat Server
• Logon
Chat Server
– Must be first
– Can’t be repeated
Logon • GetStatus
– Only after logon
GetStatus – Returns number of calls since
logon
• Logoff
Logoff – Only after logon
• Typical RFC status codes
Model Driven Development: Taking BDD/ATDD to the Next Level 34
 Simple Test Model
nc = 0 Logoff

System Under
GetStatus
Test (SUT)
Logon Logon
Logon

GetStatus assert (401) assert (401) assert (401)

assert (200)
Logoff

nc++
Logoff GetStatus

assert (200) Logoff assert (200)

assert (n == nc)
assert (200)
Model Driven Development: Taking BDD/ATDD to the Next Level 35
 Chat Server Message Sequence
Login a 1234
Hi Chatter a
Login b secret
Hi Chatter b

List Users
Users: a, b
!!!

Client Endpoint Client Endpoint Chat Server

Model Driven Development: Taking BDD/ATDD to the Next Level 36
 Chat Server Message Sequence
List Users
Users: a, b

Post “Pumpkin Latte #mypics”

Ack a1 “Pumpkin Latte #mypics”

Bcast a1 “Pumpkin Latte #mypics”

Logoff b
Ack Logoff b
!!!
Client Endpoint Client Endpoint Chat Server
Model Driven Development: Taking BDD/ATDD to the Next Level 37
Spec Explorer Demo
• Model-based testing tool
– Microsoft Research
– Used to test all MSFT APIs
– Robust and stable
– Visual Studio “power tool”
• C# code, not cartoons
• Generates standalone
executable test suite
Model Driven Development: Taking BDD/ATDD to the Next Level 38
 1. Create Test
Model Actions
• [Rule] tags a
method for
exploration
• Requires defines
a exploration
precondition –
must be true to
explore

Model Driven Development: Taking BDD/ATDD to the Next Level 39

 2. Define Accepting
State(s)
• Tag condition(s)
that terminate
an exploration

Model Driven Development: Taking BDD/ATDD to the Next Level 40

 3. Create Cord File

• Set options
• Define machines
– Limits Actions used
in an exploration
• Similar to regular
expression
• Composable
• Very powerful!

Model Driven Development: Taking BDD/ATDD to the Next Level 41

 4. Explore the Model

• Select any
explorable machine
– Explore
– Generate test suite
– Run on-the-fly test

Model Driven Development: Taking BDD/ATDD to the Next Level 42

 5. Review
Exploration Graph
• Shows action/event
sequence
• Ellipse: model state
computed by model
program machine
• Diamond: observation
“event” – get something
from SUT (input to the
model program)
• Right click to inspect
any item

Model Driven Development: Taking BDD/ATDD to the Next Level 43

 6. Define machine(s)
to generate test
suites
• Set options to
generate test
code
• Use any machine
or composite
machine

Model Driven Development: Taking BDD/ATDD to the Next Level 44

 7. Explore the Test
Suite machine
• Each root to leaf
path becomes a
test suite

Model Driven Development: Taking BDD/ATDD to the Next Level 45

 8. Generate Test Code

• Explores the test

machine
• Writes out test
code using VS
testing
framework

Model Driven Development: Taking BDD/ATDD to the Next Level 46

 8. Generate Test Code

• Not intended to
be maintained

• Re-generate any
time

• Add helper code in

the model to
extend/customize

Model Driven Development: Taking BDD/ATDD to the Next Level 47

 9. Run the Test Suite
• Uses built-in Visual
Studio test runner
• Adapters
– Start/stop chat clients
and chat server
– Other SUT
setup/cleanup

• Test code sends

messages, checks
replies

Model Driven Development: Taking BDD/ATDD to the Next Level 48

 10. Review Test
Results
Model State
(test case inputs) • Detailed run log
– VS test runner
Event Handler – States
(pass/fail) – Events
– Requirements
Requirement
Trace
• Add any other
programmable
function
Model Driven Development: Taking BDD/ATDD to the Next Level 49
 Spec Explorer
Exploration
• Machines define composable
• Interactive model development Action sequences
• Generate test suites
Machines • Options control exploration and
data generation
Cord
Declarations • Actions are explored
Options
– Rules define exploration conditions
Model Data
Model Structures – Events define expected SUT
Program response
Rules
Actions • Data structures support exploration
• Adapters abstract, control, and
Events
observe any interface, any platform
Adapters
Model Driven Development: Taking BDD/ATDD to the Next Level 50
 MBT Driven Development
Requirements, Stories, Scenarios...
Explore Ambiguous, missing, Missing,
contradictory, incorrect, incorrect
Test Model
redundant, incomplete
Coverage
Run Generate Model error, omission Requirements
Model
Test Suite Evaluate
Inputs Expected Outputs
(Test Sequences) (Test Oracle)

Control Observe Bug

SUT
Model Driven Development: Taking BDD/ATDD to the Next Level 51
More Models, Less Tests

MORE MODELS, LESS TESTS

 Model-based Testing in the Real World
• Windows protocol compliance (Web services, RPC, DCOM)
• Total effort: 250 person years (mostly junior SDETs)
• Saved 50 person years with model-based testing

Model-based Testing
1.4 Days/Requirement 42% Less
Time Per
Traditional Testing Requirement
2.4 Days/Requirement

Requirements Study Modeling Test Coding Adapter Coding Test Execution

Model Driven Development: Taking
53
BDD/ATDD to the Next Level
Scope of Test Asset Maintenance
Model-based Testing Behavior Driven Development

Model

How much MBT-

generated test code
must be maintained? Test Code
None!

Adapters

Model Driven Development: Taking BDD/ATDD to the Next Level 54

 More Models, Less Tests
• More Better Testing • Icing the BackBlob
– Rule-based – Maintain model
– Systematic and deep – Regenerate test suites
exploration – Models grow much
– Generate 1000s of more slowly than test
scenarios in seconds code

Model Driven Development: Taking BDD/ATDD to the Next Level 55

Icing the Backblob

http://www.youtube.com/watch?v=0GEJgR-bX0w
Q&A

#MoreModelsLessTests

rvbinder@sysverif.com

Model Driven Development: Taking BDD/ATDD to the Next Level 57

 Learn More - MBT
• UCCAT (User Conference on Advanced Automated Testing) Presentations
http://ucaat.etsi.org/2013/program_conf.html
• 2012 Model-based Testing User Survey
http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf
• Open Source MBT tools
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
• Overview of MBT, list of commercial MBT tools
http://www.slideshare.net/robertvbinder/model-basedtestingignite
• Spec Explorer
http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
http://msdn.microsoft.com/library/ee620411.aspx
• Microsoft Open Protocol Project
http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!
Model Driven Development: Taking
58
BDD/ATDD to the Next Level
 Learn More: BDD, etc.
• SpecFlow
http://www.specflow.org/
• Using SpecFlow with WatiN
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx

• SpecFlow Quick Start

http://www.youtube.com/watch?v=SVwBz-ZTliw

• Writing Cucumber Feature Files

https://github.com/cucumber/cucumber/wiki

• Using Lava Lamps to indicate Build Status

https://wiki.jenkins-ci.org/display/JENKINS/Hudson+Build+Status+Lava+Lamps

Model Driven Development: Taking

59
BDD/ATDD to the Next Level
 Content and Image Sources
Grand Canyon Time Lapse. Ezra Kamer
http://www.youtube.com/watch?v=NF6O6HuNJ0k
The Blob, Theatre Scene. Fairview Productions, 1958.
http://www.youtube.com/watch?v=GODDLgM1gKo
http://www.imdb.com/title/tt0051418/
Defect densities:
• Capers Jones, Software Quality: Analysis and Guidelines for Success
• Robert E. Park. Software Size Measurement: A Framework for
Counting Source Statements. Technical Report CMU/SEI-92-TR-020.
Software Engineering Institute, Carnegie Mellon University
SpecFlow
http://www.specflow.org/
All other content Copyright © Robert V. Binder, 2014.
Model Driven Development: Taking BDD/ATDD to the Next Level
Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of
Canada
http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270
Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada
http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980
Spec Explorer. Microsoft
http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-
956b-d9bfa4902745
Spec Explorer Chat Model. Microsoft.
The Blob, Final Scene. Fairview Productions, 1958.
http://www.youtube.com/watch?v=0GEJgR-bX0w
60