Software Testing

ISEB Foundation Certificate Course

1 Principles

2 Lifecycle

4 Dynamic test
5 Management
techniques

3 Static testing
6 Tools

Dynamic Testing Techniques

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing

Why dynamic test techniques?

Exhaustive testing (use of all possible inputs

and conditions) is impractical
- must use a subset of all possible test cases
- must have high probability of detecting faults
Need thought processes that help us select
test cases more intelligently
- test case design techniques are such thought
processes

What is a testing technique?

a procedure for selecting or designing tests

based on a structural or functional model of
the software
successful at finding faults
'best' practice
a way of deriving good test cases
a way of objectively measuring a test effort
Testing
Testing should
should be
be rigorous,
rigorous, thorough
thorough and
and systematic
systematic

Advantages of techniques

Different people: similar probability find faults

- gain some independence of thought
Effective testing: find more faults
- focus attention on specific types of fault
- know you're testing the right thing
Efficient testing: find faults with less effort
- avoid duplication
- systematic techniques are measurable
Using
Using techniques
techniques makes
makes testing
testing much
much more
more effective
effective

Measurement

Objective assessment of thoroughness of

testing (with respect to use of each technique)
- useful for comparison of one test effort to another
E.g.
Project A
60% Equivalence
partitions
50% Boundaries
75% Branches

Project B
40% Equivalence
partitions
45% Boundaries
60% Branches

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing

Three types of systematic technique

Static (non-execution)
examination of documentation,
source code listings, etc.
Functional (Black Box)
based on behaviour /
functionality of software
Structural (White Box)
based on structure
of software

Some test techniques

Dynamic
Dynamic

Static
Static
etc.

Reviews
Inspection
Walkthroughs

Structural
Structural

Desk-checking

Symbolic
Execution

Non-functional
Non-functional

Functional
Functional
etc.

etc.
Control
Flow

Data
Flow

etc.

Behavioural
Behavioural

Static Analysis

Performance
etc.

Statement
Branch/Decision

Definition
-Use

Branch Condition

Equivalence
Partitioning

Usability

Arcs
LCSAJ

Branch Condition
Combination

Boundary
Value Analysis
Cause-Effect Graphing
Random

State Transition

Black box versus white box?

Black box appropriate
at all levels but
dominates higher
levels of testing
White box used
predominately
at lower levels
to compliment
black box

Acceptance
Acceptance

System
System

Integration
Integration

Component
Component

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing

Black Box test design and

measurement techniques

Techniques defined in BS 7925-2

- Equivalence partitioning
Also a measurement
- Boundary value analysis
technique?
- State transition testing
= Yes
= No
- Cause-effect graphing
- Syntax testing
- Random testing
Also defines how to specify other techniques

Equivalence partitioning (EP)

- divide (partition) the inputs, outputs, etc. into areas

which are the same (equivalent)
- assumption: if one value works, all will work
- one from each partition better than all from one
invalid

valid
0

100

invalid
101

Boundary value analysis (BVA)

- faults tend to lurk near boundaries

- good place to look for faults
- test values on both sides of boundaries
invalid

valid
0

invalid
100 101

Example: Loan
application
Customer Name
Account number
Loan amount requested
Term of loan
Monthly repayment
Term:
Repayment:
Interest rate:
Total paid back:

2-64 chars.
6 digits, 1st
non-zero
500 to 9000
1 to 30 years
Minimum 10

Customer name
Number of characters:

invalid
Valid characters:

Conditions
Customer
name

Valid
Partitions
2 to 64 chars
valid chars

valid

A-Z
- a-z
space
Invalid
Partitions
<2 chars
>64 chars
invalid chars

64 65

invalid

Any
other
Valid
Boundaries
2 chars
64 chars

Invalid
Boundaries
1 chars
65 chars
0 chars

Account number
first character:

valid: non-zero
invalid: zero

number of digits:
invalid

Conditions
Account
number

Valid
Partitions
6 digits
1st non-zero

Invalid
Partitions
<6 digits
>6 digits
1st digit =0
non-digit

valid

Valid
Boundaries
100000
999999

invalid

Invalid
Boundaries
5 digits
7 digits
0 digits

Loan amount

499
invalid
Conditions
Loan
amount

Valid
Partitions
500 - 9000

500

9000
valid

Invalid
Partitions
<500
>9000
0
non-numeric
null

9001
invalid

Valid
Boundaries
500
9000

Invalid
Boundaries
499
9001

Condition template
Conditions

Valid
Partitions

Tag

Customer 2 - 64 chars
name
valid chars

V1
V2

Account
number

6 digits
1st non-zero

V3
V4

Loan
amount

500 - 9000

V5

Invalid
Partitions

Valid
Invalid
Tag
Tag
Boundaries
Boundaries
< 2 chars
X1 2 chars
B1 1 char
D1
> 64 chars
X2 64 chars
B2 65 chars
D2
invalid char X3
0 chars
D3
< 6 digits
X4 100000
B3 5 digits
D4
> 6 digits
X5 999999
B4 7 digits
D5
st
1 digit = 0 X6
0 digits
D6
non-digit
X7
< 500
X8 500
B5 499
D7
>9000
X9 9000
B6 9001
D8
0
X10
non-integer X11
null
X12
Tag

Design test cases

Test
Case

Description

Expected Outcome

New Tags
Covered

Name:
Acc no:
Loan:
Term:

John Smith
123456
2500
3 years

Term:
Repayment:
Interest rate:
Total paid:

3 years
79.86
10%
2874.96

V1, V2,
V3, V4,
V5 .....

Name:
Acc no:
Loan:
Term:

AB
100000
500
1 year

Term:
Repayment:
Interest rate:
Total paid:

1 year
44.80
7.5%
537.60

B1, B3,
B5, .....

Why do both EP and BVA?

If you do boundaries only, you have covered all the

partitions as well
- technically correct and may be OK if everything works
correctly!
- if the test fails, is the whole partition wrong, or is a
boundary in the wrong place - have to test mid-partition
anyway
- testing only extremes may not give confidence for typical
use scenarios (especially for users)
- boundaries may be harder (more costly) to set up

Test objectives?
Condition

Valid
Tag Invalid
Tag Valid
Partition
Partition
Boundary

Tag Invalid
Boundary

Tag

For a thorough approach: VP, IP, VB, IB

Under time pressure, depends on your test objective
- minimal user-confidence: VP only?
- maximum fault finding: VB first (plus IB?)

State Transition Testing (analysis)

Uses a model that shows:

- states the software may occupy
- transitions between the states
- events which cause the transitions
- actions that result from the transitions
Card inserted
Ask for PIN
Wait for
card

Cancel
Return card

Wait for
PIN

Invalid PIN
Beep

Valid PIN
Ask amount

State Transition Testing (design)

Test cases designed to achieve required

coverage:
Limitation of
- state transitions (0-switch)
switch coverage:
- transition pairs (1-switch)
covers only
- transition triples (2-switch)
valid transitions
- etc.
A more complete test set will test for possible
invalid transitions
- use state table to identify invalid transitions

State machine for display_changes

reset (R)
alter time (AT)
Display
Time (S1)
(CM)
change mode
display date
(D)

set (S)
display time (T)

Change
Time (S3)

change mode (CM)

display time (T)

Display
Date (S2)

reset (R)
alter date (AD)
set (S)
display date (D)

Change
Date (S4)

Possible transitions
Transition

Display
Time (S1)
CM/T

R/AT
S/T

Change
Time (S3)

CM/D

Display
Date (S2)

R/AD
S/D

Change
Date (S4)

Start Event/ End

State Action State

S1

CM/D

S2

S2

CM/T

S1

S1

R/AT

S3

S3

S/T

S1

S2

R/AD

S4

S4

S/D

S2

Test case for transition coverage

Display
Time (S1)
CM/T

R/AT
S/T

Change
Time (S3)

CM/D

Display
Date (S2)

R/AD
S/D

Change
Date (S4)

Step State Event

Action

S1

Reset

Alter T

S3

Set

Display T

S1

S2

Reset

Alter D

S4

Set

Display D

S2
S1

Transitions: 1 2 3 4 5 6

Chgmd Display D

Chgmd Display T

Transition pairs
Transition Start
Pair
State
Display
Time (S1)
CM/T

R/AT
S/T

Change
Time (S3)

CM/D

Display
Date (S2)

R/AD
S/D

Change
Date (S4)

1
2
3
4
5
6
7
8
9
10

S1
S1
S2
S4
S4
S2
S2
S1
S3
S3

Next
State

End
State

S2
S2
S4
S2
S2
S1
S1
S3
S1
S1

S1
S4
S2
S4
S1
S2
S3
S1
S3
S2

- covered by first test case

State table for display_changes

CM

S1

S2/D

S3/AT

S1/N

S2

S1/T

S4/AD

S2/N

S3

S3/N

S3/N

S1/T

S4

S4/N

S4/N

S2/D

Invalid
transitions

N denotes
no output

Test case design considerations

keep in mind need to confirm correct result

- particularly where transitions use same action
- test support code may be necessary
start with functionally sensible test cases
covering most likely transitions
- provides a good regression test set
add more complex test cases to cover
exceptional conditions
- invalid transitions

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing

White Box test design and

measurement techniques

Techniques defined in BS 7925-2

- Statement testing
Also a measurement
technique?
- Branch / Decision testing
= Yes
- Data flow testing
= No
- Branch condition testing
- Branch condition combination testing
- Modified condition decision testing
- LCSAJ testing
Also defines how to specify other techniques

Using structural coverage

Spe
Spe
cc

Tests

Enough
tests?

Software
Software

Results OK?

More tests

What's
covered
?

Coverage OK?

Stronger structural
techniques (different
structural elements)
Increasing coverage

The test coverage trap

better
better testing
testing

Function
Function exercised,
exercised,
insufficient
insufficient structure
structure
Functional
testedness

Structure
Structure exercised,
exercised,
insufficient
insufficient function
function
% Statement

% Decision

Structural testedness

100%
100% coverage
coverage does
does
not
not mean
mean 100%
100% tested!
tested!

% Condition
Combination

Coverage
Coverage isis not
not
Thoroughness
Thoroughness

Statement coverage

Statement
Statement coverage
coverage
is
is normally
normally measured
measured
by
by aa software
software tool.
tool.

percentage of executable statements

exercised by a test suite
number of statements exercised
=
total number of statements
example:
- program has 100 statements
- tests exercise 87 statements
- statement coverage = 87%

Typical
Typical ad
ad hoc
hoc testing
testing achieves
achieves 60
60 -- 75%
75%

Example of statement coverage

1

read(a)

IF a > 6 THEN

b=a

ENDIF

print b

Statement
numbers

Test
case

Input

Expected
output

As all 5 statements are covered by

this test case, we have achieved
100% statement coverage

Decision coverage
(Branch coverage)

Decision
Decision coverage
coverage
is
is normally
normally measured
measured
by
by aa software
software tool.
tool.

percentage of decision outcomes

exercised by a test suite
number of decisions outcomes exercised
False
?
=
total number of decision outcomes
True
example:
- program has 120 decision outcomes
- tests exercise 60 decision outcomes
- decision coverage = 50%

Typical
Typical ad
ad hoc
hoc testing
testing achieves
achieves 40
40 -- 60%
60%

Paths through code

1234

12

12

123

?
?

Paths through code with loops

1 2 3 4 5 6 7 8 .

for as many times as it

is possible to go round
the loop (this can be
unlimited, i.e. infinite)

Example 1
Wait for card to be inserted
IF card is a valid card THEN
display Enter PIN number
IF PIN is valid THEN
select transaction
ELSE (otherwise)
display PIN invalid
ELSE (otherwise)
reject card
End

Wait

Valid
card?

Yes

Display
Enter..

No

Reject
card

Valid Yes
PIN?
No

Select
trans...

Display
PIN in..

End

Example 2
Read A
IF A > 0 THEN
IF A = 21 THEN
Print Key
ENDIF
ENDIF

Read
A>0

Yes

No

End

3
- Cyclomatic complexity: _____
- Minimum tests to achieve:
1
Statement coverage: ______
3
Branch coverage: _____

A=21
No

Yes

Print

Example 3
Read A
Read B
IF A > 0 THEN
IF B = 0 THEN
Print No values
ELSE
Print B
IF A > 21 THEN
Print A
ENDIF
ENDIF
ENDIF

Read
A>0
No

Yes

B=0

Yes

Print

No

Print
Yes
A>21

Print

No

End

4
- Cyclomatic complexity: _____
- Minimum tests to achieve:
2
Statement coverage: ______
4
Branch coverage: _____

Example 4

Read

A<0

Yes

Print

No

Print
Note: there
Read A
Read B
are 4 paths
IF A < 0 THEN
Yes
B<0
Print
Print A negative
No
ELSE
Print A positive
Print
ENDIF
IF B < 0 THEN
End
Print B negative
ELSE
3
- Cyclomatic complexity: _____
Print B positive
- Minimum tests to achieve:
ENDIF
2
Statement coverage: ______
2
Branch coverage: _____

Read

Example 5

A<0

Yes

Print

No

Read A
Read B
IF A < 0 THEN
Print A negative
ENDIF
IF B < 0 THEN
Print B negative
ENDIF

B<0

Yes

Print

No

End

3
Cyclomatic complexity: _____
Minimum tests to achieve:
1
Statement coverage: ______
2
Branch coverage: _____

Read

Example 6

A<0

Yes

Print

No

Read A
IF A < 0 THEN
Print A negative
ENDIF
IF A > 0 THEN
Print A positive
ENDIF

A>0

Yes

Print

No

End

3
Cyclomatic complexity: _____
Minimum tests to achieve:
2
Statement coverage: ______
2
Branch coverage: _____

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing

Non-systematic test techniques

Trial and error / Ad hoc

Error guessing / Experience-driven
User Testing
Unscripted Testing

AAtesting
testing approach
approach that
that isis only
only
rigorous,
rigorous, thorough
thorough and
and systematic
systematic
isis incomplete
incomplete

Error-Guessing

always worth including

after systematic techniques have been used
can find some faults that systematic
techniques can miss
a mopping up approach
supplements systematic techniques
Not
Not aa good
good approach
approach to
to start
start testing
testing with
with

Error Guessing: deriving test cases

Consider:
- past failures
- intuition
- experience
- brain storming
- What is the craziest thing we can do?

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Summary: Key Points

Test techniques are best practice: help to find faults
Black Box techniques are based on behaviour
White Box techniques are based on structure
Error Guessing supplements systematic techniques