Professional Documents
Culture Documents
Mark Fewster
Lloyd Roden
Clive Bates
Grove House
40 Ryles Park Road
Macclesfield
SK11 8AH UK
Llwyncynhwyra
Cwmdu
Llandeilo
SA19 7EW UK
95 Stonebridge
Orton Malborne
Peterborough
PE2 5NT UK
2 Nursery Close
Hilperton Village
Wiltshire
BA14 7RP UK
www.grove.co.uk
Grove Consultants
Contents
www.grove.co.uk
011 44 8702 406172
Page 2
objectives
- to find defects
- to give confidence
useful,
- to assess risk
easy
to do
measures
- defect-based measures
found (Defect Detection Percentage)
fixed (Defect Fix Percentage)
not covered here
but can be useful
- confidence-based measures
(plus other metrics)
- risk-based measures
www.grove.co.uk
011 44 8702 406172
Page 3
or
Start
Defects found
afterwards
Release
Not found
- yet
Benchmark point
www.grove.co.uk
011 44 8702 406172
Page 4
Defects
Found
100%
90%
80%
70%
60%
DDP
50%
40%
30%
20%
10%
0%
50
45
40
35
30
25
20
15
10
5
0
Release
Time
Defects found
Defects found
Total defects
50
10 after testing: 38
12
19
24
27
35
37
0
50
62
69
74
77
85
87
in testing: 42
found: 88
DDP =
50
= 57
81 %
72
68
65
59
62
69
74
77
85
87
88
www.grove.co.uk
011 44 8702 406172
Page 5
Contents
DDP example
Testing
Live
Running
DDP
after
Live
150
50
75%
DDP =
150
150 + 50
150
200
75%
www.grove.co.uk
011 44 8702 406172
DDP Exercise 1
DDP Exercise 1
The following data has been recorded for a project.
Calculate the DDP of each testing stage based on all the defect information.
Fault Information
Testing stage
Number of
faults
DDP
299
40
Release testing
19
10
Pilot
20
Hint: you dont really need a calculator just round the numbers to the nearest 10 and you will be
close enough!
Grove Consultants
Page 1 of 2
DDP Exercise 1
DDP Exercise 1 Solution: DDP Calculation
Testing stage
No. faults
299
40
3) Release testing
19
DDP
75%
300 / 400
40%
40 / 100
33%
20 / 60
10
25%
10 / 40
5) Pilot
20
n/a
Grove Consultants
Page 2 of 2
Page 6
defects
Percentage
299
75%
40
10%
Release testing
19
5%
10
2.5%
2.5%
20
5%
Pilot
Live use (1 mo.)
100%
Source: Client
defects
Percentage
299
75%
40
40%
Release testing
19
33%
10
25%
31%
Pilot
Live use (1 mo.)
20
live use determines DDP
www.grove.co.uk
011 44 8702 406172
Page 7
UAT
100
50
Live
UAT
DDP
after
Live
All test
DDP
after
Live
67%
100
DDP =
ST DDP ST DDP
after
after
UAT
Live
100 + 50
100
150
67%
UAT
Live
100
50
100
DDP =
100
100 + 50 + 100
ST DDP ST DDP
after
after
UAT
Live
67%
UAT
DDP
after
Live
All test
DDP
after
Live
40%
100
250
40%
www.grove.co.uk
011 44 8702 406172
Page 8
UAT
Live
100
50
100
50
DDP =
50 + 100
ST DDP ST DDP
after
after
UAT
Live
67%
40%
50
150
UAT
DDP
after
Live
All test
DDP
after
Live
33%
33%
UAT
Live
100
50
100
DDP =
100 + 50
100 + 50 + 100
ST DDP ST DDP
after
after
UAT
Live
67%
40%
150
250
UAT
DDP
after
Live
All test
DDP
after
Live
33%
60%
60%
www.grove.co.uk
011 44 8702 406172
DDP Exercise 2
DDP Exercise 2
The following data has been recorded for a project.
Calculate the DDPs in the columns on the right. The first one has been done as an example.
(A calculator may be useful for some of these your mobile phone has one!)
Fault Information
Release
System Test
Live running
(1 month)
Release 1
100
50
100
Release 2
150
50
10
Release 3
200
50
50
Release 4
50
25
125
ST DDP
after UAT
ST DDP
after LR
67%
40%
UAT DDP
after LR
33%
60%
Grove Consultants
Page 1 of 2
DDP Exercise 2
DDP Exercise 2 Solution: DDP Calculations
Release
System Test
Live running
(1 month)
ST DDP
after UAT
ST DDP
after LR
UAT DDP
after LR
Release 1
100
50
100
67%
40%
33%
60%
Release 2
150
50
10
75%
71%
83%
95%
Release 3
200
50
50
80%
67%
50%
83%
Release 4
50
25
125
67%
25%
17%
38%
Grove Consultants
Page 2 of 2
Page 9
Contents
10 mo
Year 1
70%
50% est
Year 2
92%
Finance (insurance)
Operating system
Scientific software
(chemical analysis)
www.grove.co.uk
011 44 8702 406172
Page 10
Information Technology
RESULTS SO FAR
E ffe c tiv e n e s s o f T e s tin g
100
90
80
S y s te m T e s t
UAT
70
60
Skynet
Alibi/Claims
CARRS Release 4
ACRaSh v1.2
IMR Pre-Screening
IBIS Release 3
CARRS Release 3
40
BWS Phase 1b
50
Information Technology
MESSAGES
Conclusions
UAT more variable than ST mainly personnel
Target zone for ST : 75 - 90%
Factors behind the figures
size, complexity, tester experience, time, documentation
whether UAT started before ST was finished
20
15
10
o r h e re ?
14/12/01
13/12/01
12/12/01
11/12/01
cost of finding
10/12/01
09/12/01
s to p p e d h e re ?
25
08/12/01
30
07/12/01
Page 20
www.grove.co.uk
011 44 8702 406172
Page 11
Project
or App.
Months
DDP
DDP Status
ESTIMATED
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
FINAL
NEW
Comments
Major re-engineering
Reporting System
Eur impl. of US system
Enhancement Release
Global Enhancements
Global Enhancements
Rolling DDP
Software Testing Defect Detection Percentage Measure
(rolling quarterly produced values looking back four quarters)
# Projects
Analysed
Historical Estimate
Rolling 1 Qtrs DDP to Q1 Y1
Target
n/a
n/a
n/a
n/a
n/a
80
80
80
80
90
90
90
90
Defects in
Testing
Total
Defects
Prod'n
Bugs
1111
1400
289
1171
1466
295
1211
1508
297
1492
1807
315
2034
2129
95
1974
2063
89
2086
2204
118
1976
2087
111
DDP
50
79
80
80
83
96
96
95
95
?
?
?
?
www.grove.co.uk
011 44 8702 406172
Page 12
Contents
DDP benefits
www.grove.co.uk
011 44 8702 406172
Page 13
DDP limitations
www.grove.co.uk
011 44 8702 406172
Page 14
communication
- DDP is a communication tool, giving information about the
testing that has been done
dealing with misconceptions
- very important that those who see DDP understand what it is
and what it means (and doesnt mean)
interpreting results
- the measure will raise questions to be answered
can be used to predict defects not found yet
- if DDP is stable, using current test defect numbers
- needs caution
www.grove.co.uk
011 44 8702 406172
Page 15
starting to measure
- start by calculating DDP of a release that is now live
getting data from live running
- if you dont normally have live defect data,
ask for it / search it out
what DDP to measure first
- most people start with System Test
- consider looking at highest severity only to start
ease of data collection & calculation
- ideally should be automatic and painless
- can use test management tool or defect tracking tool
www.grove.co.uk
011 44 8702 406172
Page 16
www.grove.co.uk
011 44 8702 406172
Page 17
What defects?
- simplest: all test defects / all defects so far
- by severity level
How "deep" to go?
- deeper levels give more detailed information
- deeper levels more complex to measure
Start simple
- simple information is much better than none
- learn from what information you have
Conclusion
www.grove.co.uk
011 44 8702 406172