Defect Metrics

Defect Management &
Metrics
Society for Software Quality
April 24, 2010
021209.01 SSQ Presentation
2010 Holl Consulting
Who Is This Guy?
30 years in software
and hardware/software
Quality
19 years in Quality &
Process Management
6 years as Director of
Software Best Practices
and Corporate Software
Metrics Leader at Intuit
Past SSQ President
L
S
Primary Relevant Experience
Designed Intuits corporate defect
management process
Assisted in rollout to ~1800 engineers
Guided its evolution for six years
As Corporate Software Metrics Leader,
fostered quantitative project management
and process improvement
Made mistakes and learned from them
Agenda
Summary of Previous Talk
Defect Management Metrics
Metrics for Project Management
Metrics for Process Improvement
Advanced Metric Models
Last time
We reviewed cultural issues related to
defect management
We went over a defect management
process:
Policies
Workflow/States
Fields
Data Collection
At different states in the workflow, different
data are collected
Most data are collected at submit time
Additional data are entered by the
developer upon resolution
And finally by the tester upon verification
Agenda
When Does A Defect Count?
It depends on your goals
Defect Management: To track defects for resolution in
order to release products
Just log functional problems found in test; deviations from the
requirements that the customer will see; defects that you dont
want to fall through the cracks.
Defect Analysis: To improve the process or predict
If your goal is to get better (learn) or predict the future, you need
to capture much more information.
Without early lifecycle data (requirements, etc.) predictive
models arent that reliable
World-class companies dont just manage defects they analyze information.
Data Collection Costs
Estimate Assumptions:
18,000 defects
the defects are corrected (9,000)
Data collection time: 10 minutes (validated at HP)
Average defect cost: 16 hours (data from Michael Fagan)
Data collection reduces defects 5%
Time to record (18k *10m) 3,000 hrs
Time to correct (9k *16h) 144,000 hrs
Defect reduction (9k * 5%) 450 defects
Time savings (450 * 16h) 7,200 hrs
ROI 240%
Sampling is valid, but
sampling has to be
managed to get good
random samples and to
ensure that adequate
(statistically significant)
samples are being
collected.
This doesnt quantify the value of being able to predict quality and schedules
Simple Measures
Defect cost =

Fix rate =

Duration =

Last two include queue time, so they depend on workload
Its helpful to understand the standard deviation too
Bonus: By type, root cause, injection phase
* Excel NetWorkDays is a useful function
(Total Resolution Effort)
Number of resolved defects
Period of time (week, month, etc.)
((Closed Date) (Submit Date))*
Relative Defect Costs
If you record injection and discovery
phases and effort, you can compute
relative costs
Example:
Cost of a requirements defect = 1
Cost of a design defect = 10
Cost of a coding defect = 100
Cost of a production defect = 1000
Similar to Phase Containment
Start with matrix of injection and discovery
Enter the average rework (or duration)
costs in each cell (resolved defects only)
Injection Phase
Discovery
Phase
Req's Design Code Test Production
Requirements 2.4
Design 15.3 4.1
Code 22.7 9.5 1.2
Testing 43.8 18.5 4.3 1.2
Production 85.3 38.5 18.3 2.2 1
Similar to Phase Containment
Normalize by dividing each cell by the
Injection=Discovery value
Injection Phase
Discovery
Phase
Req's Design Code Test Production
Requirements 1.00
Design 6.38 1.00
Code 9.46 2.32 1.00
Testing 18.25 4.51 3.58 1.00
Production 35.54 9.39 15.25 1.83 1.00
Logging Defects
This is not
How many defects leak?
This is
What percent of known defects are recorded?
If you only start logging defects in test, youre
missing over half of your data
Try to capture data from reviews/inspections and
developer unit tests
Prioritizing Defects
This is not about how to set priority, but
Priority means the most important defects get
resources first
To measure this, we need to know the queue
time (from submit to work starts)
Therefore, we need an Accepted state, or
something to note when work began
Why cant we use Submit to Verify duration?
Aging Defects
Time in Resolver queue
Time in Verifier queue
Alternately, use
durations minus
effort as a proxy
Average Queue Time
0
2
4
6
8
10
12
0-Test
Stopper
1 2 3 4 5
Priority
D
a
y
s

i
n

Q
u
e
u
e
Defect Backlog
Many groups dont count defects left over from
prior releases; they only focus on defects found
and not fixed in the current project
This is dangerous because a product defect
backlog can grow unmanaged
ALL of the open defects are released, not just
the new ones
When producing the report of open defects,
query for all open defects in the product
The Calculation
Week
1
2
3
4
Total number of defects
Number closed before the week started
Number that were opened after the week ended
= Number open in the week
Calculating Backlog in Excel
Query Submit Date and Closed Date
Calculating Backlog in Excel
This uses named ranges, where
D.Num.CRs = Total number of CRs opened
D.T.Closed = Range of Closed Dates
D.Submit.Date = Range of Submit Dates
Array formula. <Shift-Ctrl-Return>
The Result
Product Defect Backlog
0
200
400
600
800
1000
1200
1
/
7
/
2
0
0
6
2
/
7
/
2
0
0
6
3
/
7
/
2
0
0
6
4
/
7
/
2
0
0
6
5
/
7
/
2
0
0
6
6
/
7
/
2
0
0
6
7
/
7
/
2
0
0
6
8
/
7
/
2
0
0
6
9
/
7
/
2
0
0
6
1
0
/
7
/
2
0
0
6
1
1
/
7
/
2
0
0
6
1
2
/
7
/
2
0
0
6
1
/
7
/
2
0
0
7
2
/
7
/
2
0
0
7
3
/
7
/
2
0
0
7
4
/
7
/
2
0
0
7
5
/
7
/
2
0
0
7
6
/
7
/
2
0
0
7
7
/
7
/
2
0
0
7
8
/
7
/
2
0
0
7
9
/
7
/
2
0
0
7
1
0
/
7
/
2
0
0
7
1
1
/
7
/
2
0
0
7
1
2
/
7
/
2
0
0
7
1
/
7
/
2
0
0
8
2
/
7
/
2
0
0
8
3
/
7
/
2
0
0
8
4
/
7
/
2
0
0
8
#

O
p
e
n

D
e
f
e
c
t
s
Agenda
Some Project Management Defect Metrics
Reviews & Inspections
Defect Discovery & Closure
Defect Density
A Project Axiom
The quality of product in one
phase is an indicator of the
quality in later phases
Why?
There are two answers
Review Metrics
Reviews & inspections have
proven to be the most cost-
effective way to detect and
remove defects
Inspect your key project artifacts
(requirements, designs, code, tests, etc.)
Review/Inspection Data
There are many data collected during a
review or inspection
The ones that tell us about quality are:
Number of Operational defects discovered
Number of Minor defects found
per changed line of inspected doc/code
Number of reviews per doc/module
Project Defect Density
Consistency in categorizing inspection
defects is important
Have clear guidelines for Operational and
Minor defects
Defect Density
Project
=
Operational Defects found by inspection
Delta SLOC (or LOT)
Projecting Defect Leakage
Inspected total of 5340 SLOC
Found 47 operational defects
Density = 47/5340 = 8.8 defects/KSLOC
Project changed 18,463 SLOC
Estimated defects = 18.463 * 8.8 = 162
Remaining = 162 - 47 = 115
What should you do? Inspect more? Or Test for them?
Verification
Use Defect Density to estimate expected
yield
Similar to estimate described with reviews
Defect discovery & closure
Use extrapolation to predict when defect
exit criteria will be met
Defect Discovery
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

F
o
u
n
d

Total Defects Opened
Weekends
Defect Discovery Curve Assumptions
Time is a proxy for exposure
Remove weekends to smooth the curve
Number of test cases run would be better,
assuming
Test coverage is good
Not continuing to add new test case
Well-Behaved Curve
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

F
o
u
n
d

Release Target Date
Add: Defect Closure
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

Total Defects Opened & Closed
Opened
Closed
Remaining
Open
Add: To QA
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

Total Defects Opened, Closed, & To QA
Opened
To QA
Closed
Opened, waiting for
development fix
Turned over from
development to QA,
awaiting verification
Finding The Balance
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

Total Defects Opened, Closed, & To QA
Opened
To QA
Closed
Whats
happening
here?
QA is a
bottleneck
Dev is a
bottleneck
Add: Legacy & Target Release
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

Total Defects Opened & Closed
Total
Targeted
To QA
Closed
Inherited from
previous history,
targeted for or found
in the project/release
Extrapolation to Release Date
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

Project Defects
Total
Targeted
To QA
Closed
Release Target Date
Should I be
concerned?
Extrapolation to Release Date
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

Project Defects
Total
Targeted
To QA
Closed
Release Target Date
When will the
project be done?
Computing in Excel
Query defects into Excel
Convenient: Use named ranges for data
Computing in Excel
In a second sheet, create a date column and count
the defects submitted and closed for each date.
Array formula. <Shift-Ctrl-Return>
Real Life Example
Aardvark 4.7 Defect Discovery & Closure
(With extrapolations)
0
50
100
150
200
250
300
350
400
450
500
6
/
1
7
/
0
8
6
/
2
4
/
0
8
7
/
1
/
0
8
7
/
8
/
0
8
7
/
1
5
/
0
8
7
/
2
2
/
0
8
7
/
2
9
/
0
8
8
/
5
/
0
8
8
/
1
2
/
0
8
8
/
1
9
/
0
8
8
/
2
6
/
0
8
T
o
t
a
l

C
R
s
0
10
20
30
40
50
60
70
80
90
100
F
o
u
n
d

i
n

4
.
7
Open
Closed
Found in 4.7
6/16 = Start of testing
Target
Release
Date
2
nd
order polynomial fit
Release Criteria: Defects
Goals:
Provide good-enough quality to customers
Dont release any bad defects
Dont release too many defects
Minimize support costs
Customers will need support proportional to
product quality
Defect Discovery
Time
C
u
m
u
l
a
t
i
v
e

D
e
f
e
c
t
s

F
o
u
n
d

Release Target Date
Should I release?
Bad Defects
Bad is a function of
Severity = the consequences of the failure
Probability that it will occur
The number of customers likely to experience it
May also include important customers or functions
These are often complex to calculate, so most
companies just use Severity, tempered by other
factors
Quality Threshold
No more than the following number of open
defects:

Where New = defects discovered in the current project, and
Old = defects discovered in previous releases
Severity* New Old
Critical 0 0
Important 5 10
Moderate 30 100
Trivial 50 200
Release team (stakeholders) need to
determine values based on history. The goals
are:
Not to release any Critical defects
Not to introduce too many new defects
Manage the backlog
Whats Good Enough?
The two market leaders Quicken
TurboTax, produced by Intuit, and
TaxCut, produced by H&R Block
It also offers better end-
of-return reviews
TurboTax offers better videos than
TaxCut
Whats Good Enough?
TaxCut offers easier tagging of
items youd like to deal with later.
It also appears to be more
scrupulously proofread than
TurboTax: Twice in TurboTaxs
screens in my tests, incorrect
information was displayed, though
the tax returns themselves were
unaffected.
Whats Good Enough?
if youre choosing which program
to go with for the first time this year,
Id go with TaxCut solely because
of the text errors in TurboTax.
I have a higher standard for accuracy in tax
programs than I do for any other type of
software, and while the errors we found didnt
affect the bottom line, they did affect my
confidence in the programs makers.
Its unfortunate, because I felt
TurboTax offered better features.
Detroit Free Press
> 1.7M Readers
+ web site
+ wire service
The article came from the
Monterey Peninsula Herald
Predicting Defects
There are several models for predicting
defects
The predictions of interest are:
How many defects are going to be created?
When will I find them?
How long will it take to fix them?
When will I be done?
How good will the result be?
Using Defect Density
Phase Defect Density

DD
Phase
=

At first this may not seem right, but its a reasonable
predictor
Which is more reliable?
67 specification defects per 200 pages of documentation
(0.34 defects/page)
67 specification defects per 4300 SLOC (15.6 defects/KSLOC)
Defects Injected
Phase
Total Project Affected SLOC
Using Defect Density
Project/Process Defect Density
Number of defects per
affected SLOC
Affected SLOC can be
difficult to get without a
good tool.
The best Ive found is
CodeReports
(aka SourceROI)
http://www.polariguana.com/sourceroi.php
Release A Release B
Defects
452 563
Total SLOC
8,600 43,500
Defect
Density
52.6 12.9
Project
Changed
SLOC
5,400 4,350
Project
Defect
Density
83.7 129.4
Calculating DD per Phase
# Defects Injected Per Phase
Project KSLOC
Number of Defects Injection Phase
Discovery Phase Req's Design Code Test Production
Requirements 46 0 0 0 0
Design 8 26 0 0 0
Code 83 19 83 0 0
Testing 33 22 37 2 0
Production 3 2 16 0 1
Total Injected 173 69 136 2 1
Defect Density 4.94 1.97 3.89 0.06 0.03
Project
SLOC =
35,000

Note: if this is
only test defect
data, youre
missing an
opportunity to get
better information.
DD
Phase
=
Next Project
Estimated to be 20,000 SLOC
Historical
Defect
Density
Likely
to
Inject
Phase
Containment
Likely
to Leak
Effort to
Resolve*
Total
Rework
Requirements 4.94 99 26.6% 26 85.3 2242.2
Design 1.97 39 17.3% 7 38.5 263.3
Code 3.89 78 62.1% 48 18.3 882.9
Testing 0.06 1 81.7% 1 2.2 2.1
Production 0.03 1 100.0% 1 1 0.6
Total 218 Grand Total 3391.0
* Worst case cost (defects discovered in test)
Defect Discovery & Closure
Predictions can be
made in Excel by
extrapolating and
seeing where the
discovery and
close lines meet
Cumulative Defects Submitted
0
50000
100000
150000
200000
250000
300000
1
0
/
1
4
/
2
0
0
6
1
0
/
2
8
/
2
0
0
6
1
1
/
1
1
/
2
0
0
6
1
1
/
2
5
/
2
0
0
6
1
2
/
9
/
2
0
0
6
1
2
/
2
3
/
2
0
0
6
1
/
6
/
2
0
0
7
1
/
2
0
/
2
0
0
7
2
/
3
/
2
0
0
7
2
/
1
7
/
2
0
0
7
3
/
3
/
2
0
0
7
3
/
1
7
/
2
0
0
7
3
/
3
1
/
2
0
0
7
4
/
1
4
/
2
0
0
7
4
/
2
8
/
2
0
0
7
5
/
1
2
/
2
0
0
7
5
/
2
6
/
2
0
0
7
6
/
9
/
2
0
0
7
6
/
2
3
/
2
0
0
7
7
/
7
/
2
0
0
7
7
/
2
1
/
2
0
0
7
8
/
4
/
2
0
0
7
Extrapolation
This method is
quick and in
some cases
surprisingly
good
But not in all
cases
Other Excel Extrapolations
Adding a trend line uses all the data
You may only want to use some of the
data
Create a parallel column of the data you
want to extrapolate and extrapolate only
that curve
Example
Simple Linear Extrapolation
To create a linear extrapolation on the last
few data points, start by extending the
date
column:
Select the cells that
look linear
Copy and Paste
Special - Values
Then drag the cells down to the last
extrapolated date
Excel will
do a linear
extrapolation
of the data
Extrapolation
In order for extrapolations to be reliable,
the discovery rate must have peaked
Weekly Defects Submitted
0
10
20
30
40
50
60
1
/
1
/
2
0
0
8
1
/
1
5
/
2
0
0
8
1
/
2
9
/
2
0
0
8
2
/
1
2
/
2
0
0
8
2
/
2
6
/
2
0
0
8
3
/
1
1
/
2
0
0
8
3
/
2
5
/
2
0
0
8
4
/
8
/
2
0
0
8
4
/
2
2
/
2
0
0
8
5
/
6
/
2
0
0
8
5
/
2
0
/
2
0
0
8
6
/
3
/
2
0
0
8
6
/
1
7
/
2
0
0
8
7
/
1
/
2
0
0
8
7
/
1
5
/
2
0
0
8
7
/
2
9
/
2
0
0
8
8
/
1
2
/
2
0
0
8
8
/
2
6
/
2
0
0
8
9
/
9
/
2
0
0
8
9
/
2
3
/
2
0
0
8
1
0
/
7
/
2
0
0
8
Done?
What does it mean to be done (from a
defect perspective)?
No bad defects remain
Predictable/manageable defects remain
Two categories of defects remain
Those you know
Those you dont know
Latent Defects
Latent defects (unknown at release time)
can be predicted using defect density and
Rayleigh PDF
What needs to happen before the
predictions become reliable?
The defect discovery rate must flatten
d f(t)
dt
= 0
Discovery Rate
Cumulative Defect Submitted
0
100
200
300
400
500
600
1
/
1
/
2
0
0
8
1
/
1
5
/
2
0
0
8
1
/
2
9
/
2
0
0
8
2
/
1
2
/
2
0
0
8
2
/
2
6
/
2
0
0
8
3
/
1
1
/
2
0
0
8
3
/
2
5
/
2
0
0
8
4
/
8
/
2
0
0
8
4
/
2
2
/
2
0
0
8
5
/
6
/
2
0
0
8
5
/
2
0
/
2
0
0
8
6
/
3
/
2
0
0
8
6
/
1
7
/
2
0
0
8
7
/
1
/
2
0
0
8
7
/
1
5
/
2
0
0
8
7
/
2
9
/
2
0
0
8
8
/
1
2
/
2
0
0
8
8
/
2
6
/
2
0
0
8
9
/
9
/
2
0
0
8
9
/
2
3
/
2
0
0
8
1
0
/
7
/
2
0
0
8
Weekly Defects Submitted
0
10
20
30
40
50
60
1
/
1
/
2
0
0
8
1
/
1
5
/
2
0
0
8
1
/
2
9
/
2
0
0
8
2
/
1
2
/
2
0
0
8
2
/
2
6
/
2
0
0
8
3
/
1
1
/
2
0
0
8
3
/
2
5
/
2
0
0
8
4
/
8
/
2
0
0
8
4
/
2
2
/
2
0
0
8
5
/
6
/
2
0
0
8
5
/
2
0
/
2
0
0
8
6
/
3
/
2
0
0
8
6
/
1
7
/
2
0
0
8
7
/
1
/
2
0
0
8
7
/
1
5
/
2
0
0
8
7
/
2
9
/
2
0
0
8
8
/
1
2
/
2
0
0
8
8
/
2
6
/
2
0
0
8
9
/
9
/
2
0
0
8
9
/
2
3
/
2
0
0
8
1
0
/
7
/
2
0
0
8
Fitting Data to a Model
Fitting data
to a curve
helps us tell
the rate is
decreasing
Using Excel
As before, create a linear extrapolation of
the last <criteria> days:
Flattening criteria example:
The discover rate must less than 5 defects
per day in the past two weeks of testing
Using Excel
Add linear trend line
Select Display equation on chart
Using Excel
This gives the slope of the line m,
where

y=mx + b

In this case,
m = 2.2 defects/day
Not good enough
Cumulative Defect Submitted
y = 2.163x - 85395
0
100
200
300
400
500
600
1
/
1
/
2
0
0
8
1
/
1
5
/
2
0
0
8
1
/
2
9
/
2
0
0
8
2
/
1
2
/
2
0
0
8
2
/
2
6
/
2
0
0
8
3
/
1
1
/
2
0
0
8
3
/
2
5
/
2
0
0
8
4
/
8
/
2
0
0
8
4
/
2
2
/
2
0
0
8
5
/
6
/
2
0
0
8
5
/
2
0
/
2
0
0
8
6
/
3
/
2
0
0
8
6
/
1
7
/
2
0
0
8
7
/
1
/
2
0
0
8
7
/
1
5
/
2
0
0
8
7
/
2
9
/
2
0
0
8
8
/
1
2
/
2
0
0
8
8
/
2
6
/
2
0
0
8
9
/
9
/
2
0
0
8
9
/
2
3
/
2
0
0
8
1
0
/
7
/
2
0
0
8
Latent or Known Defect Risk
Probability of occurrence in production
Risk = Severity * Probability
Need to have a decent measure/estimate
of probability
Need a usage profile
Also, frequency does not always equal
value
I may not use the Restore function very
often, but it better work
And Finally
Track
latent/released
defects
carefully
They are your
best indicator of
Quality
Release Defect Density
From:
Software Quality
Measurement:
A Framework for Counting
Problems and Defects
CMU/SEI-92-TR-022
http://www.sei.cmu.edu/pub/d
ocuments/92.reports/pdf/tr22.
92.pdf

Agenda
A True Story
A while back, I looked in the mirror and
decided I needed to loose some weight.
What is the first thing Im going to do?
My First Step
My First Step On A Scale
If I dont measure myself today, I wont
know
If the changes Im planning to make are taking
me the right direction
Which changes have the best affect
How far Ive come
When Im done
Its important to establish a baseline

What Are
The Goals?
`Would you tell me,
please, which way I
ought to go from here?'
`That depends a good deal on where you want to get
to,' said the Cat.
`I don't much care where--' said Alice.
`Then it doesn't matter which way you go,' said the Cat.
Closure Data Collection
Submit reason
Test case failed
Pass criteria unclear
Tested wrong version
Setup/config wrong
Data-related error
Other
No
Correction made
Duplicate
Change not justified
Injection Phase
Root Cause
Defect?
Resolution
Yes
Classification, Mode
Corrected Component(s)
This is where we record what weve learned
Total
Resolution
Effort
Defect Classification Models
Industry guidelines to standardize
information collection
The most prevalent are from:
IEEE
IBM
Hewlett Packard
HP Defect Origins, Types, and Modes
Defect Leakage Reasons
1. Environmental Dependency
2. Test Not Run
3. Test Case Error
4. Data Dependency
5. Inconsistent Symptoms
6. Test Case Success Criteria Unclear
7. Test Execution Error
8. Test Case Results Not Verified
9. Code Changed After Test Passed
10. No Test Case/Coverage
11. Configuration Management Problem
12. Requirements Missing/Changed
Using Leakage Data
For each defect found in production, log a leakage
reason
Use the frequency (volume) to improve the related
process
Resist temptation to weight the frequency by
something important
This assumes cause & effect, and there is none
This is different from root cause, where the cost to repair
is related to the type of defect
Specific Process Improvement Metrics
Two types of process metrics:
1. Metrics that monitor a process
(tell you the effect of changes)
Basic SPC-type charts
2. Metrics that tell you what to do
Phase Containment
Root Cause Analysis
Test Coverage
Defect Leakage Reasons
Static Code Analysis - Complexity
Should all projects be measured?
Pros:
More data points
Evaluate everything were doing
Cons:
Projects may use different processes
Variation may be too great
Suggestions:
Dont measure projects measure processes
Only measure your best projects
Tailoring is okay, but separate out the data
Hold Periodic Process Reviews
Monthly Quality Reviews focused on
Process Improvement initiatives
Measures & Analysis/Interpretation
Lessons Learned
Rules of metrics etiquette:
Preview data with stakeholders = no surprises
Phase Containment
Identifies the lifecycle activities that leak
defects
Tells you where
to improve defect
detection efforts
Phase Containment
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Requirements Design Code Build Testing
%

D
e
f
e
c
t
s

C
o
n
t
a
i
n
e
d
Root Cause Analysis
Pareto the types of
errors being created,
weighted by cost (effort)
Root Cause Total Hours
0
500
1000
1500
2000
2500
3000
R
e
q
u
i
r
e
m
e
n
t
s

A
m
b
i
g
u
o
u
s
/
v
a
g
u
e

R
e
q
u
i
r
e
m
e
n
t
s

M
i
s
s
i
n
g

L
o
g
i
c

f
l
o
w

M
i
s
s
i
n
g

l
o
g
i
c

R
e
q
u
i
r
e
m
e
n
t
s

C
h
a
n
g
e
d

W
r
o
n
g

v
a
r
i
a
b
l
e

u
s
e
d

O
t
h
e
r

C
a
s
e

s
t
a
t
e
m
e
n
t

T
y
p
o

S
p
e
c
i
f
i
c
a
t
i
o
n

M
i
s
s
i
n
g
G
a
r
b
a
g
e

c
o
l
l
e
c
t
i
o
n

E
x
c
e
p
t
i
o
n

h
a
n
d
l
i
n
g

P
a
r
a
m
e
t
e
r

m
i
s
m
a
t
c
h

R
e
q
u
i
r
e
m
e
n
t
s

W
r
o
n
g

U
n
i
n
i
t
i
a
l
i
z
e
d

v
a
r
i
a
b
l
e

B
a
d

v
a
l
i
d
a
t
i
o
n

c
h
e
c
k

M
i
s
s
i
n
g

e
n
d

s
t
a
t
e
m
e
n
t

E
n
v

n
o
t

m
a
t
c
h

t
e
s
t
S
p
e
c
i
f
i
c
a
t
i
o
n
-
r
e
q
u
i
r
e
m
e
n
t

m
i
s
m
a
t
c
h
C
o
d
i
n
g

s
t
a
n
d
a
r
d

n
o
t

f
o
l
l
o
w
e
d

S
p
e
c
i
f
i
c
a
t
i
o
n

A
m
b
i
g
u
o
u
s
E
n
v

c
a
p
a
c
i
t
y
S
p
e
c
i
f
i
c
a
t
i
o
n

W
r
o
n
g
W
r
o
n
g

m
e
t
h
o
d

c
a
l
l
e
d

D
e
s
i
g
n

l
o
g
i
c

w
r
o
n
g
D
e
s
i
g
n
-
s
p
e
c

m
i
s
m
a
t
c
h
R
e
q
u
i
r
e
m
e
n
t
s

I
n
c
o
n
s
i
s
t
e
n
t

E
n
v

h
a
r
d
w
a
r
e

f
a
i
l
u
r
e
D
o
c

t
y
p
o
D
o
c

v
a
g
u
e
E
n
v

O
p
e
r
a
t
i
o
n
a
l

E
r
r
o
r
I
n
s
t
a
l
l

m
i
s
s
i
n
g

f
i
l
e
s
R
e
q
u
i
r
e
m
e
n
t
s

O
t
h
e
r

D
o
c

i
n
f
o

m
i
s
s
i
n
g
I
n
s
t
a
l
l

w
r
o
n
g

c
o
n
f
i
g
u
r
a
t
i
o
n
I
n
s
t
a
l
l

i
n
c
o
m
p
l
e
t
e
D
o
c

w
r
o
n
g
I
n
s
t
a
l
l

w
r
o
n
g

v
e
r
s
i
o
n
Root Cause x Rework Hours
Agenda
Defect Probabilities
Weibull distributions are useful engineering
models, where the tail approaches zero
asymptotically
Weibull Probability Density Function (PDF) is:

Where m determines the shape of the curve, and
c is the scale parameter
t
c
m
t
m
e
t
c
m
-
f(t) =
t
c
m
c
m-1
e
t
c
m
-
=
(Wikipedia)
Different Weibull Distributions
Weibull Curves
0
1
2
3
4
0 1 2 3
m=0.5
m=1
m=2 m=4
m=10
Weibull curves with
different shapes
(values of m) can
be used to model
various systems
(probability
distributions)
Weibull Probability Distribution
Weibull Curves
m = 0.1 to 5.0 in .1 increments
0.0
0.5
1.0
1.5
2.0
0.0 0.5 1.0 1.5 2.0
Rayleigh (m=2)
Weibull Curves
Weibull Curves
m = 0.1 to 5.0 in .1 increments
0.0
0.5
1.0
1.5
2.0
0.0 0.5 1.0 1.5 2.0
Rayleigh (m=2)
(almost) Normal (m=3.4)
Rayleigh Probability Distribution Function
Weibull distribution with shape (m) = 2

Let x = 2t
t
c
2
t
2
e
t
c
2
-
f(t) =
2t

c
2
e
t
c
2
-
=
x

c
2
e
x
2c
2
-
f(x) =
x

c
2
e
x
2
4c
2
-
=
Rayleigh Probability Distribution Function
x

c
2
e
x
2
4c
2
-
f(x) =
Rayleigh Curves
c = 0.1 to 0.5
0
1
2
3
4
5
6
7
8
9
0.0 0.5 1.0 1.5
c = 0.1
c = 0.2
c = 0.3
Excel Formula
Applying Rayleigh PDF
Let t
m
= time at which the curve reaches its peak

To find the peak, we set = 0

Solving gives t
m
=

To scale the area under the curve, multiply by K
Substitute c = t
m
2
d f(t)
dt
c
2
Fitting The Rayleigh PDF
2t

c
2
e
t
c
2
-
f(t) = K
2t

(t
m
2)
2
e
t
(t
m
2)
2
-
= K
2t

2t
2
m

e
- t
2
2t
2
m

= K
1

t
m

te
- t
2
2t
2
m

f(t) = K
2
If we can estimate how many defects well find (K) and when
weve hit the peak (t
m
), we can create a PDF for our project!
Building Your Estimate
Predicting Latent Defects
At peak, the area under the Rayleigh curve is 39.35%
This means 60.65% of the defects are yet to find
Example:
At peak you found a cumulative total of 58 defects.
Total = (58/39.5)*100=147.4
147 - 58 = 89 remaining.
Latent Defects = K dt
1

t
m

te
- t
2
2t
2
m

2
Release Date
1 year?
Easier that Calculus
In Excel, run a column
for cumulative total and a
column for remaining
defects
From your release date
you can easily estimate
latent defects
Cumulative Rayleigh Model
t
c
2
-
CDF = 1 - e
Rayleigh Assumptions
The defect rate observed during the
development process is positively
correlated with the defect rate in the field.
Given the same error injection rate, if more
defects are discovered and removed
earlier, fewer will remain in later stages.
Exponential Models
Exponential Model (Weibull, m=1)
1
c
e
t
c
-
f(t) =
Exponential
Weibull, m = 1
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0.0 0.5 1.0 1.5 2.0
Error detection rate
Instantaneous failure rate
Hazard rate

In general, this is a
reliability growth model
Fitting A Model to Data
EasyFit from www.mathwave.com
Model Summary
Rayleigh is Weibull distribution with m=2
Other values of m produce other
probability distributions
There are many models; try several that fit
All models are wrong. Some models are
useful.
George E. Box
Summary
A defect management program
includes
A balanced solution
Training & explaining
Metrics that are used
Chris Holl
Chris_Holl@hotmail.com
Next Class: Tools for Metrics

Defect Metrics

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Defect Metrics

Uploaded by

Copyright:

Available Formats

Defect Management &

You might also like