Defect Metrics

Defect Management & Metrics
Society for Software Quality April 24, 2010
Who Is This Guy?

30 years in software and hardware/software Quality 19 years in Quality & Process Management 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit Past SSQ President
L
S
SSQ Presentation
021209.01
2010 Holl Consulting
Primary Relevant Experience

Designed Intuits corporate defect management process Assisted in rollout to ~1800 engineers Guided its evolution for six years As Corporate Software Metrics Leader, fostered quantitative project management and process improvement Made mistakes and learned from them
SSQ Presentation
021209.01
Agenda
Summary of Previous Talk Defect Management Metrics Metrics for Project Management Metrics for Process Improvement Advanced Metric Models
SSQ Presentation
021209.01
Last time
We reviewed cultural issues related to defect management We went over a defect management process:
Policies Workflow/States Fields
SSQ Presentation
021209.01
Data Collection
At different states in the workflow, different data are collected Most data are collected at submit time Additional data are entered by the developer upon resolution And finally by the tester upon verification
SSQ Presentation
021209.01
Agenda
SSQ Presentation
021209.01
When Does A Defect Count?

It depends on your goals
Defect Management: To track defects for resolution in order to release products
Just log functional problems found in test; deviations from the requirements that the customer will see; defects that you dont want to fall through the cracks.
Defect Analysis: To improve the process or predict

If your goal is to get better (learn) or predict the future, you need to capture much more information. Without early lifecycle data (requirements, etc.) predictive models arent that reliable
World-class companies dont just manage defects they analyze information.
SSQ Presentation
021209.01
Data Collection Costs

Estimate Assumptions:
18,000 defects the defects are corrected (9,000) Data collection time: 10 minutes (validated at HP) Average defect cost: 16 hours (data from Michael Fagan) Data collection reduces defects 5% Sampling is valid, but sampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected.
Time to record (18k *10m) Time to correct (9k *16h) Time savings (450 * 16h) ROI
3,000 hrs 144,000 hrs 7,200 hrs 240%
Defect reduction (9k * 5%) 450 defects
This doesnt quantify the value of being able to predict quality and schedules
SSQ Presentation
021209.01
Simple Measures
Defect cost = Fix rate = Duration = (Total Resolution Effort) Number of resolved defects Number of resolved defects Period of time (week, month, etc.) ((Closed Date) (Submit Date))* Number of resolved defects
Last two include queue time, so they depend on workload Its helpful to understand the standard deviation too Bonus: By type, root cause, injection phase
* Excel NetWorkDays is a useful function
SSQ Presentation
021209.01
Relative Defect Costs

If you record injection and discovery phases and effort, you can compute relative costs Example:
Cost of a requirements defect = 1 Cost of a design defect = 10 Cost of a coding defect = 100 Cost of a production defect = 1000
SSQ Presentation
021209.01
Similar to Phase Containment

Start with matrix of injection and discovery Enter the average rework (or duration) costs in each cell (resolved defects only)
Injection Phase Discovery Phase Requirements Design Code Testing Production Req's 2.4 15.3 22.7 43.8 85.3 4.1 9.5 18.5 38.5 1.2 4.3 18.3 1.2 2.2 1 Design Code Test Production
SSQ Presentation
021209.01
Similar to Phase Containment

Normalize by dividing each cell by the Injection=Discovery value
Injection Phase Discovery Phase Requirements Design Code Testing Production Req's 1.00 6.38 9.46 18.25 35.54 1.00 2.32 4.51 9.39 1.00 3.58 15.25 1.00 1.83 1.00 Design Code Test Production
SSQ Presentation
021209.01
Logging Defects
This is not
How many defects leak?
This is
What percent of known defects are recorded?
If you only start logging defects in test, youre missing over half of your data Try to capture data from reviews/inspections and developer unit tests
SSQ Presentation
021209.01
Prioritizing Defects
This is not about how to set priority, but Priority means the most important defects get resources first To measure this, we need to know the queue time (from submit to work starts) Therefore, we need an Accepted state, or something to note when work began Why cant we use Submit to Verify duration?
SSQ Presentation
021209.01
Aging Defects
Time in Resolver queue Time in Verifier queue
12 Days in Queue
Alternately, use durations minus effort as a proxy
Average Queue Time

10 8 6 4 2 0 0-Test Stopper 1 2 Priority 3 4 5
SSQ Presentation
021209.01
Defect Backlog
Many groups dont count defects left over from prior releases; they only focus on defects found and not fixed in the current project This is dangerous because a product defect backlog can grow unmanaged ALL of the open defects are released, not just the new ones When producing the report of open defects, query for all open defects in the product
SSQ Presentation
021209.01
The Calculation
Week 1 2 3 4
Total number of defects Number closed before the week started Number that were opened after the week ended = Number open in the week
SSQ Presentation
021209.01
Calculating Backlog in Excel

Query Submit Date and Closed Date
SSQ Presentation
021209.01
Calculating Backlog in Excel

Array formula. <Shift-Ctrl-Return>
This uses named ranges, where D.Num.CRs D.T.Closed = Total number of CRs opened = Range of Closed Dates
D.Submit.Date = Range of Submit Dates
SSQ Presentation
021209.01
Product Defect Backlog
The Result
SSQ Presentation
021209.01
Agenda
SSQ Presentation
021209.01
Some Project Management Defect Metrics
Reviews & Inspections Defect Discovery & Closure Defect Density
SSQ Presentation
021209.01
A Project Axiom The quality of product in one phase is an indicator of the quality in later phases Why?
There are two answers
SSQ Presentation
021209.01
Review Metrics
Reviews & inspections have proven to be the most costeffective way to detect and remove defects Inspect your key project artifacts
(requirements, designs, code, tests, etc.)
SSQ Presentation
021209.01
Review/Inspection Data
There are many data collected during a review or inspection The ones that tell us about quality are:
Number of Operational defects discovered Number of Minor defects found
per changed line of inspected doc/code Number of reviews per doc/module

SSQ Presentation
021209.01
Project Defect Density

Consistency in categorizing inspection defects is important
Have clear guidelines for Operational and Minor defects
Defect Density Project = Operational Defects found by inspection Delta SLOC (or LOT)
SSQ Presentation
021209.01
Projecting Defect Leakage

Inspected total of 5340 SLOC Found 47 operational defects Density = 47/5340 = 8.8 defects/KSLOC Project changed 18,463 SLOC Estimated defects = 18.463 * 8.8 = 162 Remaining = 162 - 47 = 115 What should you do? Inspect more? Or Test for them?
SSQ Presentation
021209.01
Verification
Use Defect Density to estimate expected yield
Similar to estimate described with reviews
Defect discovery & closure Use extrapolation to predict when defect exit criteria will be met
SSQ Presentation
021209.01
Defect Discovery
Total Defects Opened
Cumulative Defects Found
Weekends
Time
SSQ Presentation
021209.01
Defect Discovery Curve Assumptions Time is a proxy for exposure

Remove weekends to smooth the curve
Number of test cases run would be better, assuming

Test coverage is good Not continuing to add new test case
SSQ Presentation
021209.01
Well-Behaved Curve
Time
Release Target Date
SSQ Presentation
021209.01
Add: Defect Closure

Total Defects Opened & Closed
Remaining Open
Cumulative Defects
Opened Closed
Time
SSQ Presentation
021209.01
Add: To QA
Total Defects Opened, Closed, & To QA
Cumulative Defects
Opened, waiting for development fix
Turned over from development to QA, awaiting verification
Opened To QA Closed
Time
SSQ Presentation
021209.01
Finding The Balance

Total Defects Opened, Closed, & To QA
Cumulative Defects
Whats happening here?
Dev is a bottleneck
Opened To QA Closed
QA is a bottleneck
Time
SSQ Presentation
021209.01
Add: Legacy & Target Release

Total Defects Opened & Closed
Cumulative Defects
Inherited from previous history, targeted for or found in the project/release
Total Targeted To QA Closed
Time
SSQ Presentation
021209.01
Extrapolation to Release Date

Project Defects
Cumulative Defects
Should I be concerned?
Time
Release Target Date
SSQ Presentation
021209.01
Extrapolation to Release Date

Project Defects
Cumulative Defects
When will the project be done?
Time
Release Target Date
SSQ Presentation
021209.01
Computing in Excel
Query defects into Excel Convenient: Use named ranges for data
SSQ Presentation
021209.01
Computing in Excel
In a second sheet, create a date column and count the defects submitted and closed for each date.
Array formula. <Shift-Ctrl-Return>
SSQ Presentation
021209.01
Real Life Example

500 450 400 350 300 6/16 = Start of testing 250 50 100
Aardvark 4.7 Defect Discovery & Closure

(With extrapolations)
90 80 70 Found in 4.7 60
Total CRs
2nd order polynomial fit

200 150 100
Open
40 30 20
Closed Found in 4.7
50 0
7/1/08 7/8/08 6/17/08 6/24/08 7/15/08 7/22/08 7/29/08 8/5/08 8/12/08 8/19/08
10 0
8/26/08 Target Release Date
SSQ Presentation
021209.01
Release Criteria: Defects

Goals:
Provide good-enough quality to customers
Dont release any bad defects Dont release too many defects
Minimize support costs

Customers will need support proportional to product quality
SSQ Presentation
021209.01
Defect Discovery
Should I release?
Time
Release Target Date
SSQ Presentation
021209.01
Bad Defects
Bad is a function of
Severity = the consequences of the failure Probability that it will occur The number of customers likely to experience it May also include important customers or functions
These are often complex to calculate, so most companies just use Severity, tempered by other factors
SSQ Presentation
021209.01
Quality Threshold
No more than the following number of open defects:
Severity* Critical Important Moderate Trivial New 0 5 30 50 Old 0 10 100 200 Release team (stakeholders) need to determine values based on history. The goals are: Not to release any Critical defects Not to introduce too many new defects Manage the backlog
Where New = defects discovered in the current project, and Old = defects discovered in previous releases
SSQ Presentation
021209.01
Whats Good Enough?

The two market leaders Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block
TurboTax offers better videos than TaxCut
It also offers better endof-return reviews
SSQ Presentation
021209.01
Whats Good Enough?

TaxCut offers easier tagging of items youd like to deal with later.
It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTaxs screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected.
SSQ Presentation
021209.01
Whats Good Enough?

if youre choosing which program to go with for the first time this year, Id go with TaxCut solely because of the text errors in TurboTax.
I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didnt affect the bottom line, they did affect my confidence in the programs makers. Its unfortunate, because I felt TurboTax offered better features.
SSQ Presentation
021209.01
Detroit Free Press

> 1.7M Readers + web site + wire service
The article came from the Monterey Peninsula Herald
SSQ Presentation
021209.01
Predicting Defects
There are several models for predicting defects The predictions of interest are:
How many defects are going to be created? When will I find them? How long will it take to fix them? When will I be done? How good will the result be?
SSQ Presentation
021209.01
Using Defect Density

Phase Defect Density DDPhase = Defects InjectedPhase Total Project Affected SLOC
At first this may not seem right, but its a reasonable predictor Which is more reliable?
67 specification defects per 200 pages of documentation (0.34 defects/page) 67 specification defects per 4300 SLOC (15.6 defects/KSLOC)
SSQ Presentation
021209.01
Using Defect Density

Project/Process Defect Density
Number of defects per affected SLOC Affected SLOC can be difficult to get without a good tool. The best Ive found is CodeReports (aka SourceROI)
Release A Defects
Total SLOC Defect Density Project Changed SLOC Project Defect Density
Release B
452 8,600 52.6 5,400 83.7
563 43,500 12.9 4,350 129.4
http://www.polariguana.com/sourceroi.php
SSQ Presentation
021209.01
Calculating DD per Phase

# Defects Injected Per Phase DDPhase = Project KSLOC
Number of Defects Injection Phase Req's Design Code Test Production 46 8 83 33 3 173 4.94 0 26 19 22 2 69 1.97 0 0 83 37 16 136 3.89 0 0 0 2 0 2 0.06 0 0 0 0 1 1 0.03
Project SLOC = 35,000

Note: if this is only test defect data, youre missing an opportunity to get better information.
Discovery Phase Requirements Design Code Testing Production Total Injected Defect Density
021209.01
SSQ Presentation
Next Project
Estimated to be 20,000 SLOC
Historical Defect Density Requirements Design Code Testing Production 4.94 1.97 3.89 0.06 0.03 Total Likely to Inject 99 39 78 1 1 218 Phase Likely Effort to Containment to Leak Resolve* 26.6% 17.3% 62.1% 81.7% 100.0% 26 7 48 1 1 85.3 38.5 18.3 2.2 1 Grand Total Total Rework 2242.2 263.3 882.9 2.1 0.6 3391.0
* Worst case cost (defects discovered in test)
SSQ Presentation
021209.01
SSQ Presentation
Predictions can be made in Excel by extrapolating and seeing where the discovery and close lines meet
Defect Discovery & Closure
021209.01
100000
150000
200000
250000
300000
50000

10 /1 4 10 /2 0 /2 06 8 11 /2 0 /1 06 1 11 /2 0 /2 06 5/ 2 12 00 /9 6 12 / 20 /2 06 3/ 2 1/ 006 6/ 2 1/ 00 20 7 /2 0 2/ 07 3/ 20 2/ 17 0 7 /2 0 3/ 07 3/ 20 3/ 17 0 7 / 3/ 20 0 31 7 / 4/ 20 0 14 7 /2 4/ 0 0 28 7 / 5/ 20 0 12 7 /2 5/ 0 0 26 7 /2 0 6/ 07 9/ 20 6/ 23 0 7 /2 0 7/ 07 7/ 20 7/ 21 0 7 /2 0 8/ 07 4/ 20 07
Cumulative Defects Submitted
Extrapolation
This method is quick and in some cases surprisingly good But not in all cases
SSQ Presentation
021209.01
Other Excel Extrapolations

Adding a trend line uses all the data You may only want to use some of the data Create a parallel column of the data you want to extrapolate and extrapolate only that curve
SSQ Presentation
021209.01
Example
SSQ Presentation
021209.01
Simple Linear Extrapolation

To create a linear extrapolation on the last few data points, start by extending the date column:
SSQ Presentation
021209.01

Select the cells that look linear Copy and Paste Special - Values
SSQ Presentation
021209.01

Then drag the cells down to the last extrapolated date Excel will do a linear extrapolation of the data
SSQ Presentation
021209.01
SSQ Presentation
10 20 30 40 50 60 0
1/1/2008 1/15/2008 1/29/2008 2/12/2008 2/26/2008 3/11/2008 3/25/2008 4/8/2008 4/22/2008 5/6/2008 5/20/2008 6/3/2008 6/17/2008 7/1/2008 7/15/2008 7/29/2008 8/12/2008 8/26/2008 9/9/2008 9/23/2008 10/7/2008
021209.01
In order for extrapolations to be reliable, the discovery rate must have peaked
Extrapolation
Weekly Defects Submitted
Done?
What does it mean to be done (from a defect perspective)?
No bad defects remain Predictable/manageable defects remain
Two categories of defects remain

Those you know Those you dont know
SSQ Presentation
021209.01
Latent Defects
Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF What needs to happen before the predictions become reliable?
The defect discovery rate must flatten
d f(t) =0 dt
SSQ Presentation
021209.01
100
200
300
400
500
600
SSQ Presentation
021209.01
1/1/2008 1/15/2008 1/29/2008
2/12/2008 2/26/2008 3/11/2008 3/25/2008 4/8/2008 4/22/2008 5/6/2008 5/20/2008 6/3/2008 6/17/2008 7/1/2008 7/15/2008 7/29/2008 8/12/2008 10 20 30 40 50 60 0 1/1/2008 1/15/2008 1/29/2008 2/12/2008 2/26/2008 3/11/2008 3/25/2008 4/8/2008 4/22/2008 5/6/2008 5/20/2008 6/3/2008 6/17/2008 7/1/2008 7/15/2008 7/29/2008 8/12/2008 8/26/2008 9/9/2008 9/23/2008 10/7/2008 8/26/2008 9/9/2008 9/23/2008 10/7/2008
Discovery Rate
Cumulative Defect Submitted
Weekly Defects Submitted
Fitting Data to a Model

Fitting data to a curve helps us tell the rate is decreasing
SSQ Presentation
021209.01
Using Excel
As before, create a linear extrapolation of the last <criteria> days: Flattening criteria example:
The discover rate must less than 5 defects per day in the past two weeks of testing
SSQ Presentation
021209.01
Using Excel
Add linear trend line Select Display equation on chart
SSQ Presentation
021209.01
Using Excel
This gives the slope of the line m, where
Cumulative Defect Submitted
y=mx + b
In this case, m = 2.2 defects/day Not good enough
600 500 y = 2.163x - 85395 400 300 200 100 0
1/1/2008
4/8/2008
5/6/2008
6/3/2008
7/1/2008
1/15/2008
1/29/2008
2/12/2008
2/26/2008
3/11/2008
3/25/2008
4/22/2008
5/20/2008
6/17/2008
7/15/2008
7/29/2008
8/12/2008
8/26/2008
9/9/2008
9/23/2008
SSQ Presentation
021209.01
10/7/2008
Latent or Known Defect Risk

Probability of occurrence in production
Risk = Severity * Probability
Need to have a decent measure/estimate of probability

Need a usage profile
Also, frequency does not always equal value

I may not use the Restore function very often, but it better work
SSQ Presentation
021209.01
And Finally
Track latent/released defects carefully They are your best indicator of Quality
SSQ Presentation
021209.01
Release Defect Density

From: Software Quality Measurement: A Framework for Counting Problems and Defects CMU/SEI-92-TR-022 http://www.sei.cmu.edu/pub/d ocuments/92.reports/pdf/tr22. 92.pdf
SSQ Presentation
021209.01
Agenda
SSQ Presentation
021209.01
A True Story
A while back, I looked in the mirror and decided I needed to loose some weight. What is the first thing Im going to do?
SSQ Presentation
021209.01
My First Step
SSQ Presentation
021209.01
My First Step On A Scale

If I dont measure myself today, I wont know
If the changes Im planning to make are taking me the right direction Which changes have the best affect How far Ive come When Im done
Its important to establish a baseline

SSQ Presentation
021209.01
What Are The Goals?

`Would you tell me, please, which way I ought to go from here?' `That depends a good deal on where you want to get to,' said the Cat. `I don't much care where--' said Alice. `Then it doesn't matter which way you go,' said the Cat.
SSQ Presentation
021209.01
Closure Data Collection

This is where we record what weve learned
Total Resolution Effort Correction made Duplicate Change not justified Defect?
Yes No
Submit reason Test case failed Pass criteria unclear Tested wrong version Setup/config wrong Data-related error Other
Resolution Injection Phase Classification, Mode Corrected Component(s) Root Cause
SSQ Presentation
021209.01
Defect Classification Models

Industry guidelines to standardize information collection The most prevalent are from:
IEEE IBM Hewlett Packard
SSQ Presentation
021209.01
HP Defect Origins, Types, and Modes
SSQ Presentation
021209.01
Defect Leakage Reasons

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Environmental Dependency Test Not Run Test Case Error Data Dependency Inconsistent Symptoms Test Case Success Criteria Unclear Test Execution Error Test Case Results Not Verified Code Changed After Test Passed No Test Case/Coverage Configuration Management Problem Requirements Missing/Changed
SSQ Presentation
021209.01
Using Leakage Data

For each defect found in production, log a leakage reason Use the frequency (volume) to improve the related process Resist temptation to weight the frequency by something important
This assumes cause & effect, and there is none This is different from root cause, where the cost to repair is related to the type of defect
SSQ Presentation
021209.01
Specific Process Improvement Metrics

Two types of process metrics:
1. Metrics that monitor a process (tell you the effect of changes)
Basic SPC-type charts
2. Metrics that tell you what to do

Phase Containment Root Cause Analysis Test Coverage Defect Leakage Reasons Static Code Analysis - Complexity
SSQ Presentation
021209.01
Should all projects be measured?

Pros:
More data points Evaluate everything were doing
Cons:
Projects may use different processes Variation may be too great
Suggestions:
Dont measure projects measure processes Only measure your best projects Tailoring is okay, but separate out the data
SSQ Presentation
021209.01
Hold Periodic Process Reviews

Monthly Quality Reviews focused on
Process Improvement initiatives Measures & Analysis/Interpretation Lessons Learned
Rules of metrics etiquette:

Preview data with stakeholders = no surprises
SSQ Presentation
021209.01
Phase Containment
Identifies the lifecycle activities that leak defects Tells you where to improve defect detection efforts
Phase Containment
100% 90% 80% % Defects Contained 70% 60% 50% 40% 30% 20% 10% 0% Requirements Design Code Build Testing
SSQ Presentation
021209.01
SSQ Presentation
021209.01
Pareto the types of errors being created, 3000 weighted by cost (effort) 2500
Root Cause Analysis
1000
1500
2000
500
0
Requirements Ambiguous/vague Requirements Missing Logic flow Missing logic Requirements Changed Wrong variable used Other Case statement Typo Specification Missing Garbage collection Exception handling Parameter m ismatch Requirements Wrong Uninitialized variable Bad validation check Missing end statement Env not match test
Root Cause x Rework Hours
Root Cause T
Agenda
SSQ Presentation
021209.01
Defect Probabilities
Weibull distributions are useful engineering models, where the tail approaches zero asymptotically Weibull Probability Density Function (PDF) is:
m m
m t f(t) = t c
m - t c
m t = c c
m-1 - t c
(Wikipedia)
Where m determines the shape of the curve, and c is the scale parameter
SSQ Presentation
021209.01
Different Weibull Distributions

4
Weibull Curves
Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions)
3 m=0.5 m=10
m=1 1
m=2
m=4
0 0 1 2 3
SSQ Presentation
021209.01
Weibull Probability Distribution

2.0
Weibull Curves
m = 0.1 to 5.0 in .1 increments
1.5
Rayleigh (m=2)
1.0
0.5
0.0 0.0 0.5 1.0 1.5 2.0
SSQ Presentation
021209.01
Weibull Curves
2.0
Weibull Curves
m = 0.1 to 5.0 in .1 increments
1.5
(almost) Normal (m=3.4) Rayleigh (m=2)

1.0
0.5
0.0 0.0 0.5 1.0 1.5 2.0
SSQ Presentation
021209.01
Rayleigh Probability Distribution Function

Weibull distribution with shape (m) = 2
2 2
2 t f(t) = t c
Let x = 2t
2 - t
2t = 2 c
2
t c
f(x) = c2 e
SSQ Presentation
021209.01
- x 2c
= c2
x2 - 2 4c
Rayleigh Probability Distribution Function

9
f(x) =
2 e c
8 7
Rayleigh Curves
c = 0.1 to 0.5
c = 0.1
x2 4c 2
6 5 4 c = 0.2 3 c = 0.3 2 1 0 0.0 0.5 1.0 1.5
SSQ Presentation
021209.01
Excel Formula
SSQ Presentation
021209.01
Applying Rayleigh PDF

Let tm = time at which the curve reaches its peak To find the peak, we set
d f(t) dt
=0
c Solving gives tm = 2
To scale the area under the curve, multiply by K Substitute c = tm 2
SSQ Presentation
021209.01
Fitting The Rayleigh PDF

2 2
2t f(t) = K 2 c
t c
=K
2t
(tm 2)
t
(tm 2)
e 2
2
= K 2t2 e m
2t
- t2
2t2m
f(t) = K t m
- t2
te
2t2m
If we can estimate how many defects well find (K) and when weve hit the peak (tm), we can create a PDF for our project!
SSQ Presentation
021209.01
Building Your Estimate
SSQ Presentation
021209.01
Predicting Latent Defects

At peak, the area under the Rayleigh curve is 39.35% This means 60.65% of the defects are yet to find Example:
At peak you found a cumulative total of 58 defects. Total = (58/39.5)*100=147.4 147 - 58 = 89 remaining.
1 year?
Latent Defects = K
1
tm
- t2
te
2t2m
dt
Release Date
SSQ Presentation
021209.01
Easier that Calculus

In Excel, run a column for cumulative total and a column for remaining defects From your release date you can easily estimate latent defects
SSQ Presentation
021209.01
Cumulative Rayleigh Model
CDF = 1 - e
t c
SSQ Presentation
021209.01
Rayleigh Assumptions
The defect rate observed during the development process is positively correlated with the defect rate in the field. Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages.
SSQ Presentation
021209.01
Exponential Models
Exponential Model (Weibull, m=1) 1 f(t) = c
t c
1.2
Exponential
1.0
Weibull,
m=1
Error detection rate Instantaneous failure rate Hazard rate
0.8
0.6
In general, this is a reliability growth model
0.4
0.2
0.0 0.0 0.5 1.0 1.5 2.0
SSQ Presentation
021209.01
Fitting A Model to Data

EasyFit from www.mathwave.com
SSQ Presentation
021209.01
Model Summary
Rayleigh is Weibull distribution with m=2 Other values of m produce other probability distributions There are many models; try several that fit All models are wrong. Some models are useful.
George E. Box
SSQ Presentation
021209.01
Summary
A defect management program includes
A balanced solution Training & explaining Metrics that are used
SSQ Presentation
021209.01
Questions?
Chris Holl
Chris_Holl@hotmail.com
Next Class: Tools for Metrics

Defect Metrics

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Defect Metrics

Uploaded by

Copyright:

Available Formats

Defect Management & Metrics

Society for Software Quality April 24, 2010

Who Is This Guy?

2010 Holl Consulting

Primary Relevant Experience

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

When Does A Defect Count?

Defect Analysis: To improve the process or predict

Data Collection Costs

3,000 hrs 144,000 hrs 7,200 hrs 240%

Defect reduction (9k * 5%) 450 defects

2010 Holl Consulting

2010 Holl Consulting

Relative Defect Costs

2010 Holl Consulting

Similar to Phase Containment

2010 Holl Consulting

Similar to Phase Containment

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

Alternately, use durations minus effort as a proxy

Average Queue Time

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

Calculating Backlog in Excel

2010 Holl Consulting

Calculating Backlog in Excel

D.Submit.Date = Range of Submit Dates

2010 Holl Consulting

Product Defect Backlog

2010 Holl Consulting

2010 Holl Consulting

Some Project Management Defect Metrics

Reviews & Inspections Defect Discovery & Closure Defect Density

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

per changed line of inspected doc/code Number of reviews per doc/module

2010 Holl Consulting

Project Defect Density

2010 Holl Consulting

Projecting Defect Leakage

2010 Holl Consulting

2010 Holl Consulting

2010 Holl Consulting

Defect Discovery Curve Assumptions Time is a proxy for exposure

Number of test cases run would be better, assuming

2010 Holl Consulting

Release Target Date

2010 Holl Consulting

Add: Defect Closure

2010 Holl Consulting

Opened, waiting for development fix

Turned over from development to QA, awaiting verification

2010 Holl Consulting

Finding The Balance

Whats happening here?