The Cost of Defects

Joseph Cheng
APSC 210 606

Ashley Whitehead
Atimi Software

Submitted on
May 8, 2009
 Letter of Transmittal

To whom it may concern:

This report was prepared for Ashley Whitehead, a Quality Assurance

lead at Atimi Software. It details the concept of defect fixing, a crucial
element in any software development cycle.

Ashley Whitehead has reviewed this report, and all the contained
information is confirmed to be non-confidential and within the public
domain.

This report focuses on the economic perspective of defects, detailing

how certain factors can improve development efficiency and reduce
total project costs. This work was done from January to May 2009.

Sincerely,

Joseph Cheng
The Cost of Defects

Joseph Cheng
14157077

January – May 2009

Submitted to
UBC Science Co-op
Engineering Physics

I certify that this report is non-confidential and may be submitted to UBC

 Abstract

This report compares different factors that could influence the cost and

efficiency of fixing defects, errors that prevent software from behaving

properly. Defect fixing consumes most of a developer’s time; a Capers

Jones study in 1998 found that 74 percent of a developer’s time is

spent fixing defects rather than developing. Customers care most

about getting what they want, in this case a functional product.

However, it is sometimes too time consuming and inefficient to fix

certain defects. By weighing the consequences and risks of defects,

companies will be able to increase profit and better satisfy customers.

iii
 Summary

Defect fixing is a very time consuming task for developers. Dealing

with this problem effectively will significantly reduce the costs spent on

a project and greater success in the market. The key to this problem is

time, because the longer detects go undetected in a project, the

harder they will be to fix. To do this, quality assurance staff is needed

to weed out design errors before coding takes place. During the

development phase, the software should be continually tested starting

from the basics. Also during the development phase, teams should

ideally be openly discussing issues, and resolve them in an orderly

fashion. Lastly, both developers and customers need to realize that

almost all software is defective. At some point, defect fixing must stop

because changing the code any further risks introducing worse

problems.

iv
 Table of Contents

Abstract…………………………………………………………………………………

………………. ii

Summary…………………………………………………………………………………

……………... iii

Table of Contents………………………………………………………………………

……………. iv

Introduction……………………………………………………………………………

……………… 1

Quality Assurance……………………………………………………………………

……………... 2

Quality Control…………………………………………………………………………

…………….. 3

Company Culture………………………………………………………………………

……………. 4

Variable……………………………………………………………………………

……………. 5

Routine……………………………………………………………………………

…………….. 5

Steering……………………………………………………………………………

……………. 6

Risky Changes…………………………………………………………………………

v
……………… 7

Conclusion………………………………………………………………………………

……………… 8

Works Cited………………………………………………………………………………

……….…… 9

Bibliography……………………………………………………………………………

……………… 10

vi
 Introduction

All software companies face the unique challenge of fixing defects.

Defects fixing requires a lot of time and effort from developers,

meaning more costs; therefore dealing with them efficiently is vital.

The solution to this problem is not always clear, and usually comes

through experience because of the number of factors involved.

Ultimately the results of this challenge will determine whether the

project will profit the company or not.

This report aims to establish how a company’s approach to defect

fixing will affect the costs of a project. In particular, the effects of

having competitive testing staff, good management, and experienced

developers and how they affect the project’s costs will be explored.
 Quality Assurance

Software quality assurance involves monitoring and improving the

software development process to ensure certain procedures and

standards are followed. It is a process that attempts to prevent defects.

(Hower) Defect prevention begins with the requirement analysis –

understanding what the customer wants and transforming it into

product specifications without introducing errors. This is important

because errors in the software requirements and design documents are

more frequent than in the source code itself. (Soni)

An integral part of quality assurance is to analyze defects, find the root

cause, determine a quick solution and take preventive action. These

preventive measures, after approval from team members are

embedded into the company as a baseline for upcoming projects. This

methodology provides companies a long-term solution and experience

from learning from their mistakes. Preventing defects that originate

from the design phase could save as much as 64 percent of the total

defect cost. (Soni)

viii
 Quality Control

Software quality control involves operating a system or application in a

controlled environment, and assessing the results. Testing conditions

and procedures, commonly called a “test case” should cover both

normal and abnormal behaviour. (Hower) Ideally, quality control staff

should start testing as early as possible to maximize efficiency,

because detecting defects early on dramatically affect the costs of

fixing them. (Soni)

Compared to finding a defect in the early stages, defects found late in

the development process can cost almost six times more. Defects

found during programming cost around a quarter as much as defects

found after coding is finished. Overall, fixing problems that crop up late

in the development process contributes to 85 percent of the defect

fixing effort. (van Megan and Meyerhoff)

ix
 Company Culture

The approach companies take towards defect fixing depend largely on

the management. The attitude the manager takes towards quality will

significantly affect how the project will be dealt with. For example,

compare the mind-set of the two following engineering managers:

"We'll fix that when we have time. In the meantime, just keep

developing! How can you possibly tell how much it will cost to fix

later?"

— An Engineering manager, eager to continue software development

"Let's find all the defects before system test. The customer will wait for

the product." — An Engineering manager, concerned that shipping a

product with defects would prevent customers from buying the product

x
Management often has to make the choice of when to fix defects, and

their beliefs will determine the development process within a company.

Weinberg identifies three types of company cultures, with each culture

prioritizing quality and defect fixing differently. (Rothman)

Variable Culture

The engineering manager eager to continue software development

represents a company with a variable culture. Management does not

recognize that quality is a management issue, and problems are fixed

through personal effort. The company becomes inefficient at fixing

defects, often finding more than they could fix. After shipping their

product, many issues arise as customers become disappointed about

the product. The company must go through the testing cycle again,

which incurs additional costs from developing release notes, extensive

testing on different hardware and software platforms, testing areas

around the fix, and then merging the fix back into the build. (Rothman)

Routine Culture
xi
Within a routine culture, management agrees quality is valuable, but

no time or money is allocated to ensure quality is there. However,

management does understand the consequences of releasing a

product that is not up to standards. Teams usually fix problems only

after a severe defect has been found. Typically, the company culture

also helps find all the easy defects before testing occurs. This type of

management is most effective with small-scale projects, fixing almost

all defects during the development phase. Conversely, fixes are rarely

made once the product has been shipped, but costs will significantly

increase if they need to. (Rothman)

Steering Culture

The second engineering manager fits the criteria of a steering culture.

Management in a steering culture understands the importance of

quality, and that quality is a systemic issue. In this environment,

emphasis is on preventing defects rather than testing and problems

are discussed and resolved in an orderly manner. Steering culture

management deals with defects the best, finding and dealing with

them faster than other types of company. Only minor problems are

xii
ever seen after release, and the costs to fix them are minimal.

(Rothman)

Risky Changes

At some point, developers have to stop fixing defects because it risks

introducing worse ones. Four factors need to be considered about

defects: Severity, Frequency, Cost, and Risk. Defects with high severity

and frequency are given the most priority and should be fixed. Defects
xiii
with low severity and frequency should not be fixed. Cost and risk are

tradeoffs involved with fixing the defect; if the costs and/or risk are

high, the relevant defect’s priority should go down. It is critical that the

expectations of the customer and market are understood, because

defects are sometimes acceptable. Experienced developers should be

able to make these decisions quickly, so that time and effort is not

wasted on risky problems. (Sink)

Conclusion

xiv
In 2002, a Research Triangle Institute study estimated that software

defects cost the U.S. $59.5 billion annually. The report further stated

that improving testing infrastructure would enable defects to be dealt

with more effectively and potentially eliminate $22.2 billion of the

costs. Costs can be lowered by 64 percent from having QA staff, and a

further 85 percent by having QC staff. Openly discussing problems is

also critical, especially in larger projects as it makes it much easier to

release fixes after release. Additionally, discussing problems will allow

managers and even customers to understand why some problems

occur so they can decide whether it’s worth the time and effort to fix it.

Being able to avoid or reduce these costs while still satisfying

customers is key to success in the market.

xv
 Works Cited

Bank of Montreal, Toronto, Ontario | ispw.com. 11 Jan. 1998.

ComputerWire, Inc. 11

Apr. 2009 <http://www.ispw.com/the-company/testimonials/34-

poor-quality-software.html>.

Hower, Rick. Software QA and Testing Resource Center. 17 Apr. 2009.

Digital Media

Group, Inc. 8 Apr. 2009

<http://www.softwareqatest.com/index.html>.

Rakitin, Steve. Food for Thought: All Software is Defective. Dec. 2005.

Software

Quality Consulting, Inc. 16 Apr. 2009

<http://www.swqual.com/newsletter/vol2/no11/vol2no11.html>.

Rothman, Johanna. What Does It Cost You To Fix A Defect? And Why

Should You

Care? Oct. 2000. Rothman Consulting Group, Inc. 12 Apr. 2009

<http://www.jrothman.com/Papers/Costtofixdefect.html>.

Sink, Eric. Inside IT: Why we all sell code with bugs. 25 May 2006. The

Guardian

xvi
 Newspaper. 15 4 2009

<http://www.guardian.co.uk/technology/2006/may/25/insideit.gu

ardianweeklytechnologysection >.

Soni, Mukesh. Defect Prevention: Reducing Costs and Enhancing

Quality. 19 July

2006. iSixSigma.com. 14 Apr. 2009

<http://software.isixsigma.com/library/content/c060719b.asp >.

Tan, Gang. A Collection of Well-Known Software Failures. 22 Dec. 2008.

Lehigh

University. 15 Apr. 2009

< http://www.cse.lehigh.edu/~gtan/bug/softwarebug.html>.

van Megan, Rudolf and Dirk Meyerhoff. "Costs and benefits of early

defect detection:

experiences from developing client server and host applications."

Software Quality Journal 4 Apr. 1995: 247-56.

xvii
 Bibliography

Software Quality Assurance. N.d. 14 April 2009

<http://www.software-quality-assurance.org/index.htm>.

“QA – Quality Assurance and Software Testing” Ask Numbers. N.d. 13

April 2009

<http://www.asknumbers.com/QualityAssuranceandTesting.aspx

>.

“Popular testing phrases: Testing Should Start Early.” Compendium

Developments. 1

April 2002. 14 April 2009

<http://www.compendiumdev.co.uk/essays/popular_testing_phra

ses_57.php>.

“Cost of fixing software defects ‘runs into millions’” SC Magazine. 18

July 2008. 16

April 2009

<http://www.scmagazineuk.com/Cost-of-fixing-software-defects-

runs-into-millions/article/112597/>.

“Testing vs. Quality Assurance” CM Crossroads. 30 May 2006. 11 April

2009

<http://www.cmcrossroads.com/index2.php?option=com_content

&do_pdf=1&id=6782she>.
xviii
“The Economics of Testing” Rice Consulting. N.d. 14 April 2009

<http://www.riceconsulting.com/public_pdf/STBC-WM.pdf>.

xix