How to Really

Prevent and Manage Bugs

in Agile Projects

Michael Dubakov
TargetProcess, Founder
Table of content

Zero Defects Mentality 4

Bug Source #1: Unclear User Stories 5
Clarification meeting
Stories should be ready before iteration start 6
Bug Source #2: Manual Testing in Short Iterations 7
Bug Source #3: Automated Testing in Short Iterations
Bug Source #4: Separate Testing Team 8
Bug Source #5: Bug Clusters
Agile Bug Management 9
Bugs found in iteration
Bugs found in production 10
Bug estimation 12
Agile Bug Management Anti-Patterns 13
Bug fixing phase inside iteration
Bug fixing iteration
Skip “Zero open bugs” rule in user story’s Definition of Done
Defect Tracking Systems 14
Co-located
Distributed
References 15

2
Foreword
Today, Agile is a widely adopted software development approach. Most developers
have either heard of or successfully applied popular agile practices like TDD, CI, PP, etc.

But what about testers? It seems that many teams are having difficulties trying to
integrate quality control teams into an agile development process. Moreover, it also
seems that many teams are having difficulties with overall bug management. How to
treat bugs? Does a bug have a business value? Should we estimate bugs in points or in
hours? Should we treat bugs as user stories and add to velocity?

This article not only answers the questions above and explains how to manage bugs in
agile projects, but it also shows how not to manage bugs.

3
Zero Defects Mentality
Are you familiar with a zero defects mentality? It looks very good from the first sight: “I’d like
to have zero defects in my projects.”

So, what is a zero defects mentality?

Here is the quote from the book Lean Software Development: an Agile Toolkit:

“One of the fastest ways to kill motivation is what is called in the US Army a zero defects
mentality. A zero defects mentality is an atmosphere that tolerates absolutely no
mistakes; perfection is required down to the smallest detail. The army considers a zero
defects mentality to be a serious leadership problem, because it kills the initiative
necessary for success on a battlefield”
Mary & Tom Poppendieck

Obviously, a zero defects mentality is not something that HOUSE M.D. likes ;) Moreover, I think
Dr. House hates it, because it has several unpleasant effects:

Not enough courage to refactor a complex, messy, buggy, but important piece of code.
Inability to make an important right decision. The result is a less risky, but wrong
decision.
Doing everything to avoid responsibility leads to coward and stupid behavior.

Zero defects may sound good. But in reality you still have errors after production. Even in
predictable and (quite) easily testable industries (hardware, automobile, etc.) there are
problems with 100,000 power adapters that should be replaced (or hard drive problems, or
accumulators problems, I bet you can continue the list).

How can we expect zero defects in software development? It is harder to test, harder to define
in details, harder to predict. Software development is a complex adaptive system, and we
can't predict all effects.

So bugs in production are a normal thing, and by “normal” I mean that we can't bring them to
zero. We can (and should) minimize them using all possible ways. Bugs are a form of waste.
We should do our best to reduce bugs in software projects and eradicate waste.

There are several strategies that will help:

Test Driven Development. A nice side effect of TDD is a unit tests suite. You have tests
for all new code, and you have unit tests for all old code. If you have good tests - you
have less bugs.

4
 Continuous integration. Instant feedback helps to identify problems early and fix them
early. It saves money and you have less bugs in production.
Automated regression functional testing suite. Unit tests are good, but you need
something else. Functional tests emulate user behavior and identify user interface
errors, integration errors, etc. Needles to say you should have continuous integration in
place to succeed with automated functional tests.
Root cause analysis. There are several ways to fix the bug. You may just hack the code
and apply a patch (please, don’t do it at home!). You may think deeper and fix the bug
nicely. Also you may look into the root of the problem and resolve it. Yes, it may take
time, but you will prevent future bugs from this domain.
High development skills. Ironically, high skills do not always reduce a bug rate in
production. “Stars” may be strong in some areas (business layer), but may not polish
things. It may lead to many small bugs, especially on UI.

“Stop worrying about the defect database and start worrying about why you are still
creating code where defects are discovered a significant amount of time after the code
has been written.”
Mary Poppendieck

Bug Source #1: Unclear User Stories

I think unclear specifications are a #1 source of bugs. If you deliver something that was not
intended, there will be many complaints from customers and in their opinion almost all of
these complaints are bugs. Also, without instant communication it is very easy to miss some
details and release an unpolished user story with many small glitches.

Agile development is very rapid. In many cases the path from an idea to user story in progress
is extremely short; it may be even 1 day. In this environment it is very easy to make mistakes
in a user story description. Product Owner may miss some important details and as a result
the story does something wrong, which was not expected.

“This is the first question you must ask... is it a bug, or is it a change? I've seen a lot of
bugs that have come up that were "We asked you for x, never thinking about y, so could
you please change the system so that y is covered?" It's a business scenario, so there's
little reason to expect a dev or tester to anticipate/test it. I know where we've struggled is
in injecting items into the product backlog, we tend to classify it as a bug and so we end
up with a lot of bugs, but not a lot of change in the back log. That's exactly what I think
we're supposed to be avoiding in Scrum or any other agile methodology. As we see
change, we should be taking it on as a feature and prioritizing it accordingly. Only when
it's something broken should it be called a bug.”
Jim Leonardo

The best solution to this problem is communication.

5
Clarification meeting
The important thing in software development is to put everyone on the same page.
Developer, Tester and Product Owner should conduct a small meeting about a user story, thus
forming a mini-team. If the user story has several developers and several testers, all should
participate.

The goals of the meeting are:

To ensure that everyone understands what should be implemented.

To brainstorm acceptance tests/test cases and create a check list.
To reveal and eliminate all discrepancies in a story description/specification.

Product
Owner

Discuss User
Stories,
brainstorm
acceptance tests

Developer Tester

We have these meetings at TargetProcess and they are very efficient and helpful. The side
effect of the meeting is that Product Owner receives less questions about functionality later,
thus having less interruptions (which is good).

Stories should be ready before iteration start

It is a good idea to have all user stories described in minute detail before iteration start. Thus,
testers and developers will have time to review them and prepare questions for the iteration
meeting. It may drive a much better discussion and reveal some interesting problems that
were unclear initially.

6
There is a danger to document more user stories than is required, and it will be a form of
waste. Product Owner should maintain good balance here.

Bug Source #2: Manual Testing in Short Iterations

Most testers are having hard time switching from waterfall to iterative development. Short
iterations put pressure on a testing activity. It is required to write test cases fast, execute
them fast and provide feedback fast. It is far away from a huge testing phase in waterfall.

Under pressure testers make more mistakes, think less and miss some test cases in a hurry.
Testers tend to write brief test cases and skip unusual or not obvious cases. Naturally, it is a
straight way to bugs in production.

The most common error is a testing phase in every iteration. It is a kind of mini-waterfall and
should be avoided. Each feature should have a separate flow with its own testing phase. It is
required to test by feature. Tester should start testing immediately after a feature is developed
(or even earlier if possible).

Bug Source #3: Automated Testing in Short Iterations

Test Driven Development is a great bug catcher. If you apply it right, you will improve product
quality significantly and bugs in production will be quite rare. However, it is not as easy as it
sounds.

There are two levels of TDD: Unit Tests and Functional Tests.

It is relatively easy to adopt unit tests and make it a mandatory practice in the development
team. Developers write unit tests, then write code, then refactor.
Functional tests are harder to create and maintain. And it is even harder to create them
before feature implementation. And it is especially hard to create automated functional tests
upfront on the User Interface level.

Tools like Fitnesse simplify functional testing of business logic, but still you want to have
complete UI tests using Selenium, WinRunner or any other GUI testing tool.

Here are several pieces of advice that will help you catch more bugs during iterations using
automated tests:

Adopt TDD fully on the unit tests level.

Hire highly experienced automated testers (almost like developers in skills) who can
write automated tests very fast. Speed is important here. An alternative is to force
developers to write automated functional tests.
Create automated functional tests for the previous iteration at the beginning of the
next iteration. It will be regression testing, but it is definitely better than nothing.

7
Bug Source #4: Separate Testing Team
It is common for a waterfall process to have separate functional teams: Designers, Developers,
Testers, DBA, etc. Functional teams set additional communication barriers, and poor
communication leads to many problems like:

Missing test cases.

Different understanding of a user story by Developer and Tester.
Delay between story implementation and testing phases.

Obviously, these problems lead to more bugs found in production. The best thing to do here is
to create a cross-functional team and put testers and developers into one shared space. It
should be one team, not several sub-teams.

Quite a common situation with functional departments looks like this. You have several
testers assigned to your project. Then suddenly two of them get relocated to another project
and replaced by two other testers. That is a real danger to your project.

So, your goal is a single cross-functional team. If it is not possible, you’ll have to play political
games. At least you should try to extract a dedicated testing team for the project and
minimize the power of Functional Testing Department Manager. In this case you have a
chance to join two functional teams into one.

Bug Source #5: Bug Clusters

Bugs like to live in communes in a single nest. Most likely you have modules with perfect
conditions for bug survival and reproduction. Bugs prefer the following places:

Code written by junior developers.

Code written under time pressure.
Legacy (initially beautiful) code that was passed to the offshore team, then back to the
main development team, then back offshore and then shared between several teams.
Code that somehow does not have unit tests (what a shame!).
UI (yes, most user interfaces are buggy, since developers do not unit test UI).
Quite a large and complex piece of code (there is a high chance that complex logic will
have some missed test cases as it is hard not to forget something).
Code written in JavaScript.
Integration with a third-party system.

So if you have a complex UI component written by Junior Developer in JavaScript with no unit
tests and under pressure - I bet you have several bugs in this code.

8
Anyway, bugs are nesting and so a good strategy is to find the nest and destroy it (refactor,
rewrite, add unit tests, etc). Lisa Crispin has something to say about it.

“If you already have a defect database, consider mining it for information about where
defects cluster, and focus on cleaning up the related code.” - Lisa Crispin

http://home.att.net/~lisa.crispin/CrispinFinal.pdf

Software

Bug
Bug Authentication
module

Bug
Bug Content
Bug module
Bug Bug Bug

Bug Bug Reporting

Bug module
Bug
Bug
Bug

If you have a defect tracking system, you may use tags, thus categorizing defects. It will be
quite easy to find problematic areas in your software. For example, you may find out that most
bugs live in the Reports or Email Integration areas.

Agile Bug Management

Agile development books often skip a bug management practice. Many agile people assume
that the problem is obvious, unfortunately it is not the case.

In general, there are two types of bugs: bugs found during iteration development and bugs
found in production. Strategies are quite different as you may expect.

Bugs found in iteration

There is a huge difference between a bug found during story development and a bug found by
the customer in production. A bug found during story development is much easier to fix, since
you have focus on the story, you know what you did 1-3 days ago and everything is fresh in
9
mind. You may even recall some places in the code where the bug can live right away. If a bug
is found a month from now, it will take time to switch to it, fix, include into the new build and
deliver.

So, how should we handle such bugs? First, do not estimate bugs during story development at
all. Second, do not prioritize them. Bugs are a usual part of the development process and on
average in each iteration you will have quite similar time spent on bug fixing. So why bother
with estimation and prioritization? One simple rule replaces them: all bugs found in the
iteration should be fixed. No exceptions. No excuses.

If you do not estimate bugs, they do not affect iteration velocity. If you use the same estimate
technique during several iterations, most likely you will have similar bug rates. It means that
the bug fixing deviation does not affect velocity.

Iteration #1

User Story User Story User Story User Story Time

Bug Bug Bug Bug Bug

Bug Bug
Bug

User Story User Story

Bug Bug Bug

Bug Bug
Bug Bug Bug Bug 0 open bugs

Alternatively you may estimate bugs as tasks (in hours), but I don’t see much value it this
effort.

Bugs found in production

Unfortunately, we have to deal with bugs in production. No software is bug free. There are
bugs in business software, medical software and even space shuttle software.
Sure medical software has less bugs, but they still exist (http://www.cs.tau.ac.il/~nachumd/
verify/horror.html).

Bugs as User Stories

One strategy is to put bugs into the product backlog and treat them as user stories. From the
customer’s perspective there is no much difference between bug in production and a new
feature request. The customer wants to have a problem solved, and the problem may be
expressed as a bug or may be expressed as a user story.

10
 Iteration #1

User Story Production Bug User Story User Story Production Bug Time

Bug Bug Bug

Bug Bug Bug
Bug
Development Team

Is this bug with login screen more important than the “Advanced Search” user story? It is a
subject for Product Owner to decide. He should maintain a single backlog with bugs and user
stories prioritized and select bugs and stories for the next iteration.

In such a scenario you will have to estimate bugs in points or in hours to have a fair
prediction how many bugs and stories you may take and implement in the next iteration. It
sounds somewhat counterintuitive, since bugs are a form of waste. But still prioritization and
estimation are important in this scenario.

However there is a danger in this approach:

If defects are viewed as features with negative value, they become managed as if they
were features. Development groups store up prioritized repositories of bugs, treat bugs
as user stories, outsource the fixing of bugs, and so on. While each of those can be a useful
technique or perspective in dealing with a project in transition or crisis, it is not a long-
term view that should be encouraged. After all, as the "Manifesto for Agile Software
Development" says, "Working software is the primary measure of progress." It is a little
disingenuous to pass off a feature with known defects as complete and working — "Yes,
it's done... but there are a few bugs." - Kevlin Henney

People dedicated to bug fixing

Another strategy is to have dedicated people for bug fixing. For example, you may dedicate
about 10-20% of a development team’s effort to production bug fixes. Alternatively, you may
have one developer each day on production bug fixing, and developers may rotate daily.

11
 Iteration #1

User Story User Story User Story User Story Time

Bug Bug Bug Bug Bug Bug

Bug Bug

Development Team

In this case, 25% of the development team is solely dedicated

to fixing bugs found in production (1 of 4 developers)

Production Bug Production Bug Production Bug Support Developer

Free time. Support Developer drinks bear or

joins the development team (and enjoys user
stories). But if a new bug is found in
production, she switches to it with no delays.

Using this approach, you have two separate flows: one for user stories and another for bugs
found in production. Obviously, each flow should be managed separately. While for user
stories you have usual metrics like an iteration burn down chart, for a bug flow you need to
invent something different.

If you do not estimate bugs, all you have is the bugs fixed/day metric. If you do estimate them,
you may create a bug burn up chart or points fixed/day chart.

If you have few bugs, good advice will be to not track the bug fixing flow at all. But if you
have many bugs that will take several weeks to fix, you need some metrics to forecast when
you will be able to fix all Critical bugs, all Normal bugs, all Small bugs, etc. Also you should
know how many new bugs are coming from production each week. It will definitely influence
forecasts.

Weeks to a bug free release = Total bugs / (Weekly bug fixing rate - Weekly new bugs rate)
For example, you have 40 bugs to fix, weekly bug fixing rate is 10 bugs/week, and each week
2new bugs are found. In this case
Weeks to a bug free release = 40 / (10 - 2) = 5 weeks.

Bug Estimation
You may not estimate bugs. There are several reasons not to estimate:

Often bugs are hard to estimate. Even a small problem may take a day to fix
(unexpectedly revealing dirty roots that should be cleared up).
Bugs should be fixed with no exceptions.
If you have few bugs from production and use dedicated people for bug fixing,
estimation does not bring any additional value.

Bug estimation is required when:

You have many bugs in production and want to provide some forecasting.
You treat bugs as user stories and plan iteration accordingly.
12
Agile Bug Management Anti-Patterns
There are several dangerous misunderstandings (anti-patterns) related to bug management
that are quite common. It is better to recognize the problem early and fight to win. Here are
three common bug management anti-patterns in a development process.

Bug fixing phase inside iteration

Symptom: You have several days at the end of an iteration fully dedicated to bug fixing.

Iteration #1

Implement User Stories Bug Fixing Time

User Story User Story Bug Bug Bug

User Story User Story Bug Bug Bug

If you have a bug fixing phase inside an iteration, you have mini-waterfall in each iteration,
which is not a very good idea (in agile we blame waterfall, don’t we?). A story should be
tested as soon as it is completed.

Bug fixing iteration

Symptom: You run several iterations with almost no or little bug fixing and then have one (or
even two) iterations that are fully dedicated to bug fixing.

Release #1

Time
Iteration #1 Iteration #2 Iteration #3
Implementing stories Implementing stories Bug fixing

User Story User Story User Story User Story Bug Bug Bug

User Story User Story Bug Bug Bug

A bug fixing iteration kills your iterative development. It is even worse than mini-waterfall.
Such iterations increase your work in progress dramatically. You have nothing done during
several iterations before a bug fixing iteration. You have nothing to ship. Moreover, bug fixing
iterations reduce motivation, people do not like to fix bugs during 2 weeks or a full month.

Skip “zero open bugs” rule in user story’s Definition of Done

Symptom: You have open bugs in a completed iteration.

You may have decided that some bugs discovered during story development are unimportant
and may be postponed to future releases/iteration. That is a very dangerous practice that
leads to bug accumulation. Moreover, it may be a reason for a bug fixing iteration (see above).
The best strategy is to have a “zero open bugs” rule in DoD for a user story.

13
Defect Tracking Systems
Should we use a defect tracking system? The answer is “it depends”. Here are all possible
alternatives:

Use a defect tracking system (excel, web based, desktop).

Use Task Board. Write bugs on sticky notes and stick them to Task Board.
Use index cards. Write bugs on a card and give it to Developer.
Personal communication (use voice and memory). Just tell Developer about the
problem.

Let’s see under what conditions you may use these approaches.

Co-located Distributed

Task Board, index cards,

Small team Defect tracking system
personal communication

Task Board, defect tracking

Medium Team Defect tracking system
system

Large Team Defect tracking system Defect tracking system

Obviously, a defect tracking system is a must for a distributed team. A distributed team
demands more formal communication and a single place to store all bugs.

A small co-located team may use any approach that fits their process. Medium and large co-
located teams can’t rely on personal communication and simple story cards. Task Board may
work for a medium team, but a large team most likely needs a defect tracking system.

I was surprised that our programmers find the defect-tracking system quite useful. The
ability to read clear, detailed steps to reproduce the problem saves them time. - Lisa
Crispin

14
References

Coping with Bugs on an Agile/Scrum Project

http://www.infoq.com/news/2009/07/coping-with-bugs

Software testing is an activity, not a phase or a department

http://searchsoftwarequality.techtarget.com/news/article/
0,289142,sid92_gci1337882,00.html

Fix defects right away

http://www.cyphersec.com/2009/05/21/fix-bugs-now/

15