Estimation is a crucial element
of software project planning.
Unfortunately, there are inherent
limitations in the ability to
estimate projects accurately due
to the inherent uncertainties in
software projects.
Linda M. Laird
The Limitations
of Estimation
I
n 1968, Alfred Pietrasanta of IBM System
Research Institute wrote, Anyone who
expects a quick and easy solution to the mul-
tifaceted problem of resource estimation is
going to be disappointed. Thirty years later,
Lionel Briand and his colleagues observed that
Despite the large number of cost factors col-
lected and the rigorous data collection, a lot of
uncertainty in the estimates can be observed
(Briand et al., An Assessment and Comparison
of Common Software Cost Estimation Methods,
Proc. Intl Conf. Software Eng., [ICSE], IEEE
1999, pp. 313322). Its now 2006, and we still have
problems with estimation.
In practical terms, your ability to estimate well
comes down to how much you know about a proj-
ect when youre estimating it, and how much
uncertainty is inherent.
In software, we primarily want to estimate three
aspects of a project: effort, schedule, and cost. Most
methods start by estimating sizeas thousands of
lines of code (KLOC), function pointsor some
other proxy point and using that to estimate effort
(that is, staff months). From effort, you typically
derive staffing, schedule, and cost. Estimating
effort is the primary challenge; once you have an
effort estimate, wonderful tools are available to
help you work through your schedule, staff, costs,
and risk, and tie them into your project plan.
EFFORT ESTIMATION: WHERE ARE WE?
How well do you estimate? How do you know?
Do you track your estimates through the pro-
40 IT Pro November December 2006 Published by the IEEE Computer Society
jects life cycle and understand where you
under- or overestimated so you can do better the
next time?
The 1995 Standish Groups Chaos Report cites
overruns on 89 percent of projects. Kjetil
Molokken and Magne Jrgensen, who studied
software estimation results, describe other sur-
veys as most commonly reporting that 30 to 40
percent of programs overrun (K. Molokken and
M. Jrgensen,A Review of Surveys on Software
Effort Estimation, Proc. 2003 Intl Symp.
Empirical Software Eng., IEEE CS Press, 2003,
p. 223). They conclude that the overruns root
causes are complex, the data isnt always reliable,
and those responding to the surveys may have
a tendency to over report causes that lie outside
of their responsibility... for example, customer-
related causes. Rarely does just one factor on a
project go awry. Rather, the interplay between
factors, such as schedules, changing require-
ments, new technologies, resource unavailabil-
ity, unforeseen problems, and personnel
(management and staff) leads to overruns. And
its often easiest to attribute the overrun to fac-
tors over which you have little controlthat is,
the customer.
The literature lists several common causes for
overruns:
specifying incomplete or unclear requirements
(not knowing what to do),
failing to adjust schedules when scope changes
(too much work),
1520-9202/06/$20.00 2006 IEEE
 setting overly aggressive development schedules (too
little time), and
insufficient resources (not enough people or
equipment).
These reasons are rather generic. In my view, the pri-
mary causes of software project overruns are:
Lack of education and training. Many people dont know
how to estimate, have no training in estimation, and
receive no feedback on their estimates to help them
improve.A developer who knows how to write code well
doesnt necessarily know how to estimate.
Confusion of the desired schedule/effort target with the
estimate. Development teams are frequently pushed into
dates because of business needs rather than a rational
plan to deliver on those dates.
Hope-based planning. Develop- Estimates give you the Many of the methods use an ana-
ers know the right answer: that
the project is on time and on
opportunity to adjust
budget, based on the marketing- project parameters
or management-assigned target.
Software personnels inability to to meet budgets
credibly communicate and sup- and deadlines.
port their estimates. The lack of
good estimation processes and
knowledge frequently leads to being pushed into the
shortest schedule you cant prove you wont make
instead of a rational schedule based on probable out-
comes.
Incomplete, changing, and creeping requirements.
Nothing harms a fixed-price and fixed-schedule project
more than changing and growing requirements.
Quality surprises. Projects can easily spend half of their
time in the test-and-fix phase, especially when the need
for speed causes the development team to take addi-
tional risks and turn over inadequately tested code.
DO WE CARE HOW WELL WE ESTIMATE?
The answer is a resounding yes. First, Im sure your
organization cares. Even if youre doing exploratory devel-
opment or prototyping, youll usually need some estimates.
Second, if youre in the software business, estimating well
is the only way youll be able to have a personal life instead
of working too many nights and weekends.
In most projects, estimates are crucial. If alternative
approaches exist, accurate estimates improve the decisions
you must make in a project. When bidding for jobs, if you
underestimate, you can lose money and damage your com-
pany. If you overestimate, you might unnecessarily lose a
job to a competitor. Estimating well helps you win the jobs
you should win and lose those you should lose.
Estimates give you the opportunity to adjust project
parameters to meet budgets and deadlines. If you under-
stand your cost factors, you can adjust them to hit the tar-
get. At Stevens Institute of Technology, we have seen a
marked improvement in the success rate of the senior proj-
ects in the students who also take the estimation and met-
rics class. Instead of focusing primarily on functionality
and design, they now include difficulty and size, and define
their projects to be doable in the time available.
Even when you cant adjust your cost factors, estimating
well gives you the chance to manage the risk, rather than
ignore it or be surprised later.
SOFTWARE ESTIMATION
METHODOLOGIES AND MODELS
Hundreds of documented software estimation method-
ologies, tools, and models exist. Some of the earliest ones
estimated the software cost as a percentage of the hard-
ware cost. For most, the output is an estimation of staff
effort from a wide variety of inputs.
lytical formula based on cost driv-
ers, which typically are project
characteristics such as system size,
system domain, complexity, and
development methodology. (Com-
plexity in this context is project
complexity, such as the need for
high security or reliability, rather
than design or code complexity.)
The tools and methodologies are primarily based on data
from past projects. Researchers and engineers studied proj-
ect data and determined equations and formulas that best
matched the existing data points. Some of the formulas are
extremely simple, others are complex. None are totally
accurate with all of the past projects, nor should you expect
them to be with their predictions for future projects.
Instead, they predict, based on their formulas (which are
primarily based upon their data set), the most likely effort
required for the project as described. At Stevens, I teach
over 10 methods for estimating projects. Some are simple
and quick, others take considerable time and effort. My
students estimate the same project using the different esti-
mation methodologies and models. Their results vary
widely, depending on their definition of the project, their
assumptions, and the models used.
Which methodology should you use? Most projects use
expert opinion. Excellent results have been achieved with
use case points (Bente Anda,Comparing Effort Estimates
Based on Use Case Points with Expert Estimate, Proc.
Empirical Assessment in Software Eng. 2002, http://www.
simula.no/departments/engineering/publications/SE.5.And
a.2002.a. Function points have strong advocates as well.
Whatever methodologies you select, you should test and
calibrate them to your organization using your own past
projects and determining the methodologies accuracy level.
The current recommendation is that you use more than
one methodologyin fact, the standard recommendation
November December 2006 IT Pro 41
 SOFTWARE ESTIMATION

is to use at least three methods (F. Heemstra, Software Or, as Bollinger notes, The creation of genuinely new
Cost Estimation, Information and Software Technology, software has far more in common with developing a new
vol. 34, no. 10, Oct. 1992, pp. 627639; T. Bollinger, The theory of physics than it does with producing cars or
Interplay of Art and Science in Software, Computer, vol. watches on an assembly line (Bollinger 1997).
30, no. 10, Oct. 1997, pp. 125128). Each method will add Which camp are you in? Problem solving or process exe-
insight and understanding, and using three allows you to cution? Or does it vary depending on the circumstances?
triangulate on an answer. (Three methods might seem like In other words, are you developing a new system which
overkill, especially when many organizations dont even requires creativity and invention, or are you developing a
use one. Obviously, the amount of effort spent on estima- system that you already know how to build?
tion depends on the importance of the estimations accu- Consider the analogy of building six new apartment
racy and the ease of creating the estimate.) buildings, one after the other, all of which have the same
You should choose complementary methods with dif- design and all of which are being built on the same type of
ferent biases (M. Jrgensen, Realism in Assessment of vacant land.The estimate for the first building might have
Effort Estimation Uncertainty: It been +/25 percent of the final cost.
Matters How You Ask, IEEE Estimates are typically By the third, the estimate is proba-
Trans. Software Eng., vol. 30, no. 4, bly +/5 percent. But consider the
Apr. 2004, pp. 209217). Examples
the 50-percent view, impact of changing the original
of complementary methods are meaning the probability plan. Instead of using brick and
mortar, you use precast concrete
expert opinion with analogy, for being under or over because of the expected cost sav-
expert opinion with function budget is the same. ings. Or, imagine you need to build
points, on a rocky hillside instead of flat
algorithmic models and use case vacant lots. And you decide to add
points, a penthouse. What will happen to your estimate? Plus or
expert opinions by experts with different project expe- minus 25 percent will be an achievement.
riences and responsibilities and The same is true of software projects. If youre building
use case points and analogy systems that are similar to those youve previously built
using similar teams and technologies, and you can manage
Algorithmic models based on the same underlying algo- the requirements processes, your estimates will be quite
rithms (such as all of the COCOMO derivatives) have the accurate. If youre building fundamentally new systems,
same biases as do experts who have had similar responsi- however, where invention is required, +/25 percent will
bilities on similar projects. (M. Jrgensen, Practical be an achievement.
Guidelines for Better Software Estimation, www.simula.
no/departments/engineering/.artifacts/ieee_jorgensen_ ESTIMATE UNCERTAINTIES
guidelines5.pdf). You must remember that estimates are probabilistic and
communicate them appropriately. All projects have some
SOFTWARE ESTIMATION AS ENGINEERING uncertainties. Estimates are typically the 50-percent view,
With all the available methodologies and methods, why meaning the probability for being under or over budget is
are the estimation results so ragged? In some respects, the the same. (Unfortunately, Parkinsons Law, which states that
question comes down to whether software estimation is work expands to meet the time available, holds for software
an engineering process or not. projects. So, this 50-percent view says well be on budget 50
Consider Terry Bollingers statement, if software esti- percent of the time and over budget the other 50 percent.)
mation is believed to be a codifiable engineering process To estimate more accurately, you must
analogous to house building then litigation is a reasonable
and expected consequence of inaccurate estimations understand, accept, and manage estimations inherent
(Bollinger 1997). uncertainties; and
The software engineering community is split into change the estimation terminology to include the uncer-
two camps: tainties.

the process camp espouses that quality software can be
developed on time if a particular software process or
programming technology is used, while
the problem-solving camp believes that programming is
fundamentally a process of solving problems and as such
intrinsically resists codification.
The size of the estimates uncertainty differs based on
how close you are to the projects end, as Barry Boehm
classic chartthe cone of uncertaintyillustrated in 1981.
(Boehm, Software Eng. Economics, Prentice-Hall, 1981).
This chart represents the estimate accuracy at different
phases in the software life cycle. During the projects ini-

42 IT Pro November December 2006

tial feasibility phase, the estimates accuracy is a factor of
+/4. For example, if you estimate a project to be 200 staff
months, it could take anywhere from 50 to 800 staff
months. As you progress through the projects life cycle,
the estimate accuracy increases, such that at the detailed
design phase, its a factor of approximately 1.3 rather than
the initial 4.
Most practitioners agree that improved estimation prac-
tices and processes have changed the uncertainty scale. In
addition, developers have a much greater tendency to
underestimate (due to market and organizational pres-
sures, optimism, and overconfidence) than to overestimate.
Estimation is now typically closer to +100/50 percent at
the feasibility stage, but still not much better than +20/10
percent at the detailed design stage.
Three types of uncertainties exist: statistical variance,
known risks, and unknown risks.Youll always have statis-
tical variancethe only question is its size.Your risk man-
agement plan should address the known risks. Unknown
risks are obviously the toughest to handle. Organizations
typically use experts to minimize the number of unknown
risks. The Project Management Institute (PMI) recom-
mends maintaining a 10-percent management contingency
buffer in the absence of any other information.
Its also important to effectively communicate your esti-
mates. Instead of giving a number, you should give ranges
and views based on probabilistic distributions.The pX view
means there is an X percent probability of not exceeding
the estimate; thus, a p90 view suggests that the actual
results will be less than or equal to the estimate 90 percent
of the time (Jrgenson 2004).
Consider the difference in information between the fol-
lowing statements:
My estimate is 12 staff months.
My estimate is between 10 and 14 staff months.
My p50 estimate is 12 months.
I estimate that 95 percent of the time it will be between
10 and 14 months.
I estimate that 98 percent of the time it will be less than
14 months.
The first statement is the easiest for others to deal with.
Its precise, but you arent communicating any of the uncer-
tainties or risks.The other statements provide significantly
more information and lets others help manage both the
risks and the uncertainties.
Experts can specify the ranges as part of the estima-
tion process. Unfortunately, experiments have shown
that experts tend to be consistently overconfident in
their range estimations (M. Jrgensen, Practical
Guidelines for Expert-Judgment-based Software Effort
Estimation, IEEE Software, vol. 22, no. 3, May/June
2005, pp. 5763). The reasons suggested for their over-
confidence include the need to appear confident and
Table 1. Estimation uncertainty
engineering rules.
Project phase Uncertainty (percent)
Feasibility +100/50
Requirements +50/25
Design +20/10
Coding +10/5
Testing +5
professional, belief in their abilities, and lack of imme-
diate feedback. Experiments show that you can achieve
more accurate ranges using predetermined organiza-
tional guidelines or probability distributions based on
previous projects. I recommend following the engineer-
ing rules in Table 1 until you have your own organiza-
tional baseline. For example, if your current p50 estimate
is 50 staff months and youre in the design phase, using
these engineering rules will give you an expected range
between 47 and 56 staff months.
To use the distribution from previous projects to derive
the X-percent effort, start with your best estimate (deter-
mined through other methods). For example, assume
your estimate is 10 staff years and that of previous simi-
lar projects,
30 percent came in on budget,
10 percent came in under budget by 10 percent,
40 percent came in over budget by 50 percent, and
20 percent came in over budget by less than 50 percent.
Then your estimate percentages are:
p10 estimate = 9 staff years,
p40 estimate = 10 staff years
p80 estimate = 15 staff years.
For agile methods which emrace change, expect more
variation.Todd Little published data for 120 delivered sys-
tems based on the agile methodology (T. Little, Agility,
Uncertainty, and Software Project Estimation, www.
toddlittleweb.com/Papers/Agility,%20Uncertainty%20
and%20Estimation.pdf). For these systems, he found that
estimation accuracy follows a log-normal distribution
(that is, it was underestimated far more frequently than
overestimated),
initial estimates are targets with only a small chance of
being met,
the range between the target and an estimate with 90
percent confidence is about four times greater, and
November December 2006 IT Pro 43
 SOFTWARE ESTIMATION

this behavior and uncertainty range is nearly identical ESTIMATING EARLY AND OFTEN
at all stages in the project lifecycle, in conflict with theWhen should you estimate? And how much effort should
cone of uncertainty. you spend on it?
It depends on the project and the methodology. Effort
So, to deal with estimates uncertainties: estimation is usually overheadyou arent producing
a deliverable product. So, you need to ensure that the
change your terminology from one number to a pX benefit exceeds the cost. If development costs or sched-
number; ule dont matter, you dont need to estimate. Or if
use probability intervals to specify the range of esti- youre dealing with frequently changing requirements
mates; and specifications, you need to
use methods other than gut be able to give ballpark esti-
feelings for determining the When asked to provide mates quickly and efficiently,
range, because overconfidence an estimate, you must and expect to update them
is typical; (quickly and efficiently) when
if youre using agile methods, determine whether the situation changes.
especially if youre following youre being asked In some projects, estimates
the responding to change greatly impact the quality of the
over following a plan princi- for your view or being decision making. Im sure you
ple, expect a low probability of have horror stories like mine
meeting your initial estimates
told what the answer about approaches or projects
(perhaps 10 percent) and a must be. selected based on estimates that
large estimate uncertainty were later found to being totally
throughout your projects. inaccurate (and low).
For traditional developments, you should estimate the
MANAGING THE EFFORT BUDGET job at least three times:
A primary reason for projects coming in over budget is
the confusion between the project targetthat is, what the macroestimation during the feasibility phase,
senior management or marketing wants the cost to be detailed estimation during the requirements phase, and
and the actual estimate. refined estimation during the design phase.
When asked to provide an estimate, you must deter-
mine whether youre being asked for your view or being The need for an estimates accuracy is related to its pur-
told what the answer must be. Frequently, because of pose at a particular point in time.A cost estimate at a pro-
business realities, youre told the answerthat is, youre jects feasibility stage needs only be accurate enough to
given the jobs effort budget. In this case, the same support a decision to prepare a more detailed project def-
tools and methodologies still apply. But instead of using inition. Detailed product and project definitions arent
them to create an estimate, use them interactively to cre- available at this phase, so the estimate will be high level.
ate a rational plan to reduce the effort to meet the The estimate at the requirements specification stage, how-
budget. For example, you could reduce the number of ever, is a critical project decision point and must be rea-
adjusted use case points by reducing actors and scenar- sonably accurate.Youre potentially committing significant
ios, decreasing the technical factors, improving the envi- resources based on the business case, which is based on the
ronmental factors, or even reusing large chunks of code cost estimate.The required accuracy depends on the busi-
and classes. Infinite potential solutions exist, some fea- ness model. An in-house project or consulting service
sible, some not. might be pay as you go, so the estimate is really about
Some people, when faced with an inadequate budget, try establishing credibility. For external product sales, or for
adjusting the estimation models parameters without fixed-cost projects, the required accuracy depends on when
changing the work involved. Unless you made mistakes in a final price will be set. Once the detailed design has been
the initial estimate, this will only create more overtime. completed, and hopefully most of the unknowns have
Others plan on overtime, which is best reserved for con- become known, you can create a reasonably accurate esti-
tingency and risk management. Have a plan you believe mate and plan.
you can meet without overtime. The third favorite tech- Because of the inherent uncertainties in all projects, you
nique is cutting features. This option should be the last might need to re-estimate as you learn more. Whenever
resort, unless you can prove that the feature is truly major unplanned events occur, youll need to understand
unneeded or optional. Instead, use your creativity and how they impact the schedule and effort. Dont ignore
problem-solving skills to simplify the project or increase them and just hope it will all work out. It rarely will. At
productivity. least roughly estimate the impacts to make sure your plan

44 IT Pro November December 2006

is still viable. By estimating early and often, you can take
appropriate action to adjust the program and project plan
when you find that the work is not aligned with the budget
or the schedule.
THREE GOLDEN RULES OF ESTIMATION
There are three golden rules which will significantly
improve your estimations and estimation processes.
Require all estimates to be justified. Gut feeling is not an
adequate justification.
Dont use methods or tools blindly. Try estimating pre-
vious (completed) projects to validate and tune the
methods.
Educate your estimators. Knowing how to do something
doesnt mean you know how long it will take. Train peo-
ple in estimation.Accuracy is correlated with training and
the ability to see results, not development experience.
I
mproving your estimation skills, processes, and how you
communicate your estimations will help you avoid the
difficulties inherent in estimation; however, your abil-
ity to estimate well will always be limited by the extent of
your projects uncertainty.
Acknowledgments
The ideas and conclusions in this article are the culmi-
nation of my experience and my readings of the works of
others. Id like to thank all those I referenced, as well as
those I may have inadvertently missed, for which I apolo-
gize. Id especially like to thank Magne Jrgensen and the
folks at Simula Laboratories. Their work is fundamental
to my conclusions and recommendations.
Linda M. Laird is the author of Software Measurement
and Estimation:A Practical Approach, published recently
by IEEE and Wiley. She is an adjunct professor at Stevens
Institute of Technology, Hoboken, New Jersey. Contact her
at linda_m_laird@msn.com.
For further information on this or any other computing
topic, visit our Digital Library at http://www.computer.
org/publications/dlib.
November December 2006 IT Pro 45