1

RELIABILITY AND ACCURACY OF ESTIMATES

Steven G.J. Boeschoten

Abstract

Many a book, paper and recommended practice document in Cost Engineering
literature deals with estimate accuracy. In these documents, it is usually assumed
implicitly that an accurate estimate is also a reliable estimate. However, judging
from dictionary definitions of accuracy and reliability, it is easy to see that this
assumption is not correct.
Accuracy is defined as the degree of conformity of a measure to a standard or
true value. In the case of estimates, the standard or true value is the base
estimate and the degree of conformity is the ranges of the line items.
Reliability is defined as the extent to which a test, experiment or measuring
procedure yields the same result on repeated trials. In the case of estimates, the
procedure is the estimating process as a whole.
In other words, reliability refers to qualifications of the input and work process of the
preparation of the estimate, whereas accuracy refers to qualifications of the output
of the work process, i.e. the estimate itself.
In this paper, the feasibility of a reliability classification system will be discussed, as
well as the question of how such system relates to the existing Cost Estimate
Classification System

Introduction

The white paper "What are Cost Engineering, Quantity Surveying and Project
management [3], developed during the 1996 and 2000 ICEC congresses, defines
the function of cost engineering as follows:
"To provide independent, objective, accurate, and reliable capital and operating cost
assessments usable for investment funding and project control.
In this definition, "independent and "objective refer to the function of cost
engineering while "accurate and "reliable refer to the assessment, the product, the
deliverable.
Every single person involved in cost engineering has their own ideas about what an
independent and objective cost engineer is or should be and when an estimate is
accurate and reliable, but actually reaching consensus over the exact meaning of
these words is fraught with difficulty.
It seems then that we need to create a yardstick against which each of these four
terms can be measured

Independent and Objective

The exact meaning of "independent and "objective has already adequately been
discussed in some detail. In July 2000, the International Cost Engineering Council
formally adopted the ICEC Canons of Ethics for Cost Engineers, Quantity Surveyors
and Project Management.
The Canons are based on the "Preamble" and "Fundamental Canons" of the "Code of
Ethics for Engineers" and are reprinted below by permission of the National Society
of Professional Engineers.
The behavioral issues of the profession are dealt with in six short statements.


2
ICEC Canons of Ethics for Cost Engineers,
Quantity Surveyors, and Project Managers*
(Adopted by the ICEC Executive, 21 July 2000)

Cost engineering, quantity surveying, and project management are important and
learned professions. Members of these professions are expected to exhibit the
highest standards of honesty and integrity. The services provided by cost engineers,
quantity surveyors and project managers require honesty, impartiality, fairness and
equity, and must be dedicated to the protection of public health, safety, and welfare.
All members of these professions must perform under a standard of professional
behavior that requires adherence to the highest principles of ethical conduct.

Cost engineers, quantity surveyors, and project managers, in the fulfillment of their
professional duties, shall:

1. Hold paramount the safety, health and welfare of the public.
2. Perform services only in areas of their competence.
3. Issue public statements only in an objective and truthful manner.
4. Act for each employer or client as faithful agents or trustees.
5. Avoid deceptive acts.
6. Conduct themselves honorably, responsibly, ethically, and lawfully so as to
enhance the honor, reputation, and usefulness of their professions.


Reliability and Accuracy

At the 14
th
meeting of the International Monetary Fund in Tokyo, October 2001 [4] a
useful set of definitions was presented:

Reliability of an estimate refers to the closeness of the initial estimated value(s) to
the subsequent estimated values

Accuracy of an estimate refers to the closeness between the estimated value and the
(unknown) true value that the statistics were intended to measure

These definitions exclude a statement concerning the scope on which the estimates
are based and apparently assumes a definition of estimate being the cost
assessment of an agreed scope. This assumption is in line with the generally
accepted definition of estimate.

Applying these definitions to the estimating process an initial (class 5) estimate is
reliable if the subsequent (classes 4 to 1) estimates are close to the initial estimate.
An estimate is accurate if the true value, the as-built costs, is close to the initial
estimate(s). So accuracy refers to the closeness of the as-built cost to the class 5
estimate, to be followed by the as-built cost to the class 4 estimate etc.

It is these "closenesses that need further discussion. In fact, there are two distinctly
different types of closeness: one is between estimates, i.e. the reliability, the other is
between the as-built costs and the estimates, i.e. the accuracy.
The reliability closeness is driven by the quality of the definition of the project scope.
The better the scope has been defined, the more reliable the ensuing estimate will
be. In other words, reliability is high if few scope changes are expected and therefore
3
low if many scope changes are expected. Apart from the number of changes, the
value of the changes are also a determining factor for the reliability.
The accuracy closeness is driven by the quality of the measurement or estimation of
the scope.

Every time an estimate is made, the scope is treated as fact and during the
preparation of an estimate, previous estimates are of no influence. An estimate is
just a measurement of a given situation and as a result measurements can be made
at various levels of accuracy.

The origins of changes

Both reliability and accuracy are driven by change. If something changes the
estimate as a whole, it will influence the reliability or the accuracy and, in many
cases, both reliability and accuracy.
It is therefore of the utmost importance to understand changes and what impact
changes have on reliability and accuracy.

The fact that projects and estimates are perpetually affected by change is a well-
known given. Such changes may vary considerably in nature, impact and origin.

Depending on their role in the entire process, every stakeholder has a different
perception of change.

For example, a change in the piping route that originates from a constructability
review, constitutes a scope change for the piping contractor, a design change for the
engineer, but does not constitute a change for the owner.
If, on the other hand, the change in the piping route originates from request from
the owner, all parties will consider it a change.
The owners perception of change is rather straightforward:
as long as he does not change anything, he expects the engineering contractor to
execute the project at the estimated cost. The estimated cost includes allowances for
contingency, escalation and field change allowances in order to cover for
uncertainties, inaccuracies and variations. He is
the only one allowed to change the scope of the project.
This does not necessarily mean that only owners can initiate changes. During the
development of the design and even during construction, situations may arise that
result in a scope change by the owner, but in such cases prior approval is required
from the owner before can be proceeded.

The relationship between the engineer and the contractor may be seen in similar
terms. In general, the engineer expects the contractor to deliver the agreed scope at
the agreed estimate. The estimate should cover everything except scope changes
from the engineer.

The conclusion of the above is that we have two types of changes:
Changes that affect the reliability of the estimate
Changes that affect the accuracy of the estimate.

The estimate input checklist and maturity matrix from the AACEI estimate
classification system [1] are used as a starting point for the discussion on the origins
of changes:

4

General Project Data Changes are covered by
Project Scope Description ???
Plant Production/ Facility Capacity ???
Plant Location ???
Soils & Hydrology ???
Integrated Project Plan ???
Project Master schedule Escalation
Escalation Strategy Escalation
Work Break Down Structure ----
Project Code of accounts ----
Contracting Strategy Contingency
Engineering Deliverables
Block Flow Diagrams Contingency
Plot Plans Contingency
Process Flow Diagram Contingency
Utility Flow Diagram Contingency
Piping & Instruments Diagram Contingency
Heat & Material Balances Contingency
Process Equipment List Contingency
Utility Equipment List Contingency
Electrical One-Line-Diagram Contingency
Specifications & Datasheets Contingency
General Equipment Arrangement Drawings Contingency
Spare Parts Listing Contingency
Mechanical Discipline Drawings Field change allowance
Electrical Discipline Drawings Field change allowance
Instrumentation/ Control System Discipline
Drawings
Field change allowance
Civil/Structural/Site Discipline Drawings Field change allowance


The matrix shows two sets of input: General Project Data and Engineering
Deliverables.
Engineering Deliverables are produced under the assumption that the General
Project Data do not change. Changes within Engineering Deliverables influence the
accuracy closeness. In the estimate, the consequences of these changes are covered
by contingency, escalation and field allowances.
The changes in Project Scope Description, Plant Production/ Facility Capacity, Plant
Location, Soils & Hydrology and Integrated Project Plan are not covered in the
estimate and as a result the consequences of changes in these inputs have never
been taken into consideration yet. Changes in General Project Data will influence the
reliability of the estimate.

The accuracy of estimates and how to deal with this phenomenon has been
abundantly discussed in scores of papers, books, and recommended best practices.
The remainder of this paper will focus on the concept of estimate reliability.





5
The Problem
In 1987, Robert Kimmons wrote [5]:
An idealized picture of a cost forecasting program is shown in figure 1. This figure
shows that at any point of time during the life of a project, there are four elements
of costs.

The defined elements covered by the approved scope at that point and including all
of the actual costs to date.
The undefined elements, implied by the scope, which are referred to as contingency.
The escalation on costs of all items contained in the approved job scope
The "job growth, or everything not part of the current approved scope, but will be
added to the project as it proceeds.
Figure 1 shows the relationship of these four elements over time. Gradually, as the
job progresses, the contingency, escalation and job growth become defined
elements. At the end of the job, the defined elements represent all costs, having
replaced the other three completely.
defined
contingency
escalation
scope
changes
5 4 3 2 1


If scope changes are not considered while preparing the earlier estimates it could
mean for example, that the class 5 estimate is 25% too low. The class 5 estimate is
however presented as the best possible estimate at that point in time, covering the
agreed scope. The fact that the class 5 estimate might be too low makes this
estimate unreliable.
The answer to the question "Does this class 5 estimate really represent the final
costs is, "We dont know. Needless to say that most the senior managers are far
from impressed.

The origin of the unsatisfactory answer is that the initially agreed scope might differ
from the as-built scope. Although such differences might be anticipated, they cannot
be measured. After all, no cost engineer is clairvoyant. It is, however possible, to
identify the drivers that will influence the agreed scope. These drivers are laid down
in the business case and they are also the starting point for the estimating process.

The techniques for estimating the defined elements, contingency and escalation are
well known and address the accuracy issues of the estimating process. All these
techniques measure a given scope and the better this scope has been defined, the
more accurate the estimate will be.


6
Analysis
The original definition that reliability of an estimate refers to the closeness of initial
estimates to subsequent estimates can now be rephrased as reliability of an estimate
is influenced by the scope changes of an (initial) estimate resulting in subsequent
estimates.

Ideally, a system should be built to determine the monetary value of the changes at
the time the estimate of the agreed scope is prepared without having to define and
estimate more than one scope.
Below follow some examples:
The agreed capacity of a given facility is set at 100.000 Tons per Annum (TpA). What
if the capacity is set at, for example, 150.000 tons?
The agreed location is US Gulf Coast. What if we relocate to Western Europe?
What if the product specification is changed?
What if the technology is changed?
What if all of the above changes occur, so that capacity increases and technology is
simultaneously simplified?
It could be that the impacts are neutralizing each other.
A change-quantification system might be the answer, but that would require an
exhaustive analysis of a large project database. At this moment, this is, however,
not feasible.

However, I would like to propose an alternative: the reliability rating system.

The Reliability Rating System (RR)
At the risk of repeating myself, the reliability of an estimate refers to the closeness
of the initial estimate to subsequent estimates. The drivers that determine the
closeness are the input items that were used to establish the agreed scope.
These input items are laid down in a Business Needs Memorandum, a Business Case
or similar documents and are in line with the drivers are mentioned in the estimate
input matrix. However, the definitions of these input items are too abstract to be
useful for a reliability rating system.
By rating the driver items, the ensuing rating can be used in the discussion about
improving the reliability of an estimate
In Control & Management of Capital Projects [3] a definition-rating checklist is
suggested.
These two sets of items, the Business Needs Memorandum and the definition rating
checklist result in the Reliability Rating table.
In the table below, the drivers that may change the scope are entered and against
each item a score is to be entered; the higher the score for an item, the lower the
impact on reliability is. The total reliability score is low for low reliability and high for
high reliability.

Conclusion

The Reliability Rating system I propose can be a useful instrument in determining the
quality of a scope and the ensuing estimate, as well as create transparency in which
areas contain major uncertainties and which require further investigation.
The rating system is however half the answer. An analysis of a large project database will give us
the relationship between the rating and the percentage to be added to the estimate to cover the
un-reliability.

I will be back!!
7

REFERENCES
1. AACE International; Recommended Practice No. 18R-97 Cost Estimate
Classification System; Morgantown 1997
2. Hackney, Kohn W. Control & Management of Capital Projects; McGraw-
Hill Inc. ISBN 0-07-001259-8
3. International Cost Engineering Council; A white paper, what is CE, QS
and PM. Icoste.org
4. International Monetary Fund Assessing Accuracy and Reliability: A note
on Approaches Used in National accounts and Balance of Payment
Statistics; Fourteenth Meeting of the IMF committee on Balance of
Payment Statistics, Tokyo Japan October 24-26 2001
5. Kimmons, Robert L. What Management Expects of Cost Engineers; 1987
Transactions AACE International
8
9































FACTOR level 1 level 2 Level 3 level 4
FACILITY max 25 points

Annual capacity

Max 15 points
Capacity based on
confirmed long-term
sales plans
Capacity based on
confirmed long-term
sales plans
Capacity confirmed Assumed; No firm figures

Operational requirements

Max 5 points
Operating manuals ,
plans etc available
User Requirements
Specifications
Operating philosophy &
automation philosophy
No Information available

Safety Health
Environment (SHE)
Max 5 points
All Recommendations
from SHE assessment
planned
All Recommendations
from SHE assessment
identified.
Preliminary SHE assessment
available
SHE Requirements not identified
PRODUCT Max 30 points

Product quality

Max 20 points
Re-supply to market Quality approved Quality checked with market Theoretical

Raw materials

Max 10 points
Facility operational Final study assumed of quantities &
quality
assumed of quantities & quality
TECHNOLOGY
Max 30 points


Technology Maturity

Max 15 points
Full scale Pilot Plant scale Lab scale Theoretical

Technology status

Max 10 points
Extension of existing
technology
New application for
known technology
New for the company New for the industry

Patents


Max 5 points
Existing patents New patent approved Patents position unknown Patents position unknown
LOCATION Max 5 points

Site selection


Max 5 points
Site selected; All
required information
available,
Site selected; All
required information
available, but not yet
assessed
Site selected; no information
on soil & hydrology; No utility
check
No information available
SCHEDULE
Max 10 points


Time to market

Max 5 points
Detailed project
planning
Agreed project planning Preliminary project planning Only assumed milestones

Execution Plan

Max 5 points
Contracting strategy,
Contract parties
involved.
Involved parties
identified globally
Agreed execution philosophy Assumed execution philosophy