Hollmann Hydropower

Reprinted with the permission of AACE
1265 Suncrest Town Centre Dr.,

International
Morgantown,
Phone 800-858-COST/304-296-8444
WV 26505 USA
2014 AACE® INTERNATIONAL TECHNICAL PAPER Internet: http://www.aacei.org
E-mail: info@aacei.org
RISK.1721
Variability in Accuracy Ranges:

A Case Study in the
Canadian Hydropower Industry
John K. Hollmann, PE CCP CEP DRMP,
Raminder S. Bali, P.Eng, John M. Boots, P.Eng,
Chantale Germain, P.Eng,
Michel Guevremont, P.Eng PSP,
and Kelman Kai-man Ng, P.Eng
Abstract—This paper presents a case study of the variability in accuracy ranges for cost
estimates in the Canadian hydropower industry. The study sought to improve the participants’
understanding of risks and estimate accuracy for their hydropower projects of similar scope.
The study team also sought to verify the theoretical accuracy curves identified in AACE
International’s Recommended Practice (RP) 69R-12: “Cost Estimate Classification System – As
Applied in Engineering, Procurement, and Construction for the Hydropower Industry”. The
study team collected and analyzed actual and phased estimate cost data from 24 projects with
actual costs from 50 million to 3.6 billion (2012$CAN) completed from 1974 to 2014.
Greenfield, brownfield and revamp impoundment and hydropower generation facility projects
from across Canada were included (power transmission projects were excluded.) The study
found that the range bandwidth (uncertainty) in RP 69R-12 is understated. Further, because
actual contingency estimates are biased too low, the actual range curves are biased very high
relative to those in RP 69R-12. The accuracy ranges and the underestimation of contingency are
similar for hydropower and process industry projects.
RISK.1721.1
Copyright © AACE® International.
This paper may not be reproduced or republished without expressed written consent from AACE® International
2014 AACE® INTERNATIONAL TECHNICAL PAPER
Table of Contents
Abstract ....................................................................................................................................... 1
List of Figures .............................................................................................................................. 2
List of Tables ................................................................................................................................ 2
Introduction ................................................................................................................................ 3
Background on the Study ............................................................................................................ 4
Analysis Approach ....................................................................................................................... 4
Findings for Accuracy Range by Class: Descriptive Statistics ...................................................... 6
Comparison of Findings to Other Studies and AACE RP69R-12 .................................................. 7
Comparison of Contingency Estimates to Actual Cost Growth .................................................. 8
Findings for Other Risk Drivers from Regression Analysis .......................................................... 9
Conclusion ................................................................................................................................... 9
References ................................................................................................................................... 10
List of Figures
Figure 1 – Example of the Variability in Accuracy Ranges for a Hydropower ............................ 3

Industry Estimate (Figure 1 from AACE International RP 69R-12)
Figure 2 – Dataset Actual/Base Estimate Metrics Fitted to Lognorm Distributions…………………. 6
Figure 3 – RP 69R-12 Figure 1 with p10/p90 Study Data Superimposed..………………………………… 8
List of Tables
Table 1 – Dataset Cost Estimate Accuracy Metrics (Actual/Base Estimate) ............................... 6

Table 2 – Comparison of Accuracy Studies (% Overrun of Estimate Including Contingency) .... 7
RISK.1721.2
Introduction
Accuracy is a measure of how a cost estimate differs from the final actual outcome. Risk
analysis provides forecasts of how the final actual outcome may differ from the estimate (such
as a base estimate or an amount approved for expenditure). Historical analysis helps us to
understand the variability of accuracy and to improve our risk analysis practice [1]. This study is
such an historical analysis.
Empirical estimate accuracy data has been researched for over 50 years [2]. In particular, the
accuracy of process industry project estimates (e.g., oil and gas, chemical, mining, etc.) has
been well documented [3]. Other studies have highlighted industry bias and misperceptions of
the reality of estimate accuracy [4]. However, there has been a relative void in accuracy studies
for hydropower projects with the notable exception of studies of World Bank funded projects;
mostly in developing countries [5,6]. This study of the accuracy of estimates for the well
developed Canadian hydropower industry will help fill a gap in our understanding of the
hydropower industry.
In addition, this study was needed to help verify the applicability of the theoretical accuracy
depiction presented in Figure 1 of AACE International’s new Recommended Practice 69R-12:
“Cost Estimate Classification System – As Applied in Engineering, Procurement, and
Construction for the Hydropower Industry” [7]. The questions in regard to that RP were “does
Figure 1 in RP 69R-12 reflect real accuracy ranges?” and if not, “how can we assure that this
depiction does not feed bias in stakeholder expectations?” (Figure 1 from RP 69R-12 is
reproduced below):
Figure 1 – Example of the Variability in Accuracy Ranges for a Hydropower Industry

Estimate (Figure 1 from AACE International RP 69R-12; copied with permission)
RISK.1721.3
Background on the Study
RP 69R-12 referenced above resulted from a multi-year effort led by a team of Canadian
hydropower cost experts. The initial RP goal was to document the defining scope deliverables
and their expected status to support hydropower project estimates of each Class. After
publication of the RP, the next step was to verify theoretical Figure 1 in the RP. To do this, the
Canadian study team performed an empirical analysis. The project scope and cost details of the
analysis are confidential to the team, but it is hoped that the reference information presented
here will be useful for AACE International to improve RP 69R-12.
The Canadian study team collected estimated and actual project capital cost data from 24
projects with actual costs from $50 million to $3.6 billion (in 2012 $CAN) completed from 1974
to 2012. For each project, estimate data from each scope development phase was captured,
resulting in data on 50 estimates. All projects had a Class 3 estimate, but some did not have
Class 4 and/or 5. The project scopes included greenfield, brownfield and major revamp
impoundment and power generation facilities on rivers across Canada. It excluded power
transmission projects. Most of the projects were located in semi-remote areas and included
camps, mass excavation, concrete and/or earth-filled impoundments and diversions, intake
structures, penstocks, and power houses with turbine generation equipment. To minimize bias,
the dataset represented all the recent major project data available to the participants
regardless of whether the project cost outcome met company objectives.
Analysis Approach
The primary analytical methods used were descriptive statistics and multi-variable linear
regression. The accuracy metric described by the statistics and the dependent variable of
regression was the ratio of “base estimate/actual costs”. “Base estimates” exclude contingency,
escalation and management reserves. This was used because the team wanted to understand
how actual costs differed from the base so that they could improve future predictions of this
difference (i.e., predict contingency required). The study also examined schedule duration
estimate accuracy which is not included in this paper.
The estimate/actual cost ratio was used because it tends to be close to a normal distribution
and hence is amenable to linear regression analysis. As will be discussed later, the more
commonly considered actual/estimate inverse tends to be biased to the high side which makes
regression analysis problematic.
The primary independent variables examined included (drivers of accuracy outcomes):
• Scope definition (i.e., Class)

• Cost content (e.g., % equipment or impoundment)
• Location/Company
• Proximity to populated areas
RISK.1721.4
• Cost/Schedule Strategy (i.e., cost or schedule driven)

• Terrain/Site Conditions/Weather
• New Technology or Scale
• System Complexity
• Execution Complexity
• Primary Project Type (e.g., greenfield, revamp, etc.)
• Primary Construction Contract Type
• Owner PM System Maturity
To collect the data, the team developed a form that captured the following:
• General project characteristics

• High level “base” cost estimate breakdowns at each AACE Class (per 69R-12) plus
contingency and escalation cost estimates for each
• Actual final cost
• Key planned and actual schedule milestones
• Scope change and risk event information
The actual cost data was normalized to the year of the respective estimate using the mid-point
of spending approach (actual project cash flows were not available) [8]. The normalization price
index used was derived from Statistics Canada indices for the sell price of non-residential
construction projects. Also, cost changes due to business scope change were adjusted out
(costs resulting from a change to a basic premise of the estimate such as generation capacity or
throughput.) None of the projects were observed to have experienced a catastrophic risk event.
The primary variable (risk driver) of interest was the level of scope definition. Not all projects
had data for estimates of each AACE Class as can be seen in the following number of valid
observations:
• Class 3: 21 (a group of 4 projects in a program were combined into one)

• Class 4: 17
• Class 5: 12
Data for 3 projects was excluded because extreme age and/or duration raised questions as to
the validity of the normalization. This sample size was considered adequate to gain useful
insight as to the relationship of accuracy and Class, but not enough to gain deep understanding
of the impact on accuracy of any but the most dominant of the other independent variables.
The linear regression was performed using Microsoft Excel® with an add-on package called
Analyse-it® that provides additional modeling, diagnostic and graphical capabilities.
RISK.1721.5
Findings for Accuracy Range by Class: Descriptive Statistics
Table 1 shows the dataset statistics for accuracy. Figure 2 depicts the same data fitted to
lognorm distributions. The probability values (“p-value” is the level of confidence expressed as
a percentage of values that will be less than that shown) in the table are calculated using the
Excel “Norminv” function applied to the base estimate/actual data, and then converted to the
traditional actual/base estimate ratio format (i.e., >1 means the actual cost was more than the
base estimate.) This method of inferring the population distribution from a sample is consistent
with the method described in AACE International RP 42R-08 and supported by process industry
research that indicates that estimate/actual data (as opposed to its inverse of actual/estimate)
is more or less normally distributed [1].
As an example of how to interpret this, if the ratio for Class 3 at p50 is 1.24, that indicates that
24% contingency would be needed to achieve a 50 percent confidence of underrunning. Note
the high side skewing (e.g. the Class 5 p90 of 3.01 is much further from the mean than the p10
value). Recall that these values exclude escalation and business scope change.
Actual/Base Estimate Class 3 Class 4 Class 5

number of observations 21 17 12
Mean 1.24 1.40 1.79
p90 1.63 2.09 3.01
p50 1.24 1.40 1.79
p10 0.99 1.06 1.27
Table 1 – Dataset Cost Estimate Accuracy Metrics (Actual/Base Estimate)
Figure 2 – Dataset Actual/Base Estimate Metrics Fitted to Lognorm Distributions
RISK.1721.6
Comparison of Findings to Other Studies and AACE RP69R-12
Statistically speaking, considering sample sizes and data quality, this study’s accuracy ranges are
comparable to those reported for the process and infrastructure industries [3 and 4] as well as
hydropower projects funded by the World Bank [5]. Table 2 summarizes the results of these
studies. It was assumed that funding estimates in studies [4] and [5] were based on about Class
4 scope definition because general industry front-end planning is assumed to be less defined
than planning at the companies in this study and at the clientele of Independent Project
Analysis, Inc. (IPA). Note that this study’s values were adjusted downward from Table 1 to
reflect the accuracy relative to the estimate including contingency (i.e., the funded amount)
which is the data shown in most published studies. The contingencies added to this study’s
Class 3, 4 and 5 base estimates were 10%, 12% and 15% respectively which correspond to
typical contingencies applied at the time.
This Study, Canadian Hydro Class 3 Class 4 Class 5

p90 53% 97% 186%
p50 14% 28% 64%
p10 -11% -6% 12%
IPA Inc., Process Industry [3]; p10/p90 approximated from histogram illustration
p90 40% 70% 200%
p50 1% 5% 38%
p10 -15% -15% -15%
Hollmann, Process Industry [4] average of meta-analysis
p90 70%
p50 21%
p10 -9%
Merrow, Hydro [5] Mean & Std Dev Reported; Normal distribution assumed below)
p90 (assuming normal) 65%
Mean 24%
p10 (assuming normal) -17%
Table 2 – Comparison of Accuracy Studies (% Overrun of Estimate Including
Contingency)
When comparing results in respect to RP 69R-12, one must consider two points of comparison.
The first is the bandwidth or span of the range (i.e., p90 minus p10.) The other is the absolute
value of a high or low range. Figure 3 shows this study’s results superimposed on the RP 69R-12
Figure 1. This study’s range spans are somewhat wider (more uncertain) than the worst case
spans in the RP. For example, the worst case span for Class 5 in the RP is 150% (100 – <50>)
while the span for Class 5 in this study is 174% (186 – 12.) The high and low absolute range
values indicate strong contingency under-estimation bias. Note that all the projects in the study
were greater than $50 million (in 2012 $CAN); the findings may not apply to small projects
where estimating practices often differ. [4]
RISK.1721.7
Figure 3 – RP 69R-12 Figure 1 with p10/p90 Study Data Superimposed
Comparison of Contingency Estimates to Actual Cost Growth
The projects in this study allowed only 10-15% contingency on average, even for Class 5
estimates. These contingencies appear to reflect a strong industry optimism bias. For example,
a 2012 white paper by the United States Society on Dams, suggested “An overall contingency of
as much as 50 percent is appropriate” on the conceptual engineer’s estimate (i.e., Class 5);
however, this maximum (“as much as”) contingency allowance is much less than this study’s
mean cost growth of 79% at Class 5 [9].
Contingencies (or combinations of contingency and management reserve) of 24, 40 and 79% at
p50, excluding business scope change and escalation, are suggested by this study for Class 3, 4
and 5 estimates respectively for projects of average risks. If these contingencies had been
included in the study projects, their actual range outcome would look similar to but wider than
the worst case of RP 69R-12 Figure 1. The authors are not recommending that these or any
other contingency values be assigned arbitrarily; contingency should always be based on risk
analyses. However, if a company’s risk analyses regularly result in 10-15% contingency and
narrow ranges, it is likely that risks and their impacts are not being identified or quantified
properly and/or optimism bias is controlling.
RISK.1721.8
Findings for Other Risk Drivers from Regression Analysis
An attempt was made to quantify the impacts of systemic risks other than the level of scope
definition. To do this, the data from only the Class 3 estimates was examined. Class 3 is usually
the basis for full funding decisions and hence of utmost importance to the stakeholders. Each
independent variable (risk driver) was tested alone and in various combinations using Excel
with Analyze-it.
A regression model quantifying the cost growth for Class 3 estimates was developed that had
an R2 of 0.66. Because the dataset had only 21 observations, the actual model is not shown
here to avoid any misuse (findings may not be generally applicable,) but narratively speaking,
the following variables appear to be significant systemic risk drivers:
• Proximity: The greater the distance of the project from a large population center, the
greater the cost growth. Given the effect of distance on material and labor availability
and conditions, this seems rational.
• Size: Larger projects had less cost growth. This may be due to larger projects being the
sum of parts with highs and lows that balance out, and/or smaller project estimates may
be small because of bias towards lowering base costs resulting in greater cost growth.
• Months Execution Duration: Longer projects had more cost growth despite normalizing
for escalation. This may reflect the fact that risks often drive both cost and schedule
increases rather than a causal correlation. However, the more time that passes, the
more chance that there will systemic changes in the social, political, regulatory and
other environments.
• % Equipment in Estimate: The greater the proportion of equipment, the less the cost
growth. Excluding scope change, most estimators would agree that major equipment is
less subject to risk and uncertainty than labor and bulk material costs, particularly for
impoundments subject to geologic risks.
The analysis above was repeated with Estimate Class added as an independent variable. This
could serve the participant’s as a rudimentary parametric model for systemic risk analysis [1].
While the model is confidential, it can be said that the model coefficients for Class (the level of
scope definition) are consistent with the relative range values for each Class in Table 1 and
Figure 1. The level of scope definition is clearly the predominant systemic risk driver.
Conclusion
This study of the variability in accuracy ranges for cost estimates in the Canadian hydropower
industry suggests that the actual cost uncertainty is a bit greater than the worst case theoretical
depiction of accuracy in Figure 1 of RP 69R-12. The study indicated that risks are much greater
than being estimated; contingencies of 24, 40 and 79 percent were indicated for Class 3, 4 and
5 estimates respectively on average. Our study shows that the contingency and reserves
estimated were lower than what were required. However, the Canadian hydro industry
RISK.1721.9
experience is similar to that of other process industry projects, as well as of hydropower

projects in other regions funded by the World Bank.
Using the data from the study, the participants developed a simple parametric risk analysis tool
for systemic risks in which the level of scope definition as the dominant risk driver. This
emphasizes the importance of doing disciplined Class 3 scope definition prior to full funds
authorization if cost predictability is a goal. The Canadian hydro study team will recommend
that AACE’s Cost Estimating Technical Committee consider improvements to Figure 1 and
related content in RP 69R-12 to reflect the findings of this study. The conclusions are applicable
to other process related industries, and therefore this paper may encourage improvements in
other estimate classification recommended practices.
References
1. AACE International, “Risk Analysis and Contingency Determination Using Parametric

Estimating”, Recommended Practice 42R-08, AACE International, Morgantown, WV, (latest
revision).
2. Linder, Stefan, “Fifty years of Research on Accuracy of Capital Expenditure Project Estimates:
A Review of the Findings and their Validity”, Center for Research in Controllership and
Management, WHU - Otto Beisheim Graduate School of Management, Vallendar, Germany,
April 2005.
3. Ogilvie, Alexander, Robert A. Brown Jr., Fredrick P. Biery, and Paul Barshop, “Quantifying
Estimate Accuracy and Precision for the Process Industries: A Review of Industry Data”, Cost
Engineering, November/December 2012, AACE International, Morgantown, WV.
4. Hollmann, John K., “Estimate Accuracy-Dealing with Reality”, Cost Engineering,
November/December 2012, AACE International, Morgantown, WV.
5. Merrow, Edward W. and Brett R Schroeder, “Understanding the Costs and Schedule of
Hydroelectric Projects”, 1991 AACE International Transactions, AACE International,
Morgantown, WV, 1991.
6. Bacon, Robert W, John E. Besant-Jones, and Jamshid Heidarian, “Estimating Construction
Costs and Schedules; Experience with Power Generation Projects in Developing Countries”,
World Bank, Washington DC, 1996.
7. AACE International, “Cost Estimate Classification System – As Applied in Engineering,
Procurement, and Construction for the Hydropower Industry”, Recommended Practice 69R-
12, AACE International, Morgantown, WV, (latest revision).
8. AACE International, “Escalation Principles and Methods Using Indices”, Recommended
Practice 58R-10, AACE International, Morgantown, WV, (latest revision).
9. United States Society on Dams, “Guidelines for Construction Cost Estimating for Dam
Engineers and Owners” (white paper), US Society on Dams, May 2012.
RISK.1721.10
John K. Hollmann, PE CCP CEP DRMP

Validation Estimating, LLC
jhollmann@validest.com
Raminder S. Bali, P.Eng

Manitoba Hydro
rbali@hydro.mb.ca
John M. Boots, P.Eng

BC Hydro
john.boots@bchydro.com
RISK.1721.11
Chantale Germain, P.Eng

Hydro Quebec
germain.chantale@hydro.qc.ca
Michel Guevremont, P.Eng PSP

Hydro Quebec
guevremont.michel@hydro.qc.ca
Kelman Kai-man Ng, P.Eng

BC Hydro
kelman.ng@bchydro.com
RISK.1721.12

Hollmann Hydropower

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Hollmann Hydropower

Uploaded by

Copyright:

Available Formats

Reprinted with the permission of AACE

1265 Suncrest Town Centre Dr.,

Variability in Accuracy Ranges:

Figure 1 – Example of the Variability in Accuracy Ranges for a Hydropower ............................ 3

Table 1 – Dataset Cost Estimate Accuracy Metrics (Actual/Base Estimate) ............................... 6

Figure 1 – Example of the Variability in Accuracy Ranges for a Hydropower Industry

Background on the Study

The primary independent variables examined included (drivers of accuracy outcomes):

• Scope definition (i.e., Class)

• Cost/Schedule Strategy (i.e., cost or schedule driven)

• General project characteristics

• Class 3: 21 (a group of 4 projects in a program were combined into one)

Findings for Accuracy Range by Class: Descriptive Statistics

Actual/Base Estimate Class 3 Class 4 Class 5

Figure 2 – Dataset Actual/Base Estimate Metrics Fitted to Lognorm Distributions

Comparison of Findings to Other Studies and AACE RP69R-12

This Study, Canadian Hydro Class 3 Class 4 Class 5

Figure 3 – RP 69R-12 Figure 1 with p10/p90 Study Data Superimposed

Comparison of Contingency Estimates to Actual Cost Growth

Findings for Other Risk Drivers from Regression Analysis

experience is similar to that of other process industry projects, as well as of hydropower

1. AACE International, “Risk Analysis and Contingency Determination Using Parametric

John K. Hollmann, PE CCP CEP DRMP

Raminder S. Bali, P.Eng

John M. Boots, P.Eng

Chantale Germain, P.Eng

Michel Guevremont, P.Eng PSP

Kelman Kai-man Ng, P.Eng

You might also like