Ce12 11

THE JOURNAL OF AACE INTERNATIONAL THE AUTHORITY FOR TOTAL COST MANAGEMENT
TM
COST
INTEGRATED COST-SCHEDULE
November/December 2012
ENGINEERING
www.aacei.org
RISK ANALYSIS
ESTIMATE ACCURACY:
DEALING WITH REALITY

QUANTIFYING ESTIMATE ACCURACY AND PRECISION FOR THE PROCESS INDUSTRIES:
A REVIEW OF INDUSTRY DATA

COMMON ERRORS IN
DEALING WITH PROJECT RISK
CONTENTS
COST ENGINEERING
TECHNICAL ARTICLES 5 Integrated Cost-Schedule Risk Analysis
Dr. David T. Hulett and Michael R. Nosbisch, CCC PSP
17 Estimate Accuracy: Dealing With Reality

John K. Hollmann, PE CCE CEP
28 Quantifying Estimate Accuracy and Precision for the Process Industries: A Review of Industry Data
Alexander Ogilvie, Robert A. Brown Jr., Fredrick P. Biery and Paul Barshop
39 Common Errors in Dealing With Project Risk

Joseph A. Lukas, PE CCE
ALSO FEATURED
2 2 16 38 AACE International Board of Directors Cost Engineering Journal Information 2012 Statement of Ownership 2013 Constitution and Bylaws Amendments 45 46 49 49 52 2012-2013 Nominating Committee The AACE International Online Store Professional Services Directory Index to Advertisers Calendar of Events
THE AACE INTERNATIONAL ONLINE BUTTON - This edition of the Cost Engineering journal has access to additional material on the AACE International website, www.aacei.org. Anytime you see the symbol at center, there is additional
content online associated with that article or feature. Direct your browser to www.aacei.org and look for the online button to access additional resources. If you are already reading the electronic version, just click the online button directly.
COST ENGINEERING NOVEMBER/DECEMBER 2012
AACE INTERNATIONAL BOARD OF DIRECTORS

PRESIDENT
Marlene Hyde, CCE EVP 303.940.3200 / president@aacei.org
CONTENTS
COST ENGINEERING
Established 1958
Vol. 54, No.6/November/December 2012
PRESIDENT-ELECT
John J. Ciccarelli, PE CCE PSP 303.940.3200 / preselect@aacei.org
Managing Editor Art Director Graphic Designer Advertising Sales
Marvin Gelhausen
mgelhausen@aacei.org
PAST PRESIDENT
Michael R. Nosbisch, CCC PSP 562.733.2472 / pastpres@aacei.org
Noah Kinderknecht
nkinderknecht@aacei.org
Cevin Thornbrugh
cthornbrugh@aacei.org
VICE PRESIDENT-ADMINISTRATION
Martin Darley, CCC FRICS 713.372.2426 / vpadmin@aacei.org
Garth Leech
304.296.8444 fax - 304.291.5728 gleech@aacei.org
VICE PRESIDENT-FINANCE
John C. Livengood, CFCC PSP 202.669.1360 /vpfinance@aacei.org
HEADQUARTERS
1265 Suncrest Towne Centre Dr Morgantown, WV 26505-1876 800.858.COST fax - 304.291.5728
VICE PRESIDENT-TEC
George Whyte, CCC CEP EVP 301.957.7434 /vptec@aacei.org
AACE International - The Authority for Total Cost Management
TM
VICE PRESIDENT-REGIONS
Julie Owen, CCC PSP 213.922.7313 / vpregions@aacei.org
OUR VISION - To be the recognized technical authority in cost and schedule management for programs, projects, products, assets, and services. OUR MISSION - The members of AACE enable organizations around the world to achieve thier investment expectations by managing and controlling projects, programs, and portfolios; we create value by advancing technical knowledge and professional development.
Cost Engineering (ISSN: 0274-9696/12) is published bi-monthly by AACE International, Inc, 1265 Suncrest Towne Centre Dr, Morgantown, WV 26505 USA. Periodicals postage paid at Morgantown, WV, and at additional mailing office. POSTMASTER: Send address changes to AACE International; 1265 Suncrest Towne Centre Dr, Morgantown, WV 26505-1876 USA. Customer #7012359 (APC), Publications Mail Agreement No 40624074, Return undeliverable Canadian addresses to PO Box 503, RPO West Beaver Creek, Richmond Hill, ON L4B 4R6. Single copies: US$9 members/ US$14 nonmembers (both + shipping), excluding special inserts available to AACE members only. Subscription rates: United States, US$72/year; all other countries, US$91/year. Overseas airmail delivery is available at US$99. Subscriptions are accepted on an annual-year basis only. Copyright 2012 by AACE International, Inc. All rights reserved. This publication or any part thereof may not be reproduced in any form without written permission from the publisher. AACE assumes no responsibility for statements and opinions advanced by the contributors to its publications. Views expressed by them or the editor do not necessarily represent the official position of Cost Engineering, its staff, or AACE International, Inc. Printed in York, PA, USA. Cost Engineering is a refereed journal. All technical articles are subject to review by a minimum of three experts in the field. To submit a manuscript for peer review, see author guidelines at www.aacei.org and submit a 200 word or less abstract to editor@aacei.org.. PHOTOCOPY PERMISSION: Authorization to photocopy articles herein for internal or personal use, or the internal or personal use of specific clients, is granted by AACE International, Inc., provided that the base fee of US$4.00 is paid directly to Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA. Telephone: 978.750.8400. For those organizations that have been granted a photocopy license by CCC, a separate system of payment has been arranged. The fee code for users of the transactional reporting service is ISSN-0274-9696/02 US$4.00. Payment should be sent directly to CCC. Copying for other than personal or internal reference use without the express permission of AACE is prohibited. E-mail requests for photocopy permission on bulk orders may be sent to editor@aacei.org. ADVERTISING COPY: Contact Garth Leech, 1265 Suncrest Towne Centre Dr., Morgantown, WV 26505-1876. Telephone: 304.296.8444. E-mail: gleech@aacei.com for rates. Advertisers and advertising agencies assume liability for all content (including text, representation, and illustrations) of advertisements printed and also assume responsibility for any claims arising therefrom made against the publisher. The publisher reserves the right to reject any advertising that is not considered in keeping with the publications mission and standards. The publisher reserves the right to place the word advertisement with copy which, in the publishers opinion, resembles editorial matter. All advertising accepted for publication in Cost Engineering is limited to subjects that directly relate to the cost management profession. Current rate card available on request. COST ENGINEERING DEADLINES: Submissions for Cost Engineering must be received at least 8 weeks in advance of the issue date. Send to: Editor, 1265 Suncrest Towne Centre Dr, Morgantown, WV 26505-1876 USA. Deadlines do not apply to technical papers.
DIRECTOR-REGION 1
Matthew Nicholas, PSP 416.498.1303 / dirregion1@aacei.org
DIRECTOR-REGION 2
Maria Cristina Baltazar, PSP 410.276.0076 / dirregion2@aacei.org
DIRECTOR-REGION 3
James H. Carson, CCC CEP 770.444.9799 / dirregion3@aacei.org
DIRECTOR-REGION 4
Logan Anjaneyulu, CCC CEP PSP 949.608.7371/ dirregion4@aacei.org
DIRECTOR-REGION 5
Earl J. Seabrook, III, CCC 713.372.0521 / dirregion5@aacei.org
DIRECTOR-REGION 6
Nicholas Kellar, CCC EVP PSP dirregion6@aacei.org
DIRECTOR-REGION 7
Gulam Farooq Quadir, CCE +966.50.3860575 / dirregion7@aacei.org
DIRECTOR-REGION 8
Keith Webb +61.8.93485045 / dirregion8@aacei.org
DIRECTOR-REGION 9
Jim Perry +33.14.7664729 / dirregion9@aacei.org
Policy Concerning Published Columns, Features, and Articles

Viewpoints expressed in columns, features, and articles published in Cost Engineering journal are solely those of the authors and do not represent an official position of AACE International. AACE International is not endorsing or sponsoring the authors work. All content is presented solely for informational purposes. Columns, features, and articles not designated as Technical Articles are not subject to the peer-review process.
EXECUTIVE DIRECTOR
Dennis G. Stork 304.296.8444 / dstork@aacei.org
Bechtel is among the most respected engineering, procurement, and construction companies in the world. We stand apart for our ability to get the job done right no matter how big, how complex, or how remote the project.
Opportunities in Project Controls include: Project Estimators Planners & Schedulers Trend Engineers Cost Engineers Project Controls Managers
CIVIL INFRASTRUCTURE COMMUNICATIONS MINING & METALS OIL, GAS & CHEMICAL POWER
Discover more at bechtel.com/careers

Bechtel is an Affirmative Action/Equal Opportunity Employer (AA/EOE).
The ambition to make history.
..........................................................................................................
Risk Analysis doesnt have to be
complicated.
Acumen Risk: Risk ReInvented
Proactively reduce risk exposure with the first easy-to-use and truly integrated cost/schedule risk analysis and risk register software tool.
Powerful analytics, consistently accurate results.

Coming September 18, 2012. Find out more:
www.projectacumen.com/risk
TECHNICAL ARTICLE
Integrated Cost-Schedule
Risk Analysis
Abstract: This article is based on AACE International Recommended Practice 57R09 that defines the integrated analysis of schedule and cost risk to estimate the appropriate level of cost and schedule contingency reserve on projects. The main contribution of this RP is to include the impact of schedule risk on cost risk and the need to hold cost contingency reserves. Additional benefits include the application of the risk drivers method of using risks to drive the Monte Carlo simulation of a resource-loaded CPM schedule. By using the risks, one can prioritize the risks to cost, some of which are commonly thought of as risks to schedule, so that risk mitigation may be conducted in an effective way. The methods presented in the RP are based on integrating the cost estimate with the project schedule by resource-loading and costing the schedules activities. Using Monte Carlo techniques one can simulate both time and cost simultaneously, permitting calculation of the impacts of schedule risk on cost risk. This article was presented as RISK.1043 at the 2012 AACE International Annual Meeting in San Antonio, Texas. It was ranked as the number three paper out of 99 presentations given and rated at the Annual Meeting. Key Words: Contingency, cost, estimates, mitigation, Monte Carlo, projects, risk, and schedule
consistent results for project schedule and cost in each iteration. Context Corporate culture is important. The organization should be risk-aware. Management sets this tone, must want to know the truth about the risks to the project, and must view the risk analysis as an important input to project success. Management must be seen to use the risk analysis results as they make key project decisions. Without this condition the analysis will fail, no matter how much sophisticated software and training the staff has had, because the risk data will not be high-quality. Traditionally, schedule risk has not been a major factor in assessments of cost risk. More recently cost risk analyses have included attempts to represent uncertainty in time, but usually these analyses occurred outside of the framework of the project schedule. Only recently have the tools been available to include a full analysis of the impact of schedule uncertainty on the uncertainty in cost. The Monte Carlo tools first calculated labor cost proportional to the duration of activities. This was not a complete assessment of cost risk because it ignored other cost-type risks that are not related to schedule such as risks affecting the time dependent resources
his article describes an improvement in cost risk analysis over the traditional methods that address cost risk without explicit reference, or any reference at all to the project schedule and its risk. It is now known how to represent the role that schedule risk has in driving project cost, because the longer some resources, such as engineering or construction work, the more they cost. One can also identify the risks that cause overall cost (and schedule) objectives to be placed in jeopardy, so one can use the results to conduct a risk mitigation
exercise to improve the projects prospects for success. The platform of this cost risk analysis is a resource-loaded project schedule. One may use a summary schedule or a detailed project schedule. The budget is assigned to the activities using summary resources to insert the entire budget into the schedule at the activity level. Monte Carlo simulation is the most commonly used approach to analyzing the impact of multiple risks on the overall project schedule or cost risk. Simulating a resource-loaded project schedule derives
include representation of all the work; has activities properly linked with logic; and, includes enough detail to highlight the main project milestones to be used.
Experience shows that schedules of 300 1,000 activities can be used in a risk analysis, even of projects as large as $10 billion. A detailed schedule may be used but it has several limitations: Table 1 Example Project Cost by Activity burn rate per day and the uncertainty in time-independent equipment or material cost. The integrated cost-schedule risk analysis has several inputs, uses specialized Monte Carlo simulation tools, and produces several valuable outputs. Inputs The Cost Estimate The cost estimate is a basic input to the risk analysis. Since the risk analysis calculates the probability of achieving the cost estimate and allows the organization to calculate the cost contingency reserve, the cost elements used as inputs need to be stated without contingency embedded in them. A good rule is to make the cost estimate, for each project element, the unbiased most likely estimate. Some estimators are uncomfortable about stripping the contingency amounts from the estimate, but the Monte Carlo simulation will reestimate the contingency reserve that is appropriate for: the risks to the specific projects cost plan; and, the desired level of certainty of the project management and other stakeholders. resources, some of which are timedependent and others are timeindependent: Time dependent resources cost more the longer they are employed, e.g., construction, detailed engineering, heavy-lift barges and drilling rigs, equipment, project management team or procurement. These resources may cost more or less even if the activity duration is fixed since the burn rate per day may be variable. Time independent resources such as procured equipment and raw materials, even subcontracts (particularly before they are awarded) may cost more or less than the engineering estimate, but not because they take longer to produce and deliver. It is usually too difficult to identify and correct a detailed schedule with many activities and logical relationships for best practices. Applying resources to activities is more difficult for a detailed schedule than for a summary schedule, even if summary resources are used. And, Simulation of the detailed schedule with risks attached is often time consuming.
In this article, the authors use a simple project as an example. It is a construction project estimated to cost $624 million over a 28-month period. The cost estimate is shown in table 1. The CPM Schedule The platform for the integrated costschedule risk analysis is a cost-loaded CPM schedule. To incorporate the schedule risk into the cost risk, the schedule has to be taken into account directly and transparently. For an integrated cost-schedule risk analysis (and for schedule risk analysis) a summary schedule needs to be: integrated;
The first task in the risk analysis of cost and schedule is to debug the schedule. The schedule needs to follow CPM scheduling recommended practices because it needs to calculate the milestone dates and critical paths correctly during Monte Carlo simulation. It should be noted that the scheduling requirements for schedule risk analysis and for high quality CPM schedules are the same. The scheduling principles that are particularly important to the success of a Monte Carlo simulation include: All work needed to complete the project must be represented in the schedule, because: (1) one does not know whether the critical path or risk critical path will be a priority, and (2) for integration of cost and schedule risk, one needs to be able to assign all the project cost to appropriate activities. The logic should not contain any open ends, called danglers. This means that: (1) each activity, except the first activity, needs a predecessor to drive its start date, and (2) each activity, except the final
The values of cost that will be assigned to the activities in the schedule are based on the resources that will be used to accomplish and manage the work and the daily rate of those resources. The cost of a project component may involve several assigned
Figure 1 Example Construction Project Schedule delivery, needs a relationship from its finish date to a successor. The schedule should not rely on constraints to force activities to start or finish by certain dates. It should use logic for this purpose. Date constraints can turn a CPM network into a calendar. Lags and leads are appropriate only in limited circumstances and are generally to be avoided in project scheduling. And, The schedule used should be the statused current schedule. Simple Construction Case Study A simple schedule example of a 28month construction project is shown in figure 1. This figure and several others shown in this article are screen shots from Primavera Risk Analysis, formerly Pertmaster Risk Expert, now owned by Oracle. Resources Loaded into the CPM Schedule Loading resources into the CPM schedule for the purpose of integrated cost-schedule risk analysis can be accomplished using summary resources. Summary resources are not sufficiently detailed to perform resource leveling. Their purpose is to get the entire budget into the project schedule. Simple categories of resources that can be given budgeted values and placed on the activities they work on are needed. Resources used on the simple construction project are shown in table 2. In addition, the resources need to be tagged as labor-typetimedependent resources; or materialtypetime-independent resources, as mentioned above. An alternative method of applying resources to the schedule, when the cost estimate is not specified down to the detail of the schedule activities, is to apply resources to hammocks that span the activities that get the resources. Resources Resources are applied to the schedule activities, Sometimes, in doing this, the cost estimate and schedule have evolved largely independently of one another and the cost estimates are not consistent with the activity durations. It is important that if the estimate and schedule are initially developed independently of one another, that they are reconciled prior to holding the risk assessment, or else the implied daily rates of cost will be wrong. The costs that result from placing the resources on the example project schedule are shown in table 3. THE RISK DRIVER METHOD Risk Data Inputs for the Risk Driver Method Applying first principles requires that the risk to the project cost and schedule is clearly and directly driven by identified and quantified risks. In this approach, the risks from the risk register drive the simulation. The risk driver method differs from older, more traditional approaches wherein the activity durations and costs are given a 3-point estimate which results from the influence of potentially
These points are consistent with those found in GAO Cost Estimating and Assessment Guide [1]. It is good scheduling practice to review the total float values to make sure they are reasonable. Large float values may indicate incomplete logic; and perhaps, the need to introduce and logically link additional activities.
Table 2 Resources for Example Construction Project
Table 3 Cost Example Construction Project Showing Resources several risks, which cannot be individually distinguished and kept track of. Also, since some risks will affect several activities, one cannot capture the entire influence of a risk using traditional 3-point estimates of impact on specific activities. Using the risk driver method, the risks that are chosen for the analysis are generally those that are assessed to be high and perhaps moderate risks from the risk register. Risks are usually strategic risks rather than detailed, technical risks. While the risk data are collected in interviews with project SMEs, new risks emerge and are analyzed. There may be perhaps 20-40 risks, even in the analysis of very large and complex projects. Risks to project schedule and cost include: risk events that may or may not happen; and, uncertainties that will happen but with uncertain impact. The risk also has an impact on project activities if it does occur. This impact is specified as a range of possible impacts, stated in multiples of the activitys estimated duration and cost. If the risk does occur, the durations and costs of the activities in the schedule that the risk is assigned to will be multiplied by the multiplicative impact factor that is chosen from the impact range for that iteration. The risks are then assigned to the activities and resources they affect.
Once the risks are identified from the risk register, certain risks data are collected: Probability of occurring with some measurable impact on activity durations and/or costs. In any iteration during the Monte Carlo simulation, a risk will occur or not depending on its probability.
Collection of risk data relies on the processes of the risk identification and risk prioritization. It is important during
Table 4 Example Risks and Their Parameters For The Case Stude
risk data collection to be alert to possible biases that crop up during workshops. Some people want to influence the results, while others genuinely do not understand the concepts or have some cognitive bias that has to be overcome. One approach is to conduct risk interviews with individuals, or small groups, for which there is a promise of confidentiality to the participants. This is so they can talk honestly and openly without fear that
management will be displeased with them. The degree of correlation between the activity durations has long been viewed as being important for understanding and estimating correctly the project cost risk analysis. Correlation arises if one risk affects two (or more) activities durations, or if a risk affects the cost of two time-independent resources. If a risk occurs, the degree to which their durations are longer and
shorter together is called correlation. As shown below, the risk driver method causes correlation between activity durations, so one no longer has to estimate (guess) at the correlation coefficient between each pair of activities. Probabilistic branching, or existence risk, requires another type of risk data the probability that an activity and its cost will exist on this project. Some risks may cause activities to occur only if the
Table 5 Assigning Risks to Activities
Figure 2 Histogram with Cumulative Distribution (S-Curve) for the Project Completion Date risk occurs. Some risk events such as failure of a test or a commissioning activity, if they occur, may require new activities such as finding the root cause of the failure, determining the recovery plan, executing the recovery plan, and retesting the article. These activities will all take time and increase project cost. They can be inserted in the schedule as probabilistic branches, or existence activities, with time and cost implications if they occur. Simulation Using the Risk Drivers Method In the simple example used in this article, the risks impacts are specified as ranges of multiplicative factors that are then applied to the duration or cost of the activities to which the risk is assigned. The risks operate on the cost and schedule as follows: A risk has a probability of occurring on the project. If that probability is 100%, then the risk occurs in every iteration. If the probability is less than 100%, it will occur in that percentage of iterations. The risks impacts are specified by 3point estimates of multiplicative factors, so a schedule risk will multiply the scheduled duration of the activity to which it is assigned. The 3-point estimate, for instance, (low 90%, most likely 105% and high 120%), is converted to a triangular
distribution. For any iteration, the software selects an impact multiplicative factor at random from the distribution. If the risk occurs during that iteration, the multiplicative factor selected multiplies the duration of all the activities to which the risk is assigned. The cost risk factor is applied differently depending on whether the resource is labor-type or equipment-type. o For a labor-type resource, the cost risk factor varies the daily burn rate, representing more or fewer resources applied, higher or lower cost of those resources per day. For these resources, their total cost is also affected by the uncertainty in the duration, but they may cost more or less even if their durations are as scheduled. o For equipment-type resources, the cost risk factor varies the total cost. For these resources, the cost may be uncertain but it is not affected by time.
Simulation Tools Monte Carlo simulation is the most commonly applied method for conducting quantitative risk analysis. A Monte Carlo simulation calculates the possible project cost and schedule values that may result from individual risks and translates them into projectlevel cost and schedule histograms or distributions from which statistical statements can be made. Since one does not know whether any risk will occur on any specific project
Table 6 Summary Schedule Risk Analysis Results for the Example Construction Project
10
Risk Data Used for the Construction Case Study The following is a sample case study, but the risks are similar to those found on real projects. Suppose there is a project with the activities shown in figure 1, and resources/costs as shown in table 1, and assigned to the activities as shown in table 3. Also, suppose one has identified risks through a workshop or interviews and have elicited the probability and time/cost impacts as shown in table 4. After the risks are listed and their parameters quantified, they need to be assigned to the activities and their resources. For this case study the risks are assigned according to table 5. Results From the Construction Case Study Simulation The schedule risk results from a Monte Carlo simulation are shown in the histogram for the case study in figure 2. It shows that the deterministic date of 29 April 2013, is about 4% likely to be achieved following the current plan and without further risk mitigation actions. Next, suppose that the project stakeholders have agreed that their acceptable level of confidence is at the 80th percentile. At that point, it is 80% likely that the current project plan with all of its risks will finish on that date or earlier (and, if it is applied to cost, with
Figure 3 Histogram with Cumulative Distribution (S-Curve) for the Project Cost or what its impact will be, one cannot tell when a project will finish or how much it will cost. One can only tell probabilistically when the project might finish and how much it might cost. Suppose the simulation contains 3,000 iterationsseparate runs using randomly-selected risk dataand creates 3,000 pseudo-projects. Each of the 3,000 projects could be the sample project in this article, since it is based on a different combination of risks applied to this project schedule and cost. These different combinations of input data generally compute different completion dates and project costs. The Monte Carlo simulation provides probability distributions of cost and schedule from which one can make probabilistic statements about this project.
Table 7 Summary Cost Risk Analysis Results for the Example Construction Project
Table 8 Cost, Schedule and Interaction Effects
11
that cost or less). At the P-80, the project finishes on 26 November 2013, or earlier, and needs about a 7-month contingency reserve of time. These results are shown in figure 2 and in table 6. The cost risk results, including the impact on cost of schedule risk, indicate the need for a contingency reserve of cost of about $169 million, or 27% at the 80th percentile (P-80). At that level there is an 80 percent probability that the project will cost $793 million or less, given the risks and following the current plan. These results are shown in figure 3 and table 7. One can find out whether cost-type risks or schedule-type risks are more important in determining the cost contingency to, say the P-80 point. The source of the cost contingency can be discovered by eliminating all schedule risks, so as to compute the marginal impact of cost risks, then repeating the process by eliminating the cost risks and computing the impact of schedule risks on contingency. The results are shown in table 8. Table 8 shows that if only cost risks were present (the schedule is static) the cost contingency at the P-80 could be $78 million, whereas if only schedule risks were included (no cost risk on burn rate or on procurement/materials), the contingency needed at the P-80 is $103. These results depend on the case study assumptions, but in many examples of integrated cost and schedule risk conducted on projects, the majority of the risk to cost arises from uncertainty in the schedule as it does in the example in this article.
Figure 4 Cost and Time Results From the Simulation Correlation Between Cost and Schedule The time-cost scatter diagram, shown in figure 4, is diffuse because there are some time-independent cost risks that affect the burn rate of labortype resources and total cost of procured items. The cross-hairs shown on the diagram cross at the deterministic point of 29 April 2013 and $624.2 million. The sparse collection of points in the lowerleft quadrant indicate that there is only a 1% chance that this project will satisfy both cost and schedule targets without contingency reserve. There is also a 95% chance that this project, following this plan, will overrun both cost and time objectives. There is clearly a positive slope running through the cloud or football (US version) chart, showing the strong impact on cost of schedule risks. The correlation between time and cost is 77% in this case study, which is somewhat higher than is common in these analyses. Probabilistic Branches or ProjectBusting Risks Some risks will add activities to the project schedule if they occur, and hence will add time and cost. Most often a project plan assumes that the project goes well and that there are no major problems. It is also common that something goes wrong leading to a mandatory change in plans as the project tries to recover from a discontinuous event. An example of this problem might be the failure of the project at commissioning or final testing. These activities might be: Determine the root cause of the failure; Decide what to do; Implement the action; and,
Figure 5 Activities Added to Provide for a Risk That Commissioning May Not Complete Successfully
12
specification that the commissioning will fail 40% of the time. The cost of the project goes up at the P-80, since resources are placed on the activities in the probabilistic branch. The impact on cost and schedule of a 40% probable problem during commissioning, (with the parameters shown here), are shown in table 9. Prioritized Risks to Schedule and Cost If the risk results for the overall schedule and cost are not acceptable to the customer, the analyst can prioritize the risks for the project manager who will want to mitigate the highest-priority risks. For schedule risk, one needs to identify the most important risk by taking each risk out entirely (make the probability = 0%) one-at-a-time and re-run the simulation to determine the P-80 date, allowing one to identify the risk that has the greatest marginal impact on the P80 date. Then, keeping the most important risk out, one explores the remaining risks to see which of those is next-most-important, and so forth. And, For cost risk, this is done by taking each risk out of the project one at a time, computing the impact to the P-80 cost compared to the all-in results, and finding the risk that has the largest impact on the P-80 cost. It is logical to identify the schedule risks that have cost risk implications as described above, but the list of the risks in order of priority may differ for time and for cost.
Figure 6 Schedule Impact of Probabilistic Branch on Commissioning Re-test and, hopefully, pass the test this time. activities, since when one introduces them they are given a duration of zero (0) days. Their uncertainty is represented by traditional 3-point estimates, but their risk source is known. An implementation of probabilistic branching is shown in figure 5. The probability that commissioning will not complete successfully the first time is estimated as 40%. The schedule results for adding a probabilistic branch are shown in figure 6. Notice that the schedule is slightly bimodal, with 60% of the results in the left-hand part of the distribution and 40% in the right-hand part. There is a bit of a shoulder in the cumulative distribution at 40%, that follows the
One common characteristic of these activities is that they are almost never found in the initial project schedule, which assumes success. However, in risk analysis, the possibility of test failure or some other discontinuous uncertain event, must be modeled using existence risks or probabilistic branching. Suppose that the commissioning activity might uncover a problem that takes time to fix. Simple changes can be made in the project schedule to accommodate this potentially projectbusting occurrence. One cannot use the risk drivers that are assigned to existing
Table 9 Schedule and Cost Impact of a 40% Probable Commissioning Risk
13
Table 10 Highest Priority Risks to Project Schedule at the P-80 Level of Confidence Tables 10 and 11 show which risks are the most important for schedule and for cost. Risk Mitigation Using Prioritized Risks Using the prioritized risks in table 8, one can recommend risk mitigation. The first thing to recognize is the inaccuracy of the estimates, which is viewed as moderate at risk impact multipliers of 95%, 105% and 115%. However, this risk is 100 % likely to occur, since estimating error is with one until project financial completion, and it is assigned to each activity in the project, hence its importance. The next item to be concerned about is the probability of problems during commissioning, which is also the highest schedule risk. The next largest item would be the unavailability of key engineering staff. Down the list at position five is the inaccuracy of the schedule. In fact, in the simple example the authors created for this article, only the top risk to project cost is a pure cost risk. The other important risks are mostly schedule risks (some with cost risk components, see table 4) that increase cost if their activities are longer than assumed in the schedule. These schedule risks may be missed or underestimated if the cost risk analysis does not explicitly handle the relationship of time and cost risk, as is shown in the approach described in this article. It is common to find that schedule risks are important in driving cost risk. It reinforces the benefits of integration of cost and schedule. of-effort resources, such as the project management team or QA/QC, are extreme examples of time-dependent resources, since if the project duration exceeds its planned duration the cost of these resources will increase over their budgeted amount. The integrated cost-schedule risk analysis is based on: A high quality CPM schedule. A contingency-free estimate of project costs that is loaded on the activities of the schedule using resources distinguishing them by their time-dependent and timeindependent nature. Good-quality risk data that is usually collected in risk interviews of the project team, management and others knowledgeable in the risk of the project. The risks from the risk register are used as the basis of the risk data in the risk driver method. The Risk Driver Method is based in the fundamental principle that identifiable risks drive overall cost and schedule risk and that one can model this process. And, A Monte Carlo simulation software program that can simulate schedule
Conclusion
Integrating cost and schedule risk into one analysis, based on the project schedule loaded with costed resources from the cost estimate provides both: more accurate cost estimates than if the schedule risk were ignored or incorporated only partially; and, illustrates the importance of schedule risk to cost risk when the durations of activities using labortype (time-dependent) resources are risky.
Many activities such as detailed engineering, construction or software development are mainly conducted by people who need to be paid even if their work takes longer than scheduled. Level-
14
Table 11 Highest Priority Risks to Project Cost at the P-80 Level of Confidence risk, burn-rate risk and timeindependent resource risk. The results include the standard histograms and cumulative distributions of possible cost and time results for the project. However, by simulating both cost and time simultaneously, one can collect the cost-time pairs of results and hence show the scatter diagram (football chart) that indicates the joint probability of finishing on time and on budget. Also, one can derive the probabilistic cash flow for comparison with the time-phased project budget. The risks to schedule completion and to cost can be prioritized, say at the P-80 level of confidence, to help focus the risk mitigation efforts. If the cost and schedule estimates including contingency reserves are not acceptable to the project stakeholders, the project team should conduct risk mitigation workshops and studies, deciding which risk mitigation actions to take, and re-run the Monte Carlo simulation to determine the possible improvements to the projects objectives.
REFERENCE
1. GAO Cost Estimating and Assessment Guide, US Government Accountability Office, March 2009 (GAO-09-3SP), pgs. 218-224.
ABOUT THE AUTHORS
RECOMMENDED READING
1 AACE International Recommended Practice No. 57R-09 Integrated Cost and Schedule Risk Analysis Using Monte Carlo Simulation of a CPM Model, AACE International, Morgantown, WV, 2011. Hulett, D.T., Practical Schedule Risk Analysis, Gower Publishing Limited, Farnham Surrey England. (2009): pg. 218. Hulett, D.G., Integrated CostSchedule Risk Analysis, Gower Publishing Limited, Farnham Surrey England. (2011): pg. 211.
Dr. David T. Hulett is with Hulett & Associates, LLC. He can be contacted by sending e-mail to: david.hulett@projectrisk.com
Michael R. Nosbisch, CCC PSP, and Past President of AACE International is with Project Time & Cost. He can be contacted by sending e-mail to: mike.nosbisch@ptcinc.com
15
Now: If schedules needed analysis, What is the difference between Primavera would have included it.
reviewing your P6 schedule and reviewing your P6 XER file? Everything
See more at
http://ScheduleAnalyzer.com
% % % % %
Interfaces Directly with Database Hundreds of Reports & Analyses Advanced Longest Path Analysis Built-in Report & Editing Facility Drill-down from Dashboard to P6
16
TECHNICAL ARTICLE
Estimate Accuracy:
Dealing With Reality

known to be a disaster when systemic risks are present [27]. One researcher said this behavior verges on criminal [19]. Cost disasters and criminality are economic in nature, but deadly serious to owners, investors and tax-payers; we must ultimately take responsibility for our role in their economic well-being. To help improve on the situation, this article surveys the research facts (reality), exposes flawed practices. and highlights better practices. The article summarizes data from well referenced studies by others; however, the data confirms the authors experience. The authors data and observations are added, as well as Key Words: Cost estimate accuracy, recommended practices, and risk analysis observations by others. While fact and opinion are mixed, it is hoped that readers will draw the same conclusions as ccuracy is a measure of how a and how well risk analysis methods the article and work to improve the cost estimate will differ from forecast it is generally poor. situation. Investment decision makers seem the final actual outcome. It is also a measure of cost particularly unaware of our research and Studies of Overall Estimate Accuracy uncertainty or risk (these terms are recommended practices. Sometimes they How accurate have cost estimates essentially synonymous in Total Cost are aware, but seem to ignore them. been for owners? To answer this, Worse, many cost engineers facilitate Management (TCM). references providing empirical data on Empirical estimate accuracy data has management ignorance by standardizing estimate accuracy and cost uncertainty been researched for over 50 years [30]. In their wishful thinking (i.e., tunneling or were sought. This article focuses on addition, reasonably reliable practices for neglecting sources of uncertainty) as engineering and construction projects in quantifying project cost uncertainty have exemplified by bias towards 10% the process (e.g., oil, gas, chemicals, been recommended by AACE and others. contingency and +10/-10% range [42]. mining, metals, utilities, etc.) and However, the level of industry Poor investment decisions may result infrastructure (often associated with understanding of the reality of accuracy from using risk analysis methods that are process plant projects) industries. These Abstract: This article reviews over 50 years of empirical cost estimate accuracy research and compares this reality to common but unrealistic management expectations. The empirically-based accuracy research of John Hackney, Edward Merrow, Bent Flyvbjerg, and others on large projects in the process industries is summarized. The article then highlights risk analysis methods documented in recent AACE Recommended Practices that yield outputs based upon and comparable to empirical reality. Tragically, many cost engineers are facilitating managements collective and sometimes willful biases regarding accuracy by using flawed, unreliable risk analysis methods; those who use empirically valid practices face the fate of Cassandra. The article is intended as a fundamental reference on the topic of accuracy, as well as a call for our profession to use reliable practices and speak the truth to management. Readers will gain an understanding of estimate accuracy reality, the risks that drive it, managements biases about it, and methods that analyze risks and address the biases in a way that results in more realistic accuracy forecasts, better contingency estimates and more profitable investments. This article was first presented as RISK.1027 at the 2012 AACE International Annual Meeting in San Antonio, Texas.
17
are generally characterized by complexity, unique work scopes, design change and sometimes new technology. The chosen references represent academic, research, consulting and industry practitioner sources. Empirical research on defense, aerospace and IT projects was found but excluded; their experience is analogous but more extreme [13,16, and 20]. Estimate accuracy and cost uncertainty data from 12 empirical studies are summarized in table 1. These include over 1,000 projects with samples ranging from about 20 to 250 projects each. The projects were typically large enough to affect enterprise success (i.e., typically one million US dollars up to megaprojects). The costs studied are the costs to the owners. Study purposes varied; however, the typical the questions were: what is the accuracy of our estimates and why? in reaction to a perceived preponderance of cost overruns. In summary, the approximate range of ranges for accuracy or uncertainty around the reference amounts are as follows: P10: -32% to +8% (average about -9%) P50 or mean: 0% to +88% (average about 21%) P90: +34% to 190% (average about 70%)
The accuracy shown is the percentage variation of the final actual cost from the reference estimate. The reference estimate was usually the basis for an actual or defacto investment decision by the owner. Estimate names are industry specific; for example, feasibility is the funding estimate for mining projects, but not for projects in other industries. The reference estimates usually include contingency; therefore, the accuracies are understated in respect to base estimates without contingency. From the authors experience, the contingency applied at sanction is usually between 5 and 15 percent. The statistics provided ranged from mean and standard deviations alone to distribution charts or tables or values at
various confidence levels. For comparison, the accuracies in table 1 are summarized at approximate p10/p50/p90 confidence levels where the p value indicates the percentage that underran. If a mean was provided (), it is shown as such. If p-values were not provided, they were approximated from the mean and standard deviation assuming a normal distribution; i.e., p90 equals the mean plus 1.28 times standard deviation (the ~ symbol indicates an approximation.) This approximation underestimates the high range when actual/estimate accuracy data is skewed to the high side (i.e., actual data is not normally distributed). The project samples were not scientifically random, but were not selected specifically because their estimates were inaccurate; the authors generally considered the projects in their samples to be reasonably representative. Studies done in reaction to overruns may be biased toward that experience; however, the number of studies and the variety of industries, regions and project types covered indicate that cost overruns are prevalent for large process industry projects. The quality of the datasets varied, but in general the authors lament the poor state of historical project records. For many studies, the only reliable data was the cost at the time of project funding approval (i.e., sanction or investment decision) and the cost at completion. However, some studies were corrected for major scope changes and escalation, which many practitioners would not expect an estimate to cover. The key observation is that in no case was the nominal p90 value ever less than +34% of the funding estimate (i.e., about +40 to +50% of the base estimate). Also, the average mean or median overrun is about 21%. This is the best picture we have of reality for large process industry projects with all their imperfections and risks (unfortunately, causal data is lacking). Arguably, the most notable studies are by John Hackney and Edward Merrow because these are the foundation for process industry phasegate project systems [22, 35]. However, the studies by Dr. Bent Flyvbjerg are perhaps best known in the popular press
[19]. Dr. Flyvbjerg has made the following statements regarding industry estimating practices: We conclude that the cost estimates used in public debates, media coverage, and decision making for transportation infrastructure development are highly, systematically, and significantly deceptive. (those) who value honest numbers should not trust the cost estimates presented by infrastructure promoters and forecasters. He adds, institutional checks and balancesincluding financial, professional, or even criminal penalties for consistent or foreseeable estimation errorsshould be developed to ensure the production of less deceptive cost estimates [19]. Merrow disagrees with Flyvbjerg in the following: There is widely held belief that large public sector projects tend to overrun because the estimates are deliberately low-balled. Our (IPAs) analysis of large private sector projects suggests that no Machiavellian explanation is required. Large projects have a dismal track record because we have not adjusted our practices to fit the difficulty that the projects present [33]. Regardless of motives and causes, large process and infrastructure projects (and defense, aerospace and IT) are frequently overrunning our funding estimates and by very large margins. The search found no research that showed otherwise. Further, as forecasters (as we are referred to by Flyvbjerg) we are failing to reliably predict the proper point of funding including contingency, but the range of project cost uncertainty. Studies of Estimate Accuracy Progression Versus Level of Scope Definition It is generally agreed that the less well defined the project scope is, the wider the estimate accuracy range will be. This is a premise of phase-gate project systems. Table 2 summarizes accuracy studies from among this articles sample that also addressed accuracy and uncertainty at various levels of scope definition, approximated to AACE classifications (Class 5 to 1). Complicating the comparison, each study has different data attributes and uses different scope definition rating schemes (i.e., AACE classification ratings
18
Table 1 Empirical Estimate Accuracy Studies (Typically From the Funding Estimate) were not used). However, the authors experience is that process industry funding decisions are being made based on scope definition somewhat better than AACE Class 4, but worse than Class 3. Research indicates that the design development necessary to thoroughly mitigate definitional risk includes issuedfor-design, signed-off process and instrumentation diagrams (P&IDs) for all process and utility units; the author rarely sees this level of definition at the time of funds authorization [4]. The key observation from table 2 is that even projects funded on better scope definition (AACE Class 3) tend to be overrun; there is a huge potential for overruns if the scope is more poorly defined than Class 3. Note that the Hackney and RAND models based on this data are available in working Excel tools found at the AACE website (www.aacei.org [11]).
19
What Our Estimates Say and What Owners Want Are the Same (i.e., Wishful Thinking) The next question is, are the overrunning projects within the cost range of our risk analyses? Unfortunately, they usually are not. The author has reviewed many industry risk analyses by owner companies and their EPC contractors and their p90 forecast is rarely great than 30% over the base estimate excluding contingency. Table 3 provides an indicative sample of risk analysis outcomes. The first project risk analysis shown in table 3 had a p90 value of +45%; however, the risks on this project were extreme and while the team captured some of them, the range was overrun by the next phase estimate. P50 values are often as little as 5% even for highly risky projects. The authors experience is that despite extensive risk registers and brainstorming sessions, most risk quantification is dominated by an estimators bias in which the team consciously or unconsciously perceives uncertainty in terms of estimate and takeoff assumptions and math (i.e., estimators risks). A high (p90) range of about +30% reflects the perceived worst case
uncertainty around quantities, rates, pricing, and productivity; while unrealistically assuming the scope is fixed, the execution strategy and plan is never changed, no risk events occur and if they do, risk responses are always effective. The result is a range that seems to be what the owner wants to hear. So the next question is, what does the owner want to hear? Table 4 provides an indicative sample from different industry segments of owner accuracy range expectations as stated in their phase-gate project scope development processes. The table compares the owner targets to the range-of-ranges in the AACE Recommended Practice 18R-97 [4]. Is it coincidence that the owner p90 targets in table 4 are about the same as the p90 values estimated in table 3? By quoting specific accuracy range targets in their processes, owners display a dangerous misunderstanding of risk and estimating. Once a project plan reaches the target level of scope definition (e.g., Class 3), the residual risk and its potential impact is a project scope attribute and no estimator can appreciably improve the accuracy range by doing a better estimate.
For a project with substantial risks (most large projects), the company accuracy ranges in table 4 have no relevance. Unfortunately, targets tend to pre-determine risk analysis outcomes; i.e., they drive the risk analysis outcomes seen in table 3 (owners get what they ask for). Targets are prima facie evidence of risk ignorance (tunneling) driven by the inflated expectations that phase-gate processes alone will manage risks. Further, the communication of targets by many owners is statistically meaningless. First, many misquote AACE Recommended Practices by stating that the targets are per AACE when no AACE document includes specific targets [4]. Also, few state the confidence interval represented or the reference value that the range is around (the base or the funded amount?). To wrap up the target/as-estimated versus actual accuracy discussion, figure 1 shows the averages of table 1 (Reality) and table 3 (As Estimated) as log-normal curves with p10/50/90 values comparable to the table 1 and 3 averages for those confidence levels. Several of the studies and the authors experience suggest that the lognormal distribution of
Table 2 Empirical Estimate Accuracy Studies (Progression by Level of Scope Definition)
20
actual/estimate data is representative [14,35]. Teams are assuming a p10/p90 accuracy range around the base estimate of about -10/+30% with a p50 value of about 10%, while the reality is closer to -20/+120% with a p50 of about 20%. Arguably, a 20% or even 30% overrun will not render most projects unprofitable; however, a 120% overrun at p90 would. Challenging the Data : Nowhere to Hide The following are likely challenges to this articles findings along with the authors responses: 1. The actual data includes the impact of major scope changes and escalation. Major scope change and escalation are by definition excluded from contingency [3]. In table 1, 7 of the 12 studies corrected for price changes over time and 3 corrected for major scope changes (i.e., changes to basic project premises such as plant location, product specifications or capacity).
Studies show wide accuracy ranges with or without correction. In the authors benchmarking experience, major owner scope change (as opposed to design changes which owner costs must cover) is uncommon (this may be less true for public projects). 2. We cannot forecast volatility and/or black swan events (i.e., unknown unknowns).
the average overrun was about 71% of the original project cost estimate[37]. Another mining article referenced a series of studies which indicate that overruns have been the norm in every time period since 1965 [38]: A study of 18 mining projects, covering the period 1965 to 1981, showed an average cost overrun of 33 percent, compared to feasibility study estimates. A study of 60 mining projects, covering the period from 1980 to 2001, showed average cost overruns of 22 percent, with almost half of the projects reporting overruns of more than 20 percent. A review of 16 mining projects, carried out in the 1990s, showed an average cost overrun of 25 percent.
While any one black swan event is improbable, the probability of any black swan or an equivalent confluence or compounding of lesser risks occurring during the extended duration of large projects is likely. The accuracy findings appear to hold for all time periods and regions, in both hot and cold economies. For example, Ernst & Young found that mining projects estimated during hot markets (when market risks were known) were still overrunning during the post-2008 recession; Of the companies that reported project overruns publically (between Oct 2010 and March 2011),
Historical experience alone is enough to quantify the probability and impact unknown-unknowns as a class. We may not know the risks name, but we know about what it will cost (i.e., Table 1.)
Table 3 Reported Accuracy Ranges from Owner and EPC Contractor Risk Analyses
21
Table 4 Owner Phase-Gate Target Accuracy Ranges Vs. AACE Classes and Empirical Studies
Figure 1 As Estimated (and Target) Accuracy Vs. Empirical Accuracy at Funding 3. Some systemic risks are difficult to measure and/or politically sensitive. established [4, 21, 22, 35, and 45]. While including, incompetent management, in a risk register is problematic, it is necessary to identify and quantify such risks. The risk analyst must have sufficient independence to do so. 4. Estimating all project cost risk is not part of the job (not in my work scope).
The tools for rating scope development, as well as competency and project system discipline (e.g., weak change management) are well
If one declared in the Basis of Estimate reports that, most significant risks were excluded and/or past
22
experience with similar projects was ignored, this challenge might have some validity. However, in the authors experience, such statements (or confessions) are rarely made. Unfortunately, breaking risk down (e.g., operational, project, strategic, enterprise, contextual, global, background, etc. [42]) and disseminating responsibility for its analysis and quantification is a potential recipe for forecasting failure. Risks interact and often compound and cannot readily be parsed for quantification like elements in a work breakdown. In summary, it is the authors experience that these challenges are usually just reasons for our failings; they do not excuse them. We know better and the data is clear; with empirical insight added to other methods, risk is always quantifiable albeit imperfectly.
Flawed Practices and Lost Credibility Flawed practices such as a bias toward estimators risk, misguided targets, tunneling and parsing risk quantification have been mentioned. The 1990s also brought reengineering and downsizing to the industry with the loss of empirical data and analytical skills. Concurrently, Monte Carlo simulation (MCS) for spreadsheets was introduced which made risk analysis seem simple and doable regardless of skill level. Unfortunately, MCS was applied in line-item ranging (as opposed to range
estimating) in which the team assigns cost ranges to line-items in their estimate (i.e., contributing to estimators bias) based on brainstorming, and then runs the MCS, usually without considering line-item dependency [24]. The risks listed in the register (which tend to exclude systemic risks) are not explicitly included in these models. This is the method that research has shown to be a disaster for projects with systemic risks [27]. Line-item ranging (or activity duration ranging for schedule) fails in part because of faulty application (i.e., no dependencies), but also because brainstorming is unable to elicit the impacts of systemic risks on individual estimate line items or activities, and finally, the impact of risk register events are difficult to ascribe to individual estimate line items in aggregate. This method is not an AACE Recommended Practice. It is easy to conclude from the research and observations that our risk analyses and contingency estimates are not credible for large process industry projects. Decision analysis expert John Schulyer defines a credible analysis as, one that gets used [44]. The following statements by industry executives indicate that our analyses are not useful (self-criticism by owner executives is understandably more difficult to find): Schlumberger CEO Andrew Gould stated: ...while not wishing to embarrass any of my customers, I would add that many greenfield
(upstream oil) projects suffer significant cost overruns. Indeed, as a general rule, 30 percent of such projects experience budget overruns of 50 percent [41]. Financier Jasper Bertisen of Resource Capital Funds (RCF) had this observation: the vast majority of mining projects have been coming in way over budget for the past couple of decades. As a result, RCF now automatically factors in an average cost overrun of 25% when it considers the cost of mining projects [28].
The prevailing use of flawed analyses has damaged our collective credibility. This will be difficult to remedy because poor practices have become institutionalized. For example, in the mining industry, the author commonly finds companies funding projects at a p80 level of confidence. This has evolved because (as indicated by prior quotation) managers intuitively understand that the p50 values we provide in our estimates are too low (i.e., often <10% contingency on even the riskiest projects) and they believe that the p80 level of about 15 to 20% contingency is more realistic. However, it is more realistic because in fact this forecast p80 is the p50 of the reality that we fail to predict! Cost engineers who do use realistic risk quantification practices are treated like Cassandra; management will not believe the truth after being fed
Figure 2 Actual Cost/Appropriation Estimate For Small Projects
23
Figure 3 Balancing Of Over And Underruns When All Project Sizes Are Studied Together unreality for decades. The real p80 or p90 is likely to be unprofitable; as shown in studies, the least p90 capital cost growth is >40 to 50%. If management faced this reality, no project would ever be authorized without stellar scope definition and optimization, top-notch planning, team building, risk management, and all of the other best practices we know of. Isnt that the point? Why are we facilitating anything less? Why do we let them assume that poor practices are a safe bet when they are courting disaster! The lesson from the empirical history (table 1) and the practice history (table 2) is that we need to address the entire scope of risks (project-specific, systemic, and escalation) and the empirical reality of uncertainty on large process industry projects. Research by others points in the same direction [16, 17, 18, 19, 22, 27, 32, 33, 42]. AACE is currently developing a Decision and Risk Management Professional (DRMP) Certification that will focus on risk identification and quantification competencies, including AACE Recommended Practices that document reliable methods. The Project Size Dichotomy There is less empirical research of small project estimate accuracy because these projects are individually less of a threat to overall profitability and shareholders perceptions. However, we know that the realities of small and large projects differ; small projects are biased to overestimating and underruns. As stated by one researcher, when a project team sets a soft (cost) target, about half of the unneeded funds are usually spentabout 70% of small projects underrun [29]. This research also indicated that in small project systems, overruns tend to be punished. To avoid punishment (in less disciplined cultures) teams avoid overruns by including fat (i.e., abovethe-line contingency) in the base estimate because high visibility contingency is often poorly received by management for any project size. This can bias a companys perception of risk and partly explain their misguided targets. Few researchers study small projects because not only is record keeping lax, but underruns are rewarded and are not seen as a problem despite being associated with wasteful capital spending. Figure 2 shows a representative distribution of actual/estimate values observed by the author for small project portfolios; often, no projects overrun by more than 10%. In this cresting wave pattern, most projects spend all their funds, while some return all or some of the excess; for this outcome, management and/or teams are rewarded. The more that funds are wasted, the sharper the peak between 0.9 and 1.1. Perversely, the more accurate the outcome, the less desirable (though best rewarded) it is; underruns and tight accuracy often indicate overfunding and wasted capital rather than excellent project control discipline The small vs. large project overrun dichotomy can induce a kind of corporate schizophrenia. Many owner companies have major project organizations that are separate from small or plant-based organizations. A newly formed major project group will often inherit the small project system trait of risk-ignorance (expectation of underruns.) They do not appreciate that EPC contractors for major projects prepare base estimates with less fat because reviews expose fat and there is sometimes a bias to keep estimates low to see the project get funded. The combination of small project target-
24
reinforced tunneling and risk-free base estimates is a recipe for overruns on large projects. Looking at an entire company project portfolio, the combined distribution of accuracy data for small and large projects can look serenely normal. Benchmarking data observed by the author indicates that in a population of all project sizes, the P10/P90 range is about +/-20% around the funding estimate, as illustrated in figure 3. Given just one distribution, management will be unaware that there are two conflicting realities. The Measurement Dilemma Unfortunately, accuracy (i.e., actual/estimated cost) is often misused as a measure of estimate quality (as in a high quality estimate is an accurate estimate) or estimating performance. This is inappropriate because, as discussed, the only way for an estimator to deliver a targeted accuracy for a given scope is to over-estimate the cost; risk and project performance are not in the estimators control. Faced with overruns, estimators and the team tend to hide behind the excuses discussed previously. Accuracy should be used to measure the performance of the risk management process (not the estimating process) in conjunction with project historical data including causal information so we can improve our risk identification, analysis and quantification, and treatment. Tight accuracy may indicate wasted capital funds; accuracy measures must always be accompanied by measures of project control process discipline and project cost competitiveness (lower absolute costs) or cost bias. AACE Recommended Practices (RPs) There is an AACE RP that guides the selection and development of risk quantification and contingency estimating methods [2]. This RP provides principles that any method should align with including; start with identifying risk drivers; link risk drivers and cost/schedule outcomes; and employ empiricism.
Note that the previously discussed, line-item ranging method, is not explicitly in accordance with any of above principles. Risks differ in how they impact project costs and therefore methods vary in how the risks are quantified. To cover the whole scope of risks, AACE has defined a risk breakdown [24] in respect to quantification methods that includes: Project-Specific Risk: risk affecting the specific project and plan; Systemic Risk: artifacts or inherent attributes of the system, enterprise or strategy; and, Escalation Risk: driven by economics (which regionally may involve politics). o o
Parametric Estimating Example Models as Applied for the Process Industries [11]. Escalation Risk: 58R-10: Escalation Principles and Methods Using Indices [6]. 68R-11: Escalation Estimating Using Indices and Monte Carlo Simulation [5].
Analogies for these risks suggested by others include: operational (project), strategic (enterprise), and contextual (global) risks respectively [42]. Methods that address these risk types can be integrated to generate a universal cost risk profile to support decision making. AACE also recommends that cost and schedule risk analysis be integrated. For each risk type, there are AACE RPs for risk analysis methods that apply as follows (how-to descriptions for these methods are covered in the references): o Project-Specific Risk: 41R-08: Risk Analysis and Contingency Determination Using Range Estimating [12]. 44R-08: Risk Analysis and Contingency Determination Using Expected Value [9]. 57R-09: Integrated Cost and Schedule Risk Analysis Using Monte Carlo Simulation of a CPM Model [8]. 65R-11: Integrated Cost and Schedule Risk Analysis and Contingency Determination Using Expected Value [7]. Systemic Risk: 42R-08: Risk Analysis and Contingency Determination Using Parametric Estimating [10]. 43R-08: Risk Analysis and Contingency Determination Using
Regardless of the risk analysis methods used, the findings of this article suggest that, at a minimum, you always test your p90 outcomes (the high scenario given to the business organization to test the robustness of their decision) against the empirical reality. If no other historical data is available, this article provides actual examples to consider. If your p90 is 25% or less over the base estimate, ask why NO study ever showed less than about 40% for p90; what risks are you missing? what impacts have you underestimated? Finally, and most important, ask how can we improve project practices and scope in consideration of the risk reality?
Conclusion
As a student of cost engineering and the editor/lead author of AACEs Total Cost Management Framework process [26], I am dismayed by the extreme disconnect between our practices and the long-known reality as shown in figure 1. There is an ongoing failure to effectively address the reality of project cost uncertainty and there is a lack of good historical data with causal information. This has led to a credibility crisis. It also raises an ethical question (if not a criminal one per Flyvbjerg); what does it mean if we understand reality but continue to use failed methods known to be contrary to experience to the potential detriment of our employers and clients? The AACE Canons of Ethics (item 2g) states: When, as a result of their studies, members believe a project(s) might not be successfulthey should so advise their employer or client [1]. In respect to large process industry projects, readers of this article can consider themselves so advised. There are of course practitioners who do address the entire scope of cost
25
risks (in AACE terms; systemic, projectspecific and escalation), capture data and consider the empirical record [16, 17, 18, 21, 42, and 43]. However, in the authors experience, the application of robust practices is uncommon. At a minimum, teams should at least test their worst case analysis outcomes against the empirical reality. They should study this articles references and their own enterprises historical experience (watching out for the small versus large project behavioral dichotomy). And, they should then seek to improve their practices to improve on past outcomes. The author does not agree with Dr. Flyvbjergs approach to using empiricism (i.e., reference class forecasting [19]) which implies that biases are so intractable that we are doomed to repeat the past. The article also points out that companies should not use accuracy as a cost estimating quality measure; it is a risk management and project control process quality measure. Tight accuracy is often an indicator of wasted capital; measures of project control process discipline and project cost competitiveness must accompany accuracy measures. In summary, this article references and summarizes over 50 years of empirical cost estimate accuracy research on large projects in the process industries. It shows how this reality compares (or does not compare) to what we say and do. Recommended risk analysis methods have been highlighted. Failed methods are exposed. It is hoped that the facts, observations and opinions brought together here will serve as a valuable reference on the topic of cost accuracy and uncertainty so that we can better speak the truth among ourselves and with management. The path to more realistic uncertainty forecasts, better contingency estimates and more profitable investments is clear and documented by AACE International.
REFERENCES
1. AACE International, Canon of Ethics. AACE International, Morgantown, WV, 2012. AACE International, Contingency Estimating General Principles,
2.
Recommended Practice 40R-08, AACE International, Morgantown, WV, (latest revision). 3. AACE International, Cost Engineering Terminology, Recommended Practice 10S-90, AACE International, Morgantown, WV, (latest revision). 4. AACE International, Cost Estimate Classification System As Applied in Engineering, Procurement, and Construction for the Process Industries, Recommended Practice 18R-97, AACE International, Morgantown, WV, (latest revision). 5. AACE International, Escalation Estimating Using Indices and Monte Carlo Simulation, Recommended Practice (draft) 68R-11, AACE International, Morgantown, WV, (latest revision). 6. AACE International, Escalation Principles and Methods Using Indices, Recommended Practice 58R-10, AACE International, Morgantown, WV, (latest revision). 7. AACE International, Integrated Cost and Schedule Risk Analysis and Contingency Determination Using Expected Value, Recommended Practice (draft) 65R-11, AACE International, Morgantown, WV, (latest revision). 8. AACE International, Integrated Cost and Schedule Risk Analysis Using Monte Carlo Simulation of a CPM Model, Recommended Practice 57R-09, AACE International, Morgantown, WV, (latest revision). 9. AACE International, Risk Analysis and Contingency Determination Using Expected Value, Recommended Practice 44R-08, (latest revision). 10. AACE International, Risk Analysis and Contingency Determination Using Parametric Estimating, Recommended Practice 42R-08, AACE International, Morgantown, WV, (latest revision). 11. AACE International, Risk Analysis and Contingency Determination Using Parametric Estimating Example Models as Applied for the Process Industries, Recommended Practice 43R-08, AACE International, Morgantown, WV, (latest revision).
12. AACE International, Risk Analysis and Contingency Determination Using Range Estimating, Recommended Practice 41R-08, AACE International, Morgantown, WV, (latest revision). 13. Arena, Mark, Robert S. Leonard, Sheila E. Murray, and Obaid Younossi, Historical Cost Growth of Completed Weapon System Programs TR-343, The RAND Corporation, 2006. 14. Bertisen, Jasper and Graham A. Davis, 'Bias and Error in Mine Project Capital Cost Estimation,' The Engineering Economist, 53: 2, 118 139, 2008. 15. Biery, Frederick, Improving Mineral Project Performance, Canadian Institute of Mining, Metallurgy and Petroleum Conference, May 2009. 16. Butts, Glenn and Kent Linton, The Joint Confidence Level Paradox; A History of Denial, 2009 NASA Cost Estimating Symposium, NASA, April 2009. 17. Curran, Kevin M., Value-Based Risk Management (VBRM), Cost Engineering, AACE International, AACE International, Morgantown, WV, February 2006. 18. Curran, Kevin M and Michael Curran, Handling the Truth in Risk Management, 2010 AACE International Transactions, AACE International, Morgantown, WV, 2010. 19. Flyvbjerg, Bent, Mette Skamris Holm and Sren Buhl, Underestimating Costs in Public Works Projects; Error or Lie? APA Journal, 68:3, 279-295, 2002. 20. Flyvbjerg, Bent and Alexander Budzier, Why Your IT Project May Be Riskier Than You Think, Harvard Business Review, September 2011. 21. Gough, Martin and Peter Maidment, The Stability Model Method of Risk Management and Early Prediction of Project Performance, The Revay Report; Revay and Associated Limited, February 2006. 22. Hackney, John W. (Kenneth K. Humphreys, Editor), Control & Management of Capital Projects, Chapter 18, AACE International, Morgantown, WV, 2002.
26
23. Harbuck, Robert H, Are Accurate Estimates Achievable During the Planning of Transportation Projects?, 2007 AACE International Transactions, AACE International, Morgantown, WV, 2007. 24. Hollmann, John K., The MonteCarlo Challenge: A Better Approach, 2007 AACE International Transactions, AACE International, Morgantown, WV, (latest revision). 25. Hollmann, John K., Recommended Practices for Risk Analysis and Cost Contingency Estimating, 2009 AACE International Transactions, AACE International, Morgantown, WV, 2009. 26. Hollmann, John K (editor), Total Cost Management Framework, AACE International, Morgantown, WV, 2006. 27. Juntima, Gob and Scott E. Burroughs, Exploring Techniques for Contingency Setting, 2004 AACE International Transactions, AACE International, Morgantown, WV, 2004. 28. Kosich, Dorothy, Vast Majority of Mining Projects Experience Steeper Cost Over-runs, MineWeb, 30 Nov 2011. 29. Kulkarni, Phyllis, Stop Punishing the Overruns, InSites, IPA, Inc., Sept 7, 2011. 30. Linder, Stefan, Fifty Years of Research on Accuracy of Capital Expenditure Project Estimates: A Review of the Findings and Their Validity, Center for Research in Controllership and Management, WHU - Otto Beisheim Graduate School of Management, Vallendar, Germany. April 2005. 31. Lundberg, Mattias, Anchalee Jenpanitsub and Roger Pyddoke, Cost Overruns in Swedish Transport Projects, CTS working paper
32.
33.
34.
35.
36.
37.
38.
39.
40.
2011:11, Centre for Transport Studies, Stockholm Sweden, 2011. Merrow, Edward W., Industrial Megaprojects, John Wiley & Sons, Inc., New York, NY, 2011. Merrow, Edward, W., Why Large Projects Fail More Often; Megaproject Failures: Understanding the Effects of Size, Presentation to the Joint Meeting of the AACE National Capital Section and American Society of Mechanical Engineers Section, April 20, 2011. Merrow, Edward W. and Brett R Schroeder, Understanding the Costs and Schedule of Hydroelectric Projects, 1991 AACE International Transactions, AACE International, Morgantown, WV, 1991. Merrow, Edward W., Kenneth E. Phillips and Christopher W. Myers, Understanding Cost Growth and Performance Shortfalls in Pioneer Process Plants, R-2569-DOE, p48, The RAND Corporation, 1981. Merrow, Edward W., Lorraine McDonnell and R. Yilmaz Arguden, Understanding the Outcomes of Megaprojects, R-3560-PSSP, The RAND Corporation, 1988. Mitchell, Paul, Achieving Major Capital Project Effectiveness and Corporate Performance, Effective Capital Project Execution: Mining and Metals, EYGH Limited, 2011. Noort, D.J., and C. Adams, Effective Mining Project Management Systems International Mine Management Conference, Australasian Institute of Mining and Metallurgy, October 2006. Oil & Gas Journal Online Research Center, US Pipeline Study 2009; Actual vs. Estimate (database for fee), Penwell Corporation, Houston TX, 2009. Rath, Michael, Robert Spittle, Paul Osman and Nicholas Tawa. Major Water Capital Projects: Critical
41.
42.
43.
44.
45.
46.
Factors For Improving Outcomes, Deloitte Touche Tohmatsu, March 2010. Reddall, Braden (reporting), Cost Overruns Common on Oil Projects Schlumberger, Thomson Reuters, Energy and Oil News, 2 June, 2011. Rolstads, Asbjrn, Per Willy Hetland, George Farage Jergeas and Richard E. Westney, Risk Navigation Strategies for Major Capital Projects: Beyond the Myth of Predictability, Springer- Verlag London Limited, 2011. Schroeder, Brett and Jan A. Jackson, Why Traditional Risk Management Fails in the Oil and Gas Sector: Empirical Front-Line Evidence and Effective Solutions, 2007 AACE International Transactions, AACE International, Morgantown, WV, 2007. Schulyer, John R., Investment Decision Making, Chapter 14 of The Engineers Cost Handbook, Editor Richard E. Westney, Marcel Dekker, Inc., New York NY, 1997. Stephenson, H. Lance, Cost Engineering Maturity Model (CEMM), 2011 AACE International Transactions, AACE International, Morgantown, WV, 2011. Thomas, Steve, Project Development Costs-Estimates Vs. Reality, Pincock Perspectives, November 2000.
ABOUT THE AUTHOR
John K. Hollmann is with Validation Estimating, LLC. He can be contacted by sending e-mail to: jhollmann@validest.com
27
TECHNICAL ARTICLE
Quantifying Estimate Accuracy and Precision for the Process Industries:
A Review of Industry Data

Abstract: This article quantifies industry metrics for estimate accuracy based on a sample of 462 capital projects executed by the process industries over the last 20 years, comparing estimated costs at the various stages of project definition to actual costs, normalized for a number of factors. The authors then contrasted the results from the industry survey with industry expectations, as presented in AACE International Recommended Practice 18R-97. The results of our survey indicate there is a much greater level of variability in estimate accuracy as compared to industry expectations. Further analysis demonstrates that projects which have completed a definitive project scope at authorization benefit from a reduction in estimate variability as compared to industry. This article was first presented as RISK.1100 at the 2012 AACE International Annual Meeting in San Antonio, Texas. Key Words: Capital projects, cost, estimate accuracy, process industries and scope n virtually all applications and scenarios, a cost estimates value is related to its capacity to predict the final cost of a planned scope of work. The importance of an accurate estimate confronts the cost estimator from all directionsfrom: the business/project sponsor, who is relying on the estimate to quantify the expected return on the capital invested; to the finance group, who is relying on the estimate to manage quarterly cash flow; and, to the project controls group, who is using the estimate as the basis for project cost and schedule control during execution.
All of these stakeholders have different uses for the estimate, but share a need in that they are relying on the estimate to have a reasonable degree of accuracy. The cost engineering community has devoted a substantial amount of time to consider practices and processes to improve estimate accuracy [5,16,17]. In contrast, only a limited number of articles and papers have also quantified the estimate accuracy ranges using actual industry data from completed projects [3,12,13]. In these instances, the data presented were only for a subset of industry (i.e., mining and minerals) or a specific type of project (i.e., megaprojects Review and Definitions or new technology projects), and The authors first considered the therefore were limited in an ability to definition of estimate accuracy as this quantify estimate accuracy for a concept has been such an important part
comprehensive set of industry projects. Despite these limitations, these studies were valuable as they provided a set of industry norms that quantify estimate accuracy based on actual project data. In the absence of more robust industry data, the cost engineering community has defaulted to a range of sources to quantify typical estimate accuracy, including in-house data, historical experience, or industry standards and Recommended Practices (RP) [1]. The objective of this article is to enhance industrys understanding of estimate accuracy by providing benchmarks of estimate accuracy for a range of estimate classifications. The authors calculate these metrics from a large sample of industry projects executed by the oil and gas, chemical process, and mining and minerals industries over the last 20 years. Given the importance of achieving a reasonable level of accuracy and precision for cost estimates, the goal of this article is to provide an independent measure of estimate accuracy and compare these results against current industry expectations.
28
of the cost estimating vernacular that the concept of an accurate estimate has evolved into a variety of potential definitions. In this section, the authors will review and define estimate accuracy and estimate precision, both of which are closely related. The authors will also use these concepts to introduce AACE International Recommended Practice 18R-97 (and 17R-97), which serves as a basis of comparison for our analysis [1,2]. Following a review of the definitions, the authors will consider the standard stage-gated capital project development work process for the process industries and apply the AACE International estimate classification to this process. Accuracy measures the closeness an activity is able to achieve to meet a desired value or target. For example, a field goal kicker in football is considered accurate if his field goal attempts are consistently through the field goal posts. AACE Technical Board Chair Larry Dysert defines estimate accuracy as, an indication of the degree to which the final cost outcome of a project may vary from the single point value used as the estimated cost of the project [6]. While estimate accuracy is measured based on a single point estimate, the accuracy of an estimate is best represented as a confidence interval ranging around the single point estimate. As presented in figure 1, the concept of estimate accuracy can be depicted using a standard probability density function that quantifies the potential minimum and maximum cost for a cost estimate. Based on the probability density curve as presented in figure 1, it has become an industry norm to present the estimate accuracy range as a plus or minus () confidence interval around the point estimate (e.g., a 10 percent estimate translates that the estimate will be accurate within 10 percent of the total point estimate; conversely, a 50 percent estimate will be accurate within 50 percent of the total point estimate). In addition to estimate accuracy, it is important to consider the concept of estimate precision. While accuracy measures ones ability to achieve a
target, precision is defined as the absence of variation. Cost estimates with a wide confidence interval (e.g., see the 50 percent estimate) are considered to be imprecise, while estimates with a small confidence interval are considered to be precise. Figure 2 presents the outcome matrix between accuracy and precision. As based on figure 2, precision without accuracy is a potentially dangerous phenomenon as it fools the users into an unwarranted sense of confidence. For example, one could argue (within another article, paper, or forum) that
many of the models developed by the financial industry to quantify risk as related to mortgage-backed securities focused on improving precision, but were entirely inaccurate with respect to the range of potential outcomes. In considering cost estimates and confidence intervals, both accuracy and precision should be considered within the same context. Of course, the objective of cost estimating (as with most related professions) is to achieve both accuracy and precision with respect to all cost estimates.
Figure 1 Estimate Accuracy Presented as a Probability Density Function
Figure 2 Outcome Matrix for Accuracy and Precision
29
Within the cost estimating community, the concept of a 10 or 50 percent estimate has evolved to describe potentially all aspects of an estimate, including the estimating methods, estimating tools, estimate accuracy, estimate precision, and inputs used for the estimate. Review of AACE International Recommended Practice 18R-97 AACE International Recommended Practice 18R-97 was developed to integrate all aspects of estimate accuracy within a common framework and set of guidelines In the absence of robust historical data, this RP has evolved into the de facto industry standard for estimate accuracy. Given that this RP will serve as a comparison against the results of this study, a brief summary of this RP is provided. RP 18R97 provides a set of guidelines for estimate classes considering projects managed through a standard capital project development /definition process (commonly known as the front-end loading [FEL] process). Figure 3 tabularizes the characteristics of the five estimate classes developed in this RP. The five estimate classes presented provide guidelines for how estimates progress from high-level approximations to more detailed estimates as a project progresses through the FEL process. The major guidelines for this RP are summarized as follows:
The expected precision of an estimate improves through the classification system, with Class 1 estimates exhibiting much less expected variability than Class 5 and Class 4 estimates. The estimate classes show approximately the same degree of accuracy with the average point estimate for each class approaching, on average, the actual project cost. The estimating methodology becomes more detailed and complex as estimates progress through FEL. The degree of effort required to prepare an estimate becomes exponentially greater as estimates progress through FEL. The data/project inputs required (as measured by the percent of definition/engineering complete) to develop an estimate become greater and more detailed for lower estimate classes.
and accuracy of the information which serves as the basis for that estimate. In addition to the above points, the RP provides the expected accuracy range for each estimate classification. Table 1 summarizes the expected accuracy range guidelines. The RP has reached consensus that estimates improve with respect to both accuracy and precision as a project progresses through development. However, the RP is unable to reach consensus regarding the relative degree of precision for each estimate class. As presented in table 1, each estimate class contains a range of both minimum and maximum confidence intervals. These ranges are insightful as they provide some indication of the wide range of accuracy expectations that currently reside within Industry. Given the variability of inputs for Class 4 and 5 estimates, the Industry expected ranges show the least consensus in estimate ranges as compared to the other estimate classes. Project Life Cycle for the Process Industries Given the high capital requirements for the oil and gas and associated process industries, most owner companies have adopted a formal, stage-gated process to develop business ideas into working capital assets and plants [4,9,10,11]. While the details of these processes vary by company, they
The last point is perhaps the most valuable as it formalizes the linkage between the class of estimate and quality and completeness of data, information, and deliverables that serve as inputs into the estimate. This concept (i.e., inputs driving outputs) cannot be underscored as it maintains that the key driver for estimate accuracy and precision resides in the completeness
Figure 3 Summary of AACE International Estimate Classification System
30
adhere to a stage-gated process of identify-select-define-execute-operate to transform a business opportunity to a working plant [4,11]. In this model, the capital project is developed through a series of phases with specific deliverables required to finish a specific phase. Over time, the stage-gated process has become an industry standard for the development of capital projects and is used by most owners and contractors in the process industries. Past research has indicated that the stage-gated work process improves overall capital performance [10,18]. In addition, the principles of a
stage-gated process serve as a basis for the AACE International Total Cost Management Framework [7]. Figure 4 depicts a generic stagegated work process for capital project development. In most cases, the system has five gates, with three (generically termed as appraise, select, and define) composing project definition (or preauthorization), or the FEL stage. It is this portion of the project in which the cost estimator is generally the most involved as each stage gate will require an estimate of the planned scope so the gate review can be completed (e.g., a business lead cannot
assess the financial gains of a capital investment without some estimate of the capital cost required to execute the project). Most company project systems will require an estimate of a particular quality to meet the FEL pass-gate requirements. A review of these project systems shows that these classifications generally follow the AACE International categories with some variation in the specific practices and related accuracy classifications required by each company (or even each business unit). For this reason, the cost estimates captured for each FEL stage may not, in
Table 1 Expected Estimate Accuracy Ranges for AACE International Estimate Classification (adopted from 18R-97)
Figure 4 Stage Gated Process for Project Development
31
Figure 5 Matching The AACE International Estimate Classification With Typical FEL Phases some cases, directly match the AACE International estimate classifications. As such, one must not think of these data as corresponding directly to particular AACE International classifications, but rather the range of possible AACE International classification requirements enforced by this set of companies at a particular gate. However, the authors argue, that as an industry sample, they do provide a good approximation of the AACE International estimate classes (which are approximations themselves). In addition, a review of industry project systems have many similarities to the AACE International RP 18R-97. For example, as projects move closer to fullfunds authorization, the accuracy of the estimate relative to the expected outcome should improve, as well as the requirements for greater definition and knowledge regarding the project scope and execution plan. Based on this review of these industry systems, figure 5 merges the AACE International Estimate Classification system with the Stage-Gated Work process based on this review of company standards and norms. In general, estimates at the end of appraise (or FEL 1) correspond to AACE International Class 5, estimates at the end of select (FEL 2) correspond to AACE International Class 4, and estimates at the end of define (FEL 3) correspond to AACE International Class 3. Database Description and Metrics Calculation The authors quantified the estimate accuracy ranges for industry from a sample of projects executed by the process industries over a 20-year period from 1988 to 2008. Cost data for the project sample was collected as part of an overall industry benchmarking effort as directed by Independent Project Analysis (IPA) and managed through IPAs project evaluation system benchmarking process (PES is a registered trademark of IPA). For the project data collection, interviews were conducted by IPA project analysts with the project teams with over 2,000 variables captured to reflect each projects technical scope, characteristics, use of best practices, and cost/schedule outcomes. The industry dataset employed for this study included 462 completed large capital projects for which estimated costs were collected at the various FEL phases and actual costs at project completion. Figure 6 presents summary characteristics for this industry sample. The industry projects were executed by 62 different owner companies and include a range of project scopes, project types, and locations. The industry sample consisted primarily of large capital projects with a mean cost of $64 million (with a range from $10 million to $3.3 billion). The projects were executed primarily in North America, but also in Europe, Asia, and the rest of the world (ROW)primarily the Middle East and South America. For each project, cost estimate data were collected at four distinct points: at the end of each FEL gate (estimated costs), as well as at project completion (actual costs). Cost deviation metrics were calculated based on equation 1. Project costs were normalized to a constant currency and time period to
Figure 6 Industry Database of Projects by Project Type and Industry Sector
32
remove any bias driven by general price escalation and currency fluctuation. The costs collected included all project management, engineering, materials, and labor costs expended on the project from the beginning of project definition through the completion of construction (as defined as mechanical completion). For the estimates, included were both the base estimate as well as any estimated contingency included in the estimate. Estimated escalation was excluded from this analysis as the authors normalized both estimated and actual costs to a constant dollar basis. In terms of scope and project changes, the authors included all costs associated with normal scope development and project changes, assuming there were no changes to the underlying business case and overall project scope. Therefore, costs associated with major scope changes (e.g., increase in design capacity for a unit) were excluded from the estimates to maintain an apples to apples comparison with the actual costs ,while costs associated with normal scope development (e.g. increase equipment spacing to improve unit maintenance) was included in this analysis. The currencies were normalized using published exchange rates based on the estimate month [15]. For actual costs, currencies were normalized based on an average exchange rate during the project execution schedule. Project costs were also adjusted to remove the effects of force majeure and other uncontrollable external effects (e.g., labor strikes, natural disasters, etc.). Study Results This section presents the cost deviation statistics for each FEL phase/AACE International Estimate Class based on the industry project sample. Figures 7-9 present the cost deviation distribution for estimates at the end of FEL 1, FEL 2, and FEL 3, respectively. Each figure presents the overall distribution of cost deviation, as well as the summary descriptive statistics (i.e., mean, median, standard deviation). Based on the cost deviation metrics as calculated using equation 1, a value of 0.00 means that the estimated costs matches the actual cost exactly. Values
greater than 0.00 indicate an overrun from the estimated costs, while values less than 0.00 indicate an underrun of actual costs versus the estimated costs. Figure 7 presents the distribution of cost deviations for the FEL 1 cost estimates (or Class 5 estimates). At FEL 1, the objective is to develop an order of magnitude/conceptual estimate to determine the reasonableness of a projects business case. These estimates generally serve as inputs into financial models (e.g., to calculate net present value or return on capital) to assist business in allocating capital to the most profitable business opportunities. As shown in figure 7, the accuracy of industry estimates at FEL 1 are extremely variable with the cost deviation ranging from 50 percent to over 200 percent at the extremes. Industry will tend to overrun these estimates with an mean overrun of 71 percent and median overrun of 38 percent. The standard deviation of 118 percent indicates the wide range of variability for industry FEL 1 estimates.
The probability distribution for FEL 1 estimates indicates a log-normal (or right skew) distribution with a majority of FEL 1 estimates overrunning against the final costs. The authors note that the industry sample size for FEL 1 estimates is smaller than the other estimates primarily because of the difficulty in obtaining reliable early estimates from Industry. Figure 8 presents the distribution of cost deviations for a set of estimates published at FEL 2, as compared to the final costs. This industry sample includes 462 projects executed from 62 owner companies. As discussed in the background section, there exists among companies some variability with regard to the estimating requirements at the end of FEL 2. However, in most cases, companies will require an estimate that meets the AACE International Class 4 requirements (with perhaps a few exceptions of requiring a Class 3 estimate). For Industry, the mean cost deviation for FEL 2 estimates was 15
Equation 1
Figure 7 Cost Deviation Results for FEL 1 Estimates Compared to Actual Costs
33
percent with a range of 61 to 224 percent. The median cost deviation is 5 percent, indicating that Industry will typically overrun the FEL 2 estimate. The standard deviation of 41 percent indicates a relatively large degree of variability for FEL 2 estimates. Assuming a normal distribution, approximately 67 percent of industry FEL 2 estimates would fall between 26 and 56 percent of the actual costs. As with the FEL 1 estimates, the distribution for FEL 2 estimates indicates a left skew with a greater percentage of FEL 2 estimates overrunning rather than underrunning the actual cost. Figure 9 shows the distribution of cost deviations from a set of estimates published at FEL 3, or full-funds authorization, against actuals. For most industry project systems, the FEL 3 estimate will correspond to a Class 3 estimate; however, in some cases these estimates may be closer to a Class 2 estimate. As presented in figure 9, the mean cost deviation for FEL 3 estimates is 8 percent with an overall range of 40 to 95 percent. The median value of 1 percent shows that the probability of an overrun versus an underrun is virtually identical with a 50 percent probability of both. The overall variability is reduced from the earlier FEL 1 and FEL 2 estimates, with the standard deviation of 30 percent. However, the authors note that this degree of variability is much greater than the 10 percent confidence level generally attributed to an authorization grade estimate. The distribution of industry cost growth shows a slight right skew, however, this is much less pronounced that the FEL 1 and FEL 2 estimates. Comparison With AACE International Estimate Classes The results presented in figures 7 to 9 indicate that, as based on historical performance from completed industry projects, the actual cost deviation for industry shows a greater degree of variability and inaccuracy than current expectations. As further analysis, the authors have compared the industry distributions presented in figures 7 to 9 against the expected accuracy ranges taken from the AACE International RP
18R-97 as summarized in table 2. This comparison, presented in figure 10, highlights a number of disconnects with respect to industry expectations and actual industry data. Perhaps the most striking difference is the industrys lack of precision for FEL 1 estimates, as compared to the industry expectation. As discussed in the previous sections, it was industrys expectation that while early estimates show a large degree of variability and imprecision, the overall accuracy (as measured by the average cost deviation) remains relatively constant through the estimate classes, with all industry estimates (regardless of class) having a mean cost deviation at or near 0 percent. The authors results show this assumption to be incorrect as project
systems tend to systematically underestimate projects during the initial stages of the project definition process. The mean cost deviation of 71 percent for FEL 1 estimates marks a substantial departure from industry expectations. From a cost estimators perspective, this degree of systematic underestimating is not surprising given that most projects (at that point in the FEL process) have only established the most rudimentary project scope. From a business planning perspective, the results are troubling as they indicate that the early estimates, which serve as input into any business case and metrics (NPV, IRR, ROC), are systematically much lower than the actual cost required for a typical project. From this perspective, these early estimates may be insufficient to
34
serve as a robust input into any capital allocation process. A potential contributing factor to this finding is that during the business planning process, early estimates that may be near the actual cost (or even conservative/higher than the industry cost) are at a disadvantage because the high capital costs tend to kill these projects because they may not meet a financial hurdle to move into later FEL stages. In their review of estimating bias for the mining industry, Bertisen and Davis also consider this hypothesis noting that for the mining industry, the scarcity of project financing drives project sponsors to enhance the project economics through lower capital estimates.[3] Further to this analysis, the authors employed the Shapiro Wilk test for statistical normality to test whether the FEL 1 industry estimates, on average, will overrun. As expected, the sample of industry FEL 1 estimates failed this test for statistical normality.
As with the FEL 1 estimates, the industry sample of FEL 2 estimates exhibited a lack of precision with these estimates being, on average, 14 percent above actual costs. While the overall variability for FEL 2 estimates show an improvement as compared to the FEL 1 estimates (the standard deviation for these estimates is 41 percent), this degree of variability is greater than the AACE International Class 2 expectations. The results for the FEL 2 estimates are somewhat troubling, as it is in this stage that most project systems will require a capital project to have finalized the project scope and location and completed early engineering and design sufficient to complete a process flow diagram and heat and material balances. Despite the requirements for closing scope, Industry tends to systematically underestimate the project costs. For FEL 3 (or authorization grade) estimates, both the precision and accuracy of estimates improve from the previous classes. Unlike the FEL 1 and FEL 2 estimates, the FEL 3 estimates
meet the AACE International Class 3 expectation of precision with a median cost deviation of 1 percent. The mean cost deviation of 8 percent indicates some degree of right skew in the distribution, however this skew is much less pronounced than in the previous estimate classes. However, while the precision improves, the variability of the estimates (as measured by the standard deviation of 30 percent) is much greater than industry expectations. As discussed in the review of estimate accuracy, the majority of industry project systems will require a plus or minus 10 percent estimate for the authorization gate. Given that any project which exceeds the 10 percent threshold will usually require a supplemental authorization, one could argue that this a 10 percent threshold represents either the minimum or maximum range for the actual costs or a range in which the majority of capital projects, under normal circumstances, will be completed.
Figure 10 Industry Estimate Accuracy as Compared to the AACE International RP 18R-97 Ranges
35
A review of industry data will show that approximately 66 percent of industry projects will be completed with a cost deviation ranging from -22 to 38 percent. The probability is approximately 50 percent that any given estimate will meet the 10 percent threshold. It is argued that the 10 percent estimate for authorization is an estimating concept that has evolved from its intended purpose. This concept has developed multiple meanings that may, in fact, conflict with each other. For example, it is likely while this definition may be used by the cost estimator to simply describe the overall quality of an estimate, it is considered by the business or project sponsor as an absolute range to predict the final costs. If one accepts the latter definition, then one must conclude that Industry is not achieving its objectives with respect to estimate accuracy. Another potential conclusion is that the cost estimating community, as a whole, is doing an inadequate job in communicating the potential range of actual costs for any given project when they describe an estimate in these terms. In either case, these results point to an overall lack of robustness for Industrys track record in quantifying risk (generally through contingency) for estimates. It is noted that the cost estimating community, using the most advanced risk quantification and contingency setting techniques, will set expectations to their clients (e.g., project teams, business sponsors) that the final estimate will be within 10 of the final estimate. It is also noted that this expectation is only met 50 percent of the time. Unlike the FEL 1 and FEL 2 estimates, which are systematically underestimated, it is noted that the FEL 3 estimates, on average, have a cost deviation near 0 percent. However, these estimates deviate from the AACE International RP as they show a larger degree of variability than expected with respect to both underruns and overruns, which indicates Industry is as likely to underestimate risk as overestimate risk.
Figure 11 Cost Deviation Results For FEL 3 Estimates With Finalized Project Definition Compared to Actual Costs Discussion: The Importance of the Level of Definition As presented in figure 10, the results from our industry survey show a disconnect between industrys expectation of estimate accuracy and precision as compared with actual project data. Given the degree of difference between the industry expectations and historical data, a number of contributing factors to this difference are proposed. Potential factors are summarized as follows: There could be systemic issues for a particular company or industry that contributes to a pattern of overrunning or underrunning that combine to produce a broader distribution of variability. There are execution issues that occur on projects that one is not controlling, such as poor management of resources, design changes, poor labor productivity, poor planning, and/or a host of other possibilities that drive the larger than expected cost variations. The quantification of risk (through the contingency setting exercise) is not robust enough to properly identify and quantify risks to the project [8]. And, The cost estimates are being developed with deficient inputs, such as incomplete project design or underdeveloped project execution plans.
These points (as well as many other potential contributing factors) likely contribute to the overall variability experienced by Industry. However, the authors will focus on the last point given that IPA, as part of it project benchmarking, routinely collects the use of particular engineering and design practices employed during project definition. IPA has developed a model to measure the completeness of a projects scope and key deliverables at authorization. This model, which is termed the Front-End Loading (FEL) Index is a quantitative measure of how well defined a capital project is at authorization. It is noted that AACE International RP 18R-97 references the level of project definition that corresponds to each estimate class. These data are presented in figure 3 and are expressed as the percentage of project definition complete. IPAs FEL Index is a more robust measure of the level of project definition as it measures the status of key deliverables related to the project scope (e.g., process and instrumentation diagrams, plot plans). From a cost estimating perspective, the FEL Index is useful as it provides a measure of the completeness of the project inputs used to develop the estimate. Using the FEL index, one is able to segregate industry projects which have substantial gaps and uncertainties in their design, versus
36
those projects which have completed all key deliverables to a best practical level of project definition. Figure 11 presents the distribution of cost deviations for FEL 3 estimates from a subset of industry projects that had completed all necessary design engineering deliverables to support the authorization estimate. As presented in figure 11, the cost deviation for this sample of 253 projects shows significantly less variability (i.e., greater amount of precision) than the overall industry FEL 3 distribution. The mean cost deviation for FEL 3 estimates that have achieved an appropriate level of engineering definition is 2 percent with a range of 35 to 70 percent. The standard deviation of 21 percent indicates that approximately 66 percent of this industry subset will be completed with finals costs within 20 percent of the estimated cost. Using a standard t-test for statistical difference, the authors found statistically significant differences for both the mean and standard deviation of both project sets (Pr < 0.0001 and 2*Pr < 0.0001, respectively) between the well-defined project group and projects group that had gaps in engineering The results in figure 11 clearly demonstrate that to improve estimate accuracy and precision, an organization must first focus on project inputs and completing those key deliverables which are required to produce an accurate estimate. This conclusion is not new to the industry and has been a major finding in a range of studies focusing on the outcomes of capital projects. [5,12,13,14] Given these results, one must conclude that Industry should first and foremost focus on project fundamentals to improve overall estimate accuracy. Conclusion The authors data demonstrate that the AACE International estimate classifications express a set of ranges that roughly correspond with the degree of confidence in a given point estimate. However, the current level of variability expected by industry (as quantified in the AACE International RP 18R-97) does not represent the overall variability
experienced by industry as based on a review of historical project data. Given this finding, one concludes that the estimating community, as it is currently presented to the sponsors and gatekeepers of projects, is setting the expectation of estimate accuracy that is not sustainable when compared to historical performance. In most cases, the project stakeholders have an unfounded sense of confidence that an estimate will accurately predict final cost within its published ranges. This disconnect between expectations and reality is difficult for both the cost estimating community and the project management community as a whole. Would a decision board feel comfortable authorizing a project with a Class 3 estimate if it knew that there was a less than 50 percent chance that it would actually come within its published range? The authors suspect that this is not the case. However, the authors note that one tool that is available to the estimator and the gatekeeper is the knowledge that those chances greatly improve if the project has completed all key design deliverables at authorization. This means moving forward without any open scope, having the required deliverables to support quantity take-offs, and having a functionally integrated team that is aligned on that scope. Even then, opportunities still clearly exist to improve estimate accuracy to the point at which industry is actually capable of achieving the AACE International classification standards in practice.
4.
5.
6.
7.
8.
9.
10.
11.
REFERENCES 1. AACE International. Cost Estimate Classification System -- As Applied in Engineering, Procurement, and Construction for the Process Industries. Recommended Practice 18R-97. AACE International. Morgantown, WV, (latest revision). 2. AACE International. Cost Estimate Classification System Cost Estimating and Budgeting. Recommended Practice 17R-97. AACE International, Morgantown, WV September 2003. 3. Bertisen, J. and Graham A. Davis. Bias and Error in Mine Project
12.
13.
Capital Cost Estimation. The Engineering Economist: 53, (AprilJune 2008): pp 118-139. Chuong, Yuteck. Project Estimate Reviews for Oil and Gas Projects. 2007 AACE International Transactions. Morgantown, WV 2007. Dysert, Larry. Scope Development Problems in Estimating. 1997 AACE International Transactions. Morgantown, WV: AACE International, 1997. Dysert, Larry. Is Estimate Accuracy an Oxymoron. 2006 AACE International Transactions. Morgantown, WV: AACE International 2006. Hollman, John, editor. 2006. Total Cost Management Framework: A Process for Applying Skills and Knowledge of Cost Engineering. AACE International. Morgantown, WV. Hollman, John. Estimate Accuracy: Dealing with Reality. 2012 AACE International Transactions. Morgantown, WV 2012. Jambhekar, Vijay and Stephen D. Weeks. Estimate Review and Assurance Owners Challenges. 2007 AACE International Transactions. Morgantown: WV 2007. Lavingia, Nick. Improve Profitability Through Effective Project Management and TCM. 2003 AACE International Transactions. Morgantown: WV 2003. Lavingia, Nick. How to Create a World Class Project Management Organization. 2006 AACE International Transactions. Morgantown: WV 2006. Merrow, E., Kenneth S. Phillips, and Christopher W. Myers. Understanding Cost Growth and Performance Shortfalls in Pioneer Process Plants. RAND/R-2569-DOE, Santa Monica, CA. The RAND Corporation, 1981 Merrow, E. Understanding the Outcomes of Megaprojects: A Quantitative Analysis of Very Large Civilian Projects. RAND/R-3560PSSP, Santa Monica, CA, The RAND Corporation, 1988.
37
14. Merrow, Edward W. Industrial Megaprojects. John Wiley & Sons, Inc. New York, NY 2011. 15. Oanda Currency Converter, http://www.oanda.com/currency/c onverter/. 16. Smith, Mark A. and Richard L. Tucker. An Assessment of the Potential Problems Occurring in the Engineering Phase of an Industrial Project: A Report to Texaco, Inc. Austin, TX: The University of Texas, 1983. 17. The Construction Industry Institute. Pre-Project Planning Handbook. CII Special Publication 39-2. 1995. 18. Whiteside, James D. and Tyler Humes. Front-End Engineering and Design: Influence Over a Projects Outcomes. 2010 AACE International Transactions. Morgantown, WV: 2010.
ABOUT THE AUTHORS
Alexander Ogilvie is with Independent Project Analysis. He can be contacted by sending an e-mail to: aogilvie@ipaglobal.com
Fredrick P. Biery is with Independent Project Analysis. He can be contacted by sending an e-mail to: fbiery@ipaglobal.com
Robert A. Brown, Jr., is with Independent Project Analysis. He can be contacted by sending an e-mail to: rbrown@ipaglobal.com
Paul Barshop Independent Analysis. He contacted by an e-mail to: pbarshop@ipaglobal.com
is with Project can be sending
2013 CONSTITUTION AND BYLAWS AMENDMENTS EXPLAINED

The Governance Task Force created a PowerPoint presentation to explain its proposed amendments to the AACE Constitution and Bylaws. The PowerPoint presentation may be viewed at http://www.aacei.org/mbr/gtf_recommendations.ppt An online webinar is set for 8 a.m. Eastern Time on Nov. 13 at https://www2.gotomeeting.com/register/251679458. Registration is free but does require advance sign up. AACE International Board of Directors is recommending that members approve provisions to provide more direct representation in the Associations decision-making process by its three associate boards. In addition, the Governance Task Force recommends the creation of an Executive Committee to handle routine administrative and executive functions requiring Board of Directors attention. If approved by the membership in an election in 2013, the net eect will be to expand the AACE International Board of Directors by three members in 2014-15. The recommended amendments include a provision for direct representation by the three associate boards, Technical, Education and Certication on the Board of Directors. Currently, those entities are represented by a single ocer, Vice President-TEC (Technical, Education and Certication). As a result of the direct representation on the Board of Directors, the current position of Vice President-TEC would be phased out. Each associate boards representative to the Board of Directors would be nominated by each associate boards respective nominating committee and approved by two-thirds of the respective associate board. The new positions would be entitled Vice President-Technical Board, Vice President-Education Board and Vice-President-Certication Board. All would serve as ex-ocio members of their respective associate boards and these positions could be lled by the current associate chair if desired by the associate board. The Second Recommended Change in the Board of Directors structure would be splitting the current Vice President-Regions into two positions, Vice President-North American Regions and Vice President-International Regions. Consistent with the direction of the current strategic direction to pursue international growth, this structural change will assure that there will always be international representation on the Board of Directors while also assuring adequate attention is provided for the sections in North America. The two vice presidents for regions will be elected on alternate years for continuity purposes. Regional representatives to the Board of Directors will remain unaected. A Third Recommended Change is the creation of an Executive Committee comprised of the ocers of the Association. The Executive Committee will recommend actions for consideration of the entire Board. Any actions recommended will require a subsequent two-thirds approval by the entire Board of Directors by electronic ballot. The entire Board of Directors focus on strategic and policy issues when we are deliberating as a Board of Directors. Our task force rmly believes that these recommended amendments are in the best interests of the Association in achieving our mission and vision. At the same time we want to assure that all members have an opportunity to ask questions and express their opinions prior to voting on the recommendations from February 1-March 15, 2013, Task Force Chair, Steve Revay stated.
38
TECHNICAL ARTICLE
Common
in Dealing With Project Risk

Abstract: Unfortunately, many people using risk management do not fully understand basic risk concepts and therefore use incorrect techniques in preparing and implementing risk management plans. The author has reviewed and critiqued client risk management process and procedures, along with risk management plans for projects, and the same mistakes reoccur on a regular basis. Based on these reviews, this article will present the top ten errors people make in dealing with project risks and how these mistakes greatly reduce the value of risk management. The author has also done numerous projects where risk management was successfully used, and this article will also discuss effective risk techniques that should be used on all projects. If you think you know how to deal with project risks, read this article to see if you are making any of these common errors. You may be surprised! This article was first presented as RISK.944 at the 2012 AACE International Annual Meeting in San Antonio, Texas. Key Words: Contingency, projects, risk management, and schedule that should be followed when dealing with risks. Here are the processes that are to be done as part of risk management [3]: Prepare a Risk Management Plan This should be completed early in the project planning phase. First decide if risk management is needed. On some small, simple projects it may not be value adding. But for most projects risk management is needed. The risk management plan should document the techniques, tools and responsibilities for risk identification, risk analysis, risk response planning and risk monitoring and control. Identify the Project RisksThe most commonly used techniques are brainstorming and checklists. As part of risk identification also identify any triggers for each risk, which are early warning indicators that the risk event may occur. Analyze the Project RisksThe simplest technique is assigning a value for the probability and impact of each risk and calculating the risk factor. The risks are then prioritized and the highest scores are the threats and opportunities that should be actively managed as part of risk response planning. On larger projects advanced risk analysis using
hen the word risk is used in casual conversation it invariably has a negative connotation: risk is bad and should be avoided whenever possible. The dictionary reinforces this negative connotation by defining risk as the possibility of loss or injury, or someone or something that creates or suggests a hazard [1]. However, for project management risk has a less threatening and more realistic definition as, any uncertain event or condition that, if it occurs, has a positive or negative effect on a projects objectives [2]. In this expanded scope of understanding, risk is associated with uncertaintythe possibility that things will not go as planned because of
uncertain risk events, which can have good or bad impacts on the project. Risk management includes the processes of identifying potential project risks, analyzing risks, planning responses to the most important project risks, and also monitoring and controlling risks over the life of the project. The goal of risk management is to maximize the effects of potential opportunities, and minimize or eliminate the threats to the project objectives. Risk Management Process Before exploring the common errors made in dealing with risks, lets review the suggested process steps for managing project risks. This will ensure a common understanding of the sequential actions
39
quantitative techniques, such as Monte Carlo analysis, may be appropriate. Plan Responses for Project Risks The project team should decide on actions to be taken to deal with the threats and opportunities. The responses should include the actions to be taken to deal with the risk event before it may happen, along with contingency plans that will be implemented if the risk event happens. And, Monitor Control RisksManaging risks is an ongoing process over the entire life of the project. This includes identifying and dealing with any new risks.
Error #1: Not Considering Opportunities Most project teams view risk as bad and to be avoided whenever possible, and therefore ignore the possibility that something beneficial might happen as a result of a risk event. However, successful project teams need to deal with all project risks, and using the simple definition below will help you focus on both good and bad risk events: PROJECT RISKS = OPPORTUNITIES + THREATS Consequently there are two kinds of risks to consider on projectsthreats and opportunities. Threats are easier to understand because when they occur there is a negative impact on the project objectives. In contrast, a favorable or advantageous impact on project objectives when a risk event occurs is an opportunity. Dealing with major threats obviously needs to be done on projects, but dont forget to consider and take advantage of opportunities. Many people have a difficult time thinking of project opportunities, so here are some examples from recent projects:
Exploit take steps to make sure the opportunity happens. Share shares the opportunity with another group that is better positioned to take advantage of it for the benefit of the project. Enhance take steps to increase the probability and/or the positive impact of the opportunity (note that increasing the probability and/or impact values increases the risk factor value). And, Accept take no steps to actively pursue the opportunity, but take advantage of the opportunity if it happens.
Common Errors Made in Managing Project Risks The top ten errors project teams make with risk management will now be listed. As noted, this is based on consulting work with numerous clients. The list covers the mistakes in sequential order when doing risk management on a project, and is not intended to be a prioritized list of errors. Here are the top ten risk errors made on projects: Not considering opportunities; Confusing risk causes, events, and impacts; Using checklists and not looking for other possible risk events; Underestimating impacts; Not using 100% probability during project planning; Not considering sensitivity with risks; Calling risk response planning mitigation; Not considering contingency plans along with response plans; Not making team members responsible for specific risk events; and, Not making risk management an ongoing process.
Lets look at an example for each of these response strategies, and how the identification and planning for a specific opportunity might look: ExploitYour resource plan includes adding Susan, a senior electrical engineer, starting in January, when her work on another project will finish. Its now early November, and you find out the other project has been cancelled. Looking at your schedule, you realize changing the sequencing of some project work will allow you to use Susan now and save three weeks in your project schedule. You contact the resource manager and request Susan be immediately reassigned to your project. Reworking your schedule and immediately requesting Susan is exploiting the opportunity created by the cancellation of the other project. ShareDuring the planning phase for your laboratory renovation project, you discover a new flexible piping technology that can reduce service piping installation time for the lab benches by 30% with minimal cost impact. You preliminary plan is based on using an internal maintenance group for the piping work. However, they dont have the knowledge and experience in working with the flexible piping technology. Since the lab bench piping is on the critical path and the client sponsor wants to improve the end date, you decide to outsource this work to a contractor with
Special pricing offered by a supplier. Competitive market conditions for a specific service. Sudden availability of a key resource for a short time period because of another project being postponed. Availability of some needed equipment (such as servers or desktop computers) from another company that is downsizing their operations. And, Availability of a government investment tax credit for work done before a specified date.
Most project teams do develop plans to deal with threats, but unfortunately very few plan for opportunities. That is a mistake because planning will let you take advantage of these opportunities if they do occur. How many times have you heard about lost opportunities on The rest of this article will cover each projects? That happens when specific risk mistake in more detail, along with plans are not developed to make the effective risk techniques that should be opportunities reality. used on all projects. Listed next are the four risk response strategies for opportunities [4]:
40
Figure 1 Effective Risk Identification Tool expertise in flexible piping systems, and release the internal maintenance group for work on other projects. This is a share since you are taking advantage of the opportunity (using flexible piping) to shorten the schedule, and the contractor is making a profit on the work. EnhanceYour company has been selected as the painting contractor for a bridge renovation project. The project specification calls for a paint with a 15-year life. However, your company has developed a new epoxy paint that will last 30 years or more, and it only costs 30% more than the standard paint (but has a higher profit margin since it is a proprietary product of your company). You notify the project manager, Rita, of this opportunity. She likes the added value, but the state Department of Transportation (DOT) Director will have to approve the change. Rita suggests a meeting with the DOT project engineer, Jim, to discuss the benefits and enlist his support in getting the change approved. Rita is confident Jim will support the scope change and help obtain the necessary approval within the DOT. This is an opportunity for the project since the better paint will last substantially longer, plus an opportunity for the painting contractor to earn some extra profit. The action of meeting with Jim and getting his support increases the probability of the scope change being approved. AcceptThe testing team leader informs you there is a chance the integration testing on your software development project will not take as long as the plan shows, but the probability is low and best case it might save up to one week for the one year project. You decide to record the opportunity in the risk register, but take no other actions at this time since the impact to the project is minimal. identification process by not providing a focus on actual risk events and this is important because risk events have one unique feature, and that is uncertainty! A risk cause is a definite event or circumstance that exists. A cause is a fact and has no uncertainty. For example, multiple job opportunities for top notch programmers, resulting from a shortage of computer programmers in the region, would be a risk cause. Risk events are simply things that may or may not happen. An example of a risk event would be losing one or more key software programmers to another software company who offers them more money. Risk impacts are the results that will occur if the risk event happens. For the example above, if a key software programmer leaves for a higher paying job, it probably will have an impact on the project cost and schedule.
Error #2: Confusing Risk Causes, Events and Impacts The first step taken by a team in implementing the risk management plan is risk identification, which is determining which risks may affect the project and understanding the characteristics of each specific risk event. Common tools used by many project teams for risk identification are checklists and brainstorming along with interviews of key project stakeholders. However, a common error during risk identification is not differentiating between risk causes, risk events, and impacts. This failure can dilute the risk
An effective risk identification technique is expressing each project risk as a sentence. Your sentence should include the cause, risk event, and impact as shown below [5]:
41
Because of <Cause>, <Risk Event> Could To initially develop the checklist for Occur, Resulting In <Impact> group of similar projects, get some experienced project personnel together Note that a risk event may have more and brainstorm the risk events they have than one cause, and more than one encountered on projects. The checklist impact. This technique can be used to should be considered a dynamic fully describe each identified project risk, document, and as previously unidentified and is an effective method to help keep risk events occur on other projects, these the project team focused on the actual risks should be added to the checklist. risk events. The lesson learned sessions done at project completion is also a good place to One tool that can be useful during identify new risk events that should be brainstorming sessions is the use of post- added to the risk checklist. it notes as shown in figure 1. A separate When using checklists, it can be sheet of chart paper is used to list risk helpful to segregate the risks into causes. The post-it note format has the categories. The exact grouping will risk event description in the center, and depend on the needs of the organization. the cause numbers listed on the left. The One set of categories frequently used is four possible impacts are listed on the by project phase: initiation, planning, right side of the post-it note and the design, build, test, and deployment. For applicable impacts are circled. Risk the civil engineering company doing road categories can be added at the top, and construction, a grouping that could be the post-it note is also used for risk used is design, government/regulatory, analysis, with values assigned for contractor, suppliers and weather. probability and impact, and the Some teams have used the calculated risk factor. This format has categories of cost, schedule, quality and been successfully used on numerous functionality to organize their risks. This projects. isn't a good idea, however, since these four categories are also the risk impacts, Error #3: Using Checklists and Not and many risk events will have more than Looking for Other Possible Risks one impact. What that means is that if Many organizations do similar you try using this set of categories you projects over and over. Examples include won't be able to easily place risk events a civil engineering company specializing with multiple impacts into a single in road construction, or a computer category. programming company that only does As discussed above, a risk checklist is data migration projects. Most of the risk not inherently a bad thing, but you must events will be the same from project to make sure the identification of any project, and in these situations the use of additional risks is done in conjunction a risk checklist is effective. with use of the checklist. Use The problem with this approach is brainstorming to find new risks. One complacencymaking the false technique is to award a simple 'prize' assumption that the only possible risk such as a car wash or movie ticket to the events are included in the checklist. The team member who identifies the most checklist can certainly be a good place to new risk events not already on the start, provided it is not the only thing checklist. done to identify risks. Your team also needs to scan the horizon to identify any Error #4: Underestimating Impacts other possible risk events not on the One area that some project teams checklist that may impact the project. struggle with is underestimating the For example, the civil engineering impacts of risk events. This may be company may have an excellent risk because of the optimistic nature of checklist, but the company is awarded a project teamsespecially early in the new project and the road will cross an old project. When the impacts are abandoned landfill. This will undoubtedly understated the risk factor calculations create some unique risk events for this become skewed, the prioritization new project which are not currently on becomes flawed, and the entire risk the checklist. management process is threatened.
There are four possible consequences for any risk event: cost; schedule; functionality; and quality.
Each needs to be considered when deciding the risk impact value. However, its also important to prioritize these impacts based on whats most important for the project. In some cases it may be a specific end date that must be met because of a government regulation, or maybe a fixed budget that cannot be exceeded. If the most important driver for a project is meeting a completion date, then a risk event that will delay the completion date should be assigned a bigger impact value compared to a risk event that just impacts cost. Shown in table 1 is the impact table for a specific software development project [6]. For this project the budget was fixed and was the most important project driverthe budget could not be exceeded. Functionality and quality were secondary drivers, and if the budget was in jeopardy, functionality reductions could be considered. Schedule was least important, since the company had existing software in use and could continue using the software if the completion date slipped. Lets consider a sample risk event that has a cost impact score of 8, a schedule impact value of 0, and impact scores of 4 for functionality and quality. The obvious question is what impact value would the project team use for this risk event? Is it the average of the four impact scores, which would be 4? The answer is no! The correct method is using the highest value for the impact score. In this case that value is 8, which is the cost impact score. Be careful when discussing prioritization of project drivers with clients. The answer youll frequently here is that everything is equally important! Thats an easy answer but most often not really true. If a client has a project with a fixed end date that MUST be met, cost is probably not as much a consideration. It may still be importantbut not as much as schedule. A perfect example of this is the year 2000 projects (Y2K). There was a
42
definite end date of December 31, 1999, plan for the requirements document considered. Monte Carlo is a risk and for clients that was the project driver. deliverable, listing the specific people at simulation technique, and can help better each plant site needed for requirements understand the project risk events. With Error #5: Not Using 100% Probability definition, and the dates of the planned risk simulation, when discussing the During Project Planning meetings to define requirements. The probability and impact of a risk event, At the beginning of this article a risk actions also included a contingency plan instead of a single value being identified event was defined as having uncertainty, where if a person missed a meeting, the there is a most likely value, along with an so it may be confusing now stating that a project sponsor was immediately notified optimistic and pessimistic value. Risk risk event can have 100% probability and the sponsor contacted the simulation software can show the during project planning. 100% probability appropriate plant manager to resolve the frequency of possible outcomes and a means no uncertaintyits a fact. This is problem. By the time the project planning cumulative cost and/or schedule with not a conundrum because before phase was completed, the risk probability overrun probabilities. completion of project planning you will of having missed requirements dropped take actions to deal with some of the from 100% down to around 30%. Error #7: Calling Risk Response Planning project risks, and these actions should A final cautiondont overuse the Mitigation reduce the risk event probability below 100% probability approach. It should only A common mistake made by many 100%. Project execution cannot begin be used where the risk really will happen project organizations is calling risk with risk events that have a probability of unless actions are taken and the impacts response planning mitigation. That is 100% because they are facts and the are major. Otherwise, you run the risk of incorrect, since mitigation is just one risk project plan must be prepared to deal losing credibility. response technique and is specifically with these facts. used for dealing with risks that are However, giving a risk event a Error #6: Not Considering Sensitivity threats. Another term often used to probability of 100% during project with Risk Analysis describe risk response planning is risk planning will obviously call lots of The simplest technique for management plan, and thats also attention to that risk event provided it qualitative risk analysis is assigning a incorrect since that is part of the project also has a high impactand that will value for the probability and impact of plan and describes what the team will do either result in actions taken to reduce each risk and calculating the risk factor. to manage risks on the project. One final the probability and/or impact to a lower The risks are then prioritized and the term that should not be used to describe value, or changing the project plan to highest scores are the threats and risk response planning is risk eliminate the risk, or constructing the opportunities that should be actively assessment, which more correctly means project plan to deal with the 100% reality managed as part of risk response identifying and analyzing risksso the of the risk event. planning. This simplified approach recommendation is to avoid use of this An actual project example is a ignores the sensitivity around the term. customized software development probability and impact values. In reality, The product of risk management is project for use at company these are not fixed values, but are a range called the risk register and it includes the manufacturing locations around the of values. list of identified risk, the risk analysis world. Business was good and the On smaller projects qualitative risk results and the risk response plans manufacturing areas were extremely analysis is usually good enough. including contingency plans. The correct busy. The project team correctly realized However, on larger projects advanced risk term to use for describing what steps will a major risk event would be having analysis using quantitative techniques, be taken to deal with risks is, risk incomplete project requirements such as Monte Carlo analysis, should be response plan. resulting from an inability to get the manufacturing representatives to take time to meet and adequately explain their requirements. Incomplete requirements would probably result in a solution design that didnt meet all of the manufacturing plant needs. The project team gave the risk of incomplete requirements a probability of 100% and informed the project sponsor the project would fail unless this risk was dealt with during project planning. A risk response plan was developed and implemented for this risk. The actions taken included a comprehensive schedule and resource Table 1 Project Impact Table for a Sample Software Project
43
Table 2 Risk Register Example When risk mistake #1 was discussed, the commonly used risk responses for opportunities were described. Listed next are the common risk responses for threats [4]: Reduce (Also called Mitigation) This response seeks to reduce the probability and/or impact of a negative risk event. An example of reduce is offering a performance bonus to a key project resource to mitigate the potential of the person leaving the company before project completion. The performance bonus reduces the probability of the person leaving. Eliminate (Also called Avoidance) This response looks to eliminate the threat by taking actions to cause the risk to go away completely. This is usually done by eliminating the causes. When you avoid, the risk is gone! There is no need to do any contingency planning, because the risk no longer exists. An example of avoidance is a developer looking to build a new store decides not to build in a certain town because of a hostile town board which probably will hold up the zoning approval. TransferThis response shifts some or all of the threat to a third party. The risk doesnt go away, but the ownership and impact of the risk belongs to somebody else. An example of transference is using a firm fixed price contract with a vendor for the data migration work on a software project. This provides the project team with a known cost, and the threat of a cost overrun belongs to the vendor (along with the opportunity to complete the work for less money and make more profit). AcceptThis response is deciding not to do anything about the risk. This may be because no other actions are feasible, or the risk factor isnt significant. The response can be active acceptance, which means developing a contingency plan if the risk event occurs. An example is building a new store in upstate New York in January. It will snowso active acceptance is the best approach. You cant control the weather! more likely to happen for opportunities and less likely to happen for threats. Contingency plans are more specifically the actions that will be taken if and when the risk event occurs. Risk response planning should deal with both! Lets revisit the common risk responses for threats covered in mistake #7. The mitigation example was losing a key project resource because of a person leaving the company. The contingency plan for this example would be identifying a back-up resource (either internal or a contractor) that could step in and handle the work if the person does leave the company. For the acceptance example of building a store in upstate New York in winter weather, contingency planning could be adding some extra days in the schedule as an allowance for nonworking days because of snow, and/or including money in the budget for plastic sheeting and space heaters so work can continue even if snow occurs. One final comment is that not all risks need a contingency plan! The example used for transference was using a firm fixed price contract with a vendor for the data migration work on a software project. The risk impact now belongs to the vendor so the project team doesnt need to worry about a cost overrun. The example used for avoidance was a developer looking to build a new store deciding not to build in a certain town because of a hostile town board. Since the decision was made not to build in the town with a hostile town board, the risk
In summary, mitigation is just one risk response technique, and it is specifically used for threats. Do not call risk response planning mitigationthe correct term to use for describing what steps will be taken to deal with risks is risk response plan. Error #8: Not Considering Contingency Plans Along With Response Plans As mentioned with mistake #7, the correct term to use for describing what steps will be taken to deal with risks is risk response plan, but dont forget that should also include a contingency plan! To clarify the difference, the risk responses are actions taken before the risk event occurs to make the risk event
44
no longer exists. In both cases a For each risk event, the responsible contingency plan is not pertinent. team member should provide a brief status report on the risk response actions Error #9: Not Making Team Members taken and/or planned, along with Responsible for Specific Risks changes to the probability and/or impact One observation is that some project values based on actions taken. The managers are not effective at delegating project team should also discuss if any work, and end up taking responsibility risk triggers have occurred, and whether where other team members could be any new potential risk events have used. Part of risk response planning is surfaced since the last meeting. assigning ownership of each risk to an The key point is that risk individual. management is an on-going process over Assign each risk to a specific the entire life of the project! Watch for individual on the project team, and the certain phrases that appear in peoples project manager should minimize the conversation, such as there is a chance number of risks assigned to her/himself. that or maybe this would cause The assigned person should implement problems When you hear words like the risk response plans, monitor the risk that, ask for clarification, and it may and report on the risk status. Assign each become apparent that this is a new risk risk to the team member who has the event and should be added to the risk particular skills related to the risk. register. For example, a risk concerning the requirements elicitation work on a Conclusion software project should be assigned to a Many people using risk management business analyst, while a metallurgy risk use incorrect techniques in preparing and on a construction project should be implementing risk management plans. handled by an engineer. This article has covered the top ten errors people make in dealing with project risks Error #10: Not Making Risk Management and how these mistakes greatly reduce an On-going Process the value of risk management. Check how Unfortunately, some project teams you are doing risk management on your think of risk management as a task that is projects and see if you are making any of started and completed as part of project these common errors. You may be planning. The risk register is completed surprised! and then filed away and forgotten. This is the mistakea risk management review REFERENCES should be part of every project team 1. McKechnie, Jean (Editorial Staff meeting! Shown in table 2 is a sample Supervisor), Websters New risk register, which is a simple worksheet. Twentieth Century Dictionary,
2.
3.
4.
5.
6.
Second Edition, Simon & Schuster, Inc., New York, (1979): pg. 1565. A Guide to the Project Management Body of Knowledge, Fourth Edition, Project Management Institute, Newtown Square, PA, (2008): pg. 438. Lukas, Joseph A., It Works! Risk Management on an IS Project, Proceedings of the PMI Seminars and Symposium, Project Management Institute, Newtown Square, PA, (2002): pg. 2. Brady, David C., January 12, 2012, Risk Treatment AACE International Recommended Practice No. 63R-11 Public Review Draft, (Jan. 12, 2012): pgs 6-8. Hillson, David. Project Risks, Identifying Causes, Risks and Effects, PM Network, (Sept. 2000): pgs. 4851. Lukas, Joseph A. Risk Management in the Real World: A Look at an IS Project, Inside Project Management, Element Journals, Rochester, NY. Volume 1, Number 6, (2001): pg. 13.
ABOUT THE AUTHOR
Joseph A. Lukas is with PMCentersUSA. he can be contacted by sending e-mail to: joe.lukas@pmcentersusa.com
2012-2013 NOMINATING AND AWARDS COMMITTEES

Past President Michael Nosbisch will serve as chair of the 2012-13 Nominating Committee, which will include Ginette B. Basak P.Eng FAACE; Philips Tharakan Mulackal CCE EVP, Michael B. Pritchett CCE CEP, and Stephen P. Warhoe PE CCE CFCC FAACE. The Nominating Committee will also serve as the 2012-13 Awards Committee. The responsibilities of the Nominating Committee are outlined in the Association Bylaws as follows:
Section 2The Nominating Committee shall be responsible for: (a) selecting one or more nominees for each oce, (b) obtaining the consent of each nominee to serve if elected, and (c) securing biographical data of each nominee. The entire slate of candidates, complete with biographical data, shall be sent by the Nominating Committee to the Vice President-Administration no later than October 31st of each year. Section 3The Nominating Committees slate of candidates, complete with biographical data, shall be made available to the membership along with information regarding nominations by petition no later than November 15th of each year. Section 4Other nominations for the oce of Director or the oce of an Ocer of the Association may be made by petition signed by at least twenty (20) Members in good standing. The petitioners shall be responsible for (a) obtaining in writing the agreement of the nominee to serve if elected, (b) securing the biographical data of the nominee, (c) submitting the petitions, the agreement, and the biographical data to be received by the Vice PresidentAdministration no later than December 15th of each year. Each candidates name and biographical data shall be made available to the membership no later than December 31st of each year. To be eligible to vote, a members membership dues must be paid through Dec. 31, 2012. Members also are restricted to voting for regional director candidates within the region where the members primary section aliation is located, according to the AACE records as of Dec. 31, 2012. Electronic online voting will open on February 1 and concludes at 4 p.m. Eastern US Time Zone on March 15.
45
AACE INTERNATIONAL ONLINE STORE more online at www.aacei.org

Skills and Knowledge of Cost Engineering, 5th Edition, Revised
Scott J. Amos, Editor, 2007 This updated and expanded guide for fundamentals is an excellent choice for anyone interested in a concise reference to all aspects of the profession. The new 5th edition includes twenty-seven chapters on estimating, manufacturing and operating costs, scheduling, planning progress and cost control, and much more. This is a very useful book for those studying for the certification exam. 450 pages 1595-02zip - Download - US$50.00 member/US$80 nonmember Paper version available through our Amazon.com link
Cost Engineering
The international journal of cost estimation, cost/schedule control, project management, and total cost management. Subscriptions are accepted on an annual basis. An automatic benefit of AACE International membership, also available to nonmembers. 5060-07 - US$72.00 (US) - US$171.00 (other countries) Please add US$80.00 for airmail - US$61 electronic subscription
CCC/CCE Certification Study Guide, 3rd Edition

Michael B. Pritchett, CCE, Editor, 2006 The AACE International CCC/CCE Certification Study Guide provides an all-encompassing reference text to prepare for the exam. The CCC/CCE Certification Study Guide provides background information on how to become certified; gives those studying for the certification exam a single reference text that includes theory, worked problems with answers, references, and a full discussion of key topics; allows students to maximize their study time; and provides a concise overview of the fundamentals of cost and project management. 1825-36zip - Download - US$50.00 member/US$80 nonmember Paper version available through our Amazon.com link
Cost Engineers Notebook

This CD-ROM is an important reference for any project or cost professional. It includes data and procedures related to basic skills and knowledge that all cost engineers should possess, extensive material on capital and operating cost estimation, and papers in four subject areas: cost control, planning and scheduling, project management, and economic analysis and business planning. 4060-28zip - Download - US$65.00 member/US$105.00 nonmember
AACE International Recommended Practices

Cost Engineering Terminology; Cost Estimate Classification System; Estimate Preparation Costs in the Process Industries; Project Code of Accounts; Required Skills and Knowledge of a Cost Engineer; Roles and Duties of a Planning and Scheduling Engineer; Profitability Methods; plus many more. 4060-05zip - Download - US$70.00 member/US$110.00 nonmember
PSP Certification Study Guide, 1st Edition

Peter W. Griesmyer, Editor, 2008 This study guide is intended to assist you in your study and review of the overall topics as one step toward successful Planning and Scheduling Professional certification. The outline provides a listing of the terms you should know & topics for which you should have a good understanding of how to apply the concepts to solve problems. Each chapter also contains sample exercises, which test your knowledge of that chapter's concepts. Additional sample questions are provided in an appendix. 1820-38zip - Download - US$50.00 member/US$80.00 nonmember Paper version available through our Amazon.com link
The Total Cost Management Framework

John K. Hollmann, PE CCE, Editor, 2006 4060-20zip - Download - US$50.00 member/US$70.00 nonmember Paper Version available through Amazon.com
2012 AACE International Transactions

5220-12 zip Download - US$75.00 member - US$80.00 nonmember For CD-ROM version please contact AACE International Headquarters
EVP Certification Study Guide, 2nd Edition

Ken Cressman, CCC EVP and Gary C. Humphreys, Editors, 2009 This study guide is intended to assist you in your study and review of the overall topics as one step toward successful Earned Value Professional certification. The outline provides a listing of the terms you should know & topics for which you should have a good understanding of how to apply the concepts to solve problems. Each chapter also contains sample exercises, which test your knowledge of that chapter's concepts. 1820-40zip - Download - US$50.00 member/US$80.00 nonmember Paper Version available through our Amazon.com link
The The AACE AACE International International Professional Professional Practice Practice Guides Guides (PPGs) (PPGs)
Value; Earned Value Reporting; Applications of Earned Value Project Management; and more.
PPG #14: Portfolio and Program Management,

2nd Ed. Randy R. Rapp, PE CCE, Editor, 2007
Covers: Enterprise Management: General Imperatives and Concerns; Asset Requirements Elicitation and Analysis; Asset Planning and Investment Decision-Making; Asset Performance Assessment and Change Management; and Program Management.
PPG#6: Construction Cost Estimating, 2nd Ed.

Dr. Douglas D. Gransberg, PE CCE, And Carla Lopez del Puerto, CCC, Editors, 2006
Covers: Recommended Practices; Estimating Theory; Conceptual, Parametric, and Range Estimating; Estimating Factors and Indices; Estimating Material Costs and Quantity Surveying; Estimating Labor Costs; Estimating Equipment Costs; Subcontracting Costs; Estimating Overhead and Indirect Costs; Profit, Contingencies, and Mark-Ups; Estimating International Construction Costs; and more.
PPG #15: Life-Cycle Cost Analysis

Dr. Douglas D. Gransberg, PE CCE, and Carla Lopez del Puerto, CCC, Editors, 2007
Covers: Life-Cycle Cost Theory; Life-Cycle Cost Methods, Determining Discount Rate; Estimating Capital Cost of Design and Construction; Estimating Operating Costs; Estimating Salvage/Residual Value; Estimating Sustainability; Life-Cycle Cost Risk Analysis; Life -Cycle Cost Case Studies; Life-Cycle Cost Analysis in the International Context
(PPGs) are a series of reference s that consists of selected Cost Engineering articles, AACE International Transaction papers, and other previously published
documents to which AACE has rights.
PPG#7: Cost Engineering in the Utility Industries, 2nd Ed.

Dennis M. Thompson, Editor, 2007
Covers: Auditing; Cost Estimating; Cost Modeling; Cost/Schedule Control; Generation Power Plant; Natural Gas Industry; Nuclear Power Plant; Other Energy Related Topics; Planning and Scheduling; Project Management; Utility Rates; and Utility Property Valuation.
PPG #16: Cost Engineering in the Global Environment

Kul B. Uppal, PE, Editor, 2007
Covers: General Topics on International Projects; Applicable AACE International Recommended Practices; Cost Estimating Methodology; Risk and Contingency; and Miscellaneous Topics
Price per PPG: Download Member Price US$50.00 Download Non-Member Price US$70.00 Price for the PPG Package includes all 21 PPGs: Download Member Price US$874.00 Download Non-Member Price US$1223.00
PPG#1: Contracts and Claims, 4th Ed.
James G. Zack Jr., Editor, 2008
Covers: Contract Administration; Management of Construction Schedules; Schedule Control; Schedule Float Ownership; Cost Control; Management of Change; Cost Impacts; Productivity Impacts; Management and Analysis of Delay; Concurrent Delay Issues; Pricing of Delay; and more.
PPG#8: Contingency, 2nd Ed.

Kul B. Uppal, PE, Editor, 2005
Covers: General Topics On Contingency; Cost Estimating and Contingency; Risk Analysis and Contingency; and Other Related Topics.
PPG #17: Public Sector Estimating

Joseph L. Macaluso, CCC, Editor, 2007
Covers: Basis of Estimates; Labor Costs; Overhead and Profit; Soft Costs; Bid/Estimate Reconciliation; and Change Orders
PPG #10: Project Delivery Methods, 2nd Ed.

Dr. Douglas D. Gransberg, PE CCE, Tammy L. McCuen, and Keith Molenaar, Editors, 2008
Covers: Design-Bid-Build (DBB) DBB Estimating, DBB Scheduling, DBB Project Management; Construction Management (CM) CM Estimating, CM Scheduling, CM Project Management; Design-Build (DB) DB Estimating, DB Scheduling, DB Project Management; International Project Delivery; Constructability; and Partnering.
PPG #18: Green Building

Joseph L. Macaluso, CCC, Editor, 2008
Covers: Recognition of Affects and Economic Costs on the Environment; Formulating Ways of Addressing Green Building Strategies and Associated Economic Costs; Specific Green Building Strategies and Project Costs; Budgeting and Justifying the Cost of Sustainable Practices; Evaluating Competing Sustainable Strategies: Using Value Engineering; Evaluating Competing Sustainable Strategies: Other Techniques
PPG #11: Environmental Remediation & Decommissioning, 2nd Ed.

Richard A. Selg, CCE, Editor, 2009
Covers: Environmental Remediation Planning and Scheduling Methodology; Cost Estimating, Project Controls, Cost Modeling, and Reporting; Contingency Management, Risk Analysis, and Environmental Regulations; Benchmarking and Lessons Learned; Economics of Environmental and Waste Management; Cost-Effective Waste Minimization and Pollution Prevention; Design, Construction Practices, and Other Related Topics.
PPG #19: Leadership and Management of People

John J. Hannon, CEP, Editor, 2008
Covers: Leadership; Teams; Leadership Roles; Motivation; and Ethics.
PPG#2: Risk, 2nd Ed.

Keith D. Brienzo, PE, Editor, 2007
Covers: Dictionary; Capital Investments; Cash Flow; Competitive Bidding; Contingency Analysis; Contracts; Cost Engineering; Currency Rates; Decision Trees; Economic Analysis; Escalation; Human Factors; Manufacturing; Research & Development; Safety & Health; Schedule; Technological Risk; and Value Engineering.
PPG #20: Forensic Schedule Analysis

James G. Zack, Jr., CFCC, Editor, 2008
Covers: Recommended Practice No. 29R-03 Forensic Schedule Analysis; Synopsis of Recommended Practice; Basics of Schedule Delay Analysis; MIP-Observational Static Gross; MIP-Observational Static Periodic; MIP-Observational Dynamic Contemporaneous As-Is; MIP-Observational Dynamic Contemporaneous Split; MIP-Observational Dynamic Modified or Recreated; MIPModeled Additive Single Base; MIP-Modeled Additive Multiple Base; MIP-Modeled Subtractive Single Simulation; Non-CPM Schedule Delay Analysis Techniques; General Schedule Analysis Articles
PPG #12: Construction Project Controls

Dr. Douglas D. Gransberg, PE CCE, and James E. Koch, Editors, 2002
Covers: Introduction to Construction project Controls; Cost Control; Schedule Control; Quality Control; Document Control; Computer Applications; and International Project Controls
PPG#3: Cost Engineering in Aerospace and Aviation

Sarwar A. Samad, Editor, 1998
Covers: Aerospace and Aviation.
PPG#4: Planning and Scheduling, 2nd Ed.

Trevor X. Crawford, CCC, Editor, 2006
Covers: Planning; Schedule Development; Schedule Management/Control; and Classics.
PPG #13: Parametric and Conceptual Estimating, 2nd Ed.

Douglas W. Leo, CCC, Larry R. Dysert, CCC, and Bruce Elliott, CCC, Editor, 2004
Covers: Parametric/Conceptual Estimating; Classification; Methodology; Capacity Factoring; Process and Non-Process Industries; and Systems
PPG#21: Cost Engineering in the Process Industries

Kul B. Uppal, PE CEP, Editor, 2009
Covers: General Topics on Process Industries; Cost Estimating Methodology; Project Management; International Projects; Scheduling; Construction Activities; Risk Management; Project Controls; and Applicable AACE International Recommended Practices.
PPG#5: Earned Value, 2nd Ed.

Robert A. Marshall, Editor, 2007
Covers: Why Use Earned Value?; Basics of Earned Value; Cost/Schedule Control System Criteria; Actual Physical Percent Complete; Productivity and Earned
More AACE Publications at the Online Store - www.aacei.org
PROFESSIONAL SERVICES DIRECTORY
All AACE International Books are "Print-on-Demand" Products

AACE has reformatted each of its print products (i.e., the Skills and Knowledge of Cost Engineering; CCC/CCE Certication Study Guide; EVP Study Guide; PSP Study Guide; and the TCM Framework) to be produced by a subsidiary of Amazon.com. Each of these AACE publications became available online from Amazon.com in 2012. We maintain a link from the AACE website, www.aacei.org, to Amazon.com and will recommend additional products as well.
INDEX TO ADVERTISERS
Acumen, page 4 ARES Corporation, back cover Bechtel Corporation, page 3 D.R. McNatty and Associates, this page EcoSys, inside front cover
YOUR VISIBILITY
IN THE COST ENGINEERING JOURNAL
ADVERTISE
REACH the entire AACE International membership every month by placing an ad in the Cost Engineering journal.
PLACE your products/services in front of

over 50,000 users each month with a banner ad at our website, www.aacei.org.
Faithful+Gould, page 48 Infinitrac, this page
Management Technologies, this page Ron Winter Consulting, page 16 Sage Software, inside back cover Skire, Inc., page 16
For additional information about the listed advertisers or about advertising with us, please phone Garth Leech, 1.304.296.8444 x122, or e-mail him at gleech@aacei.org
EXHIBIT at the AACE International Annual

Meetings, and contact AACE International members face to face.
CONTACT
Garth Leech
phone 1.304.296.8444 fax 1.304.291.5728 e-mail gleech@aacei.org
OR GO ONLINE AT www.aacei.org
49
Contrary to rumor, there is no certied League of TCM Heroes. Individuals who have earned certications such as the Certied Cost Consultant (CCC)/Certied Cost Engineer (CCE), Certied Estimating Professional (CEP), Certied Forensic Claims Consultant (CFCC), Earned Value Professional (EVP), Planning & Scheduling Professional (PSP) and Certied Cost Technician (CCT) are:
Not able to leap tall buildings in a single bound Not faster than a speeding bullet
Not more powerful than a locomotive Not in possession of x-ray vision
But the AACE International Salary Survey did show that certicate holders earn an average of $5,000 per year more than non-certication holders with the same age and experience!
If you want to be in the same salary league then visit www.aacei.org for more information!
ARTICLE REPRINTS AND PERMISSIONS
COST ENGINEERING
Vol. 54, No.6/November/December 2012
Members of AACE International have access to free downloads of selected articles that are published with an AACE International reference number. These articles are available at the online Virtual Library at www.aacei.org. Electronic files of each months technical articles are posted and members can download an Adobe Acrobat (PDF) version of any of the technical articles for free. You can search for articles using the reference numbers listed in the Cost Engineering journal. Non-members can subscribe to the AACE Virtual Library at an annual cost of US $100.00. AACE International no longer offers reprints of individual articles.
Pages 5-15
Integrated Cost-Schedule Risk Analysis

This article was presented as RISK.1043 at the 2012 AACE International Annual Meeting in San Antonio, Texas.
Article Reference Number - 29188 Pages 17-27
Estimate Accuracy: Dealing With Reality

This article was first presented as RISK.1027 at the 2012 AACE International Annual Meeting in San Antonio, Texas.
Article Reference Number - 29189
TO ORDER
Contact: AACE International Publications Sales at pubsales@aacei.org
Pages 28-38
Quantifying Estimate Accuracy and Precision for the Process Industries: A Review of Industry Data
Photocopying Prices:
For permission to photocopy individual articles for personal use, or to request permission for bulk photocopying, please contact the Copyright Clearance Center at 978.750.8400, and pay the required photocopying fees. For any other use or reprint requests, please e-mail: editor@aacei.org.
Article Reference Number - 29190 Pages 39-45
Common Errors in Dealing With Project Risk

Contact Us
AACE International 1265 Suncrest Towne Centre Dr Morgantown, WV 26505-1876 USA Phone: 304.296.8444 Fax: 304.291.5728
Article Reference Number - 29191
For Information Concerning Other Reuse Requests

If you are seeking permission to quote or translate into another language any material from any issue of the Cost Engineering journal, please contact our Managing Editor, Marvin Gelhausen at mgelhausen@aacei.org
51
CALENDAR OF EVENTS
NOVEMBER 2012
14-15 AACE Internationals
International Total Cost Manage ment Conference, AACE International Hyatt Regency Dubai and Galleria Dubai, United Arab Emirates Contact: phone 1-800-858-COST fax (304) 291-5728 info@aacei.org www.aacei.org
4 The Federal Design &

Construction Outlook Conference, AEC Science & Technology, LLC Washington D.C. Convention Center Washington D.C. Contact: phone 1.508.790.4751 1.800.996.3863 support@aecst.org
27-30 AACE International

Education Seminars, AACE International Marriott Wardman Park Washington, D.C. Contact: phone 1800858COST fax (304) 2915728 info@aacei.org www.aacei.org
APRIL 2013
17-19 Cost Engineering Event 2013
- Total Cost Management; A World of Opportunities, Cost Engineering Consultancy Ara Hotel Zwijndrecht, Netherlands Contact: +31 (0)78 620 0910 BDames@costengineering.eu www.costengineering.eu
30-July 3 AACE International

Education Seminars, AACE International Marriott Wardman Park Washington, D.C. Contact: phone 1800858COST fax (304) 2915728 info@aacei.org www.aacei.org
24 2012 ICEC Region 2 (Europe/Near East) Meeting, The International Cost Engineering Council (ICEC) FAST/AICE Premises Milan, Italy
Contact: www.icoste.org/category/meetings
JUNE 2013
28 - Dec 2 The First Australasia
and South East Asia Structural Engineering and Construction Conference ASEA-SEC-1), The ISEC Society Curtin University Perth, Australia Contact: www.isec-society.org
18-23 ISEC-7,
The ISEC Society Campus Center of the University of Hawaii at Manoa Honolulu, HI Contact: www.isec-society.org/ISEC_07/
Please submit items for future calendar listings at least 60 days in advance of desired publication.
AACE International, 1265 Suncrest Towne Centre Dr, Morgantown, WV 26505-1876 USA phone: 304-296-8444 fax: 304-291-5728 e-mail: editor@aacei.org website: www.aacei.org
DECEMBER 2012
3-7 National BIM Conference,
AEC Science & Technology, LLC Washington D.C. Convention Center Washington D.C. Contact: phone 1.508.790.4751 1.800.996.3863 support@aecst.org
52
JOB SITE, MEET INSIGHT.

Learn about Sage solutions for estimating and construction. Visit our website at SageCRE.com/bidready to get the facts.
Make it easier for your estimators to bid on and win more projects with our business-aware, peopleconnected applications for specialty and general contractors. Over 40,000 construction and real estate rms, including over half of the ENR 400, rely on Sage solutions to manage more than 400,000 jobs and 5.7 million subcontracts every year. Join them and gain the insight to get job-ready today.
2012 Sage Software, Inc. and its aliated entities. All rights reserved.
Sage 50 Sage 100 Contractor Sage 300 Construction and Real Estate Sage Construction Anywhere Sage Estimating Enabling Sage contractors of any size to achieve their ambitions.

Ce12 11

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ce12 11

Uploaded by

Copyright:

Available Formats

THE JOURNAL OF AACE INTERNATIONAL THE AUTHORITY FOR TOTAL COST MANAGEMENT

DEALING WITH REALITY

A REVIEW OF INDUSTRY DATA

DEALING WITH PROJECT RISK

17 Estimate Accuracy: Dealing With Reality

39 Common Errors in Dealing With Project Risk

COST ENGINEERING NOVEMBER/DECEMBER 2012

AACE INTERNATIONAL BOARD OF DIRECTORS

Managing Editor Art Director Graphic Designer Advertising Sales

AACE International - The Authority for Total Cost Management

Policy Concerning Published Columns, Features, and Articles

COST ENGINEERING NOVEMBER/DECEMBER 2012

Discover more at bechtel.com/careers

The ambition to make history.

Risk Analysis doesnt have to be

Powerful analytics, consistently accurate results.

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

Table 2 Resources for Example Construction Project

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

Table 5 Assigning Risks to Activities

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

Table 8 Cost, Schedule and Interaction Effects

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

Table 9 Schedule and Cost Impact of a 40% Probable Commissioning Risk

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

ABOUT THE AUTHORS

COST ENGINEERING NOVEMBER/DECEMBER 2012

reviewing your P6 schedule and reviewing your P6 XER file? Everything

COST ENGINEERING NOVEMBER/DECEMBER 2012

Dealing With Reality

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

Table 2 Empirical Estimate Accuracy Studies (Progression by Level of Scope Definition)

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

Figure 2 Actual Cost/Appropriation Estimate For Small Projects

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012

ABOUT THE AUTHOR

COST ENGINEERING NOVEMBER/DECEMBER 2012

Quantifying Estimate Accuracy and Precision for the Process Industries:

A Review of Industry Data

COST ENGINEERING NOVEMBER/DECEMBER 2012

Figure 1 Estimate Accuracy Presented as a Probability Density Function

Figure 2 Outcome Matrix for Accuracy and Precision

COST ENGINEERING NOVEMBER/DECEMBER 2012

Figure 3 Summary of AACE International Estimate Classification System

COST ENGINEERING NOVEMBER/DECEMBER 2012

Figure 4 Stage Gated Process for Project Development

COST ENGINEERING NOVEMBER/DECEMBER 2012

Figure 6 Industry Database of Projects by Project Type and Industry Sector

COST ENGINEERING NOVEMBER/DECEMBER 2012

COST ENGINEERING NOVEMBER/DECEMBER 2012