QUELCE - Quantifying Uncertainty For DoD Acquisition Programs 2014

QUELCE: Quantifying Uncertainty for DoD
Acquisition Programs
Jim McCurley
Senior Member of the Technical Staff
Software Engineering Institute
Carnegie Mellon University
October 2014
QUELCE - October 2014

2014 Carnegie Mellon University
Copyright 2014 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA872105-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally
funded research and development center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the United States Department of Defense.
References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer,
or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie
Mellon University or its Software Engineering Institute.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
MATERIAL IS FURNISHED ON AN AS-IS BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO
WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING,
BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY,
EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON
UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM
PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material was prepared for the exclusive use of JMS Program at AF SMC and may not be used for any other
purpose without the written consent of permission@sei.cmu.edu.
DM-0001364
QUELCE October 2014

Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE)
The Uncertainty Problem: DoD Context

..programs that breach appear to have the strongest relationship with three factors: the
total dollar size of a project, the quantity change cost category, and the estimating cost
changes.
...
Much of the data collected now does not help decision-makers determine why a breach or
unit-cost-growth has occurred or what programmatic changes would improve performance.
...
The available information makes it difficult to assert any conclusions definitively because
all factors appear interrelated, which means that an unconsidered exogenous variable may
be confounding all conclusions.
The Effect Of The NunnMcCurdy Amendment On Unitcost- Growth Of Defense Acquisition Projects, By Jacques S. Gansler,
William Lucyshyn, and Adam Spiers , Univ of MD Center for Public Policy and Private Enterprise, July 2010
_____________________
Unrealistic estimates are caused by the invalidity of major cost-estimating assumptions,

not methodological errors... PARCA deems an estimate to be unrealistic if it is based on
an uncertain assumption. Such assumptions might concern technical issues, related
programs, organizational relationships, threats, policy matters or the industrial base.
Inside the Pentagon, Vol. 27, No. 46, November 17, 2011
QUELCE October 2014
QUELCE addresses the assumptions and decisions made

during system development.
Account for change and uncertainty during the DoD acquisition life cycle.
Synthesis of Expert Elicitation, Dependency Structure Matrix
techniques, Bayesian Belief Network (BBN) modeling, Scenario
Planning and Monte Carlo simulation into a method that models
uncertainties among program change drivers as inputs to cost
models
Use of domain expert judgment, program artifacts, and and data-based
inputs
DoD domain-specific method for improving expert judgment regarding
uncertainty in program change drivers, their relationships, and impacts on
cost drivers.
Expert judgment is optimistic and over-confident
Expert calibration training improves estimates.
Repository of program change histories inform experts.
QUELCE October 2014
Early Estimation Lacks Dataand Change is

Complex
Information available at the start is not in a form typically used in
preparing an estimate.
Program does not yet have detailed scope and specifications.
Can we model the uncertainties not captured by the estimate?
Visual depiction of influential relationships, scenarios and outputs to
aid team-based model development, and explicit description and
documentation underlying an estimate.
Interdependencies cause problems to cascade.
When a project goes off the rails there is often a cascade of problems
before the magnitude of the problem becomes clear.
Scenario modeling and simulation makes impact of changes visible.
QUELCE October 2014

Changes in DODs 2011 Portfolio of Major Defense

Acquisition Programs over Time
Source: DEFENSE ACQUISITIONS: Assessments of Selected Weapon Programs,

GAO-12-400SP, March 2012
QUELCE October 2014

Estimates are a Root Cause of Cost Overruns

DoD often underestimates development costsdue in part to a lack of knowledge and
optimistic assumptions about requirements and critical technologies.
- A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes, GAO Report to the Committee on
Armed Services, U.S. Senate s, U.S. Senate, July, 2008 GAO-08-619
40% of accumulated
cost overruns
Sources Cost and time overruns for Major Defense Acquisition Programs, Joachim Hofbauer Gregory Sanders Jesse Ellman David Morrow,
Center for Strategic and International Studies, April 2011
QUELCE October 2014
DoD Programs Re-visit their Estimate Many Times
DoD 5000 with Naval POPS Gate Reviews

QUELCE October 2014
Motivation for QUELCE

Known-Unknowns
Each decision and especially the assumptions have the potential to be wrong
or at least subject to change. Thus each represents a known-unknown.
Capture the probability of change and the change effects

We can get some data about probability of change from past experience and
reference data.
Adjusting an estimate by probability of change

We should be able to estimate the probability of change and cascade effects
to develop a formal probabilistic method to add a range to the estimate.
Include assessment of risk

The scenarios devised to assess change effects tell us which risk-based
scenarios have the potential to require formal mitigation strategies, because
they will create risks that exceed the boundaries set by contingency budgets.
QUELCE October 2014

Uncertainty in Product Development

Assumption:
We can identify most sources of uncertainty and causal effects of change in
product development projects.
Challenge:
Can this knowledge (awareness) be exploited in developing better estimates
and plans for the product development work?
Examples:
Technology introduction: Introduction of 1-10mm wavelength radio frequency
allows much higher volume traffic but development of a modem is more
complicated.
Change in scope: Antenna that fits a 747 will not fit in fighter jet.
Change in interfacing external system
Change in sponsorship: Mission capability of proposed system fits a need for
a new user community provided that certain performance can be improved.
QUELCE October 2014

10
MORS*: Affordability Analysis Workshop

*Military Operations Research Society
Workshop recommends an approach similar to QUELCE
Drivers of un-affordable solutions
Mistakes and poor decisions
Focus on development costs and minimizing sustainment investment
Limiting (performance budgets) or failing to understand capability trade space
Recommendations
Use a distribution when estimating (not point estimates)

Document ground rules and assumptions for estimates
Include a sensitivity analysis for scenarios of change
Identify drivers of uncertainty using past program performance history
Use analytic methods such as Monte Carlo and probability theory
*From Affordability Analysis Workshop Report Oct 2012

QUELCE October 2014
11
Challenges to Implementation
Calibrating subject matter experts for probability judgment
Subject matter experts are (almost always) over-confident.
Reduce scale and complexity of the decision space

Change space is very large.
Development is often iterative so the same item may be a multiple driver of
change.
Multiple cost estimation relationships

Many inputs, most of them are not directly represented in the decision space.
Multiple scenarios of risk

Experts must reason about potential changes with simultaneous drivers.
Access to subject matter experts and other stakeholders is required.

Estimation is not simply a cost concern!
A facilitated method is most valuable.
QUELCE October 2014
12
QUELCE Inserts New Activity in Information Flows

Calibrated Expert Judgment
Information from Analogous Programs/Systems
Proposed Material Solution & Analysis of Alternatives
Program Execution Change Drivers

(with Driver States & Probabilities)
System Characteristics
Trade-offs
KPP selection
Systems Design
Sustainment issues
...
Operational Capability
Trade-offs
Mission / CONOPS
Capability Based Analysis
...
Technology Development
Strategy
Production Quantity
Acquisition Mgt
Scope definition/responsibility
Contract Award
Plans, Specifications, Assessments
Probabilistic
Modeling (BBN)
& Monte Carlo
Simulation
Cost Estimates
analogy
parametric
engineering
CERs
Program Execution
Scenarios Evaluated
QUELCE October 2014

13
Overview of the QUELCE Method

Reduce Complexity
2
2
1
2
2
1
Manning at program office
Prog Mgt Structure
Acquisition Management
Funding Schedule
Functional Solution Criteria (measure)
Scope Definition
Functional Measures
Systems Design
Interoperability
Building Technical Capability & Capacity (CBA)
Closing Technical Gaps (CBA)
Advocacy Change
Capability Definition
Project Social / Dev Env
Queries of Historical
MDAP Experience and
Context
3
3
3
3
2
0
3
2
2
1
1
3
2
1
1
1
2
2
1
1
2
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Drivers
Program Mgt - Contractor Relations
Mission / CONOPS
Change in Strategic Vision
Advocacy Change
nteroperability
Systems Design
Mission / CONOPS
Effects
4. Apply Uncertainty
to Cost Formula
Inputs for Basis and
Scenarios
4. Cost Factor Distributions by

Scenario of Change
Change Drivers - Cause & Effects Matrix
Causes
QUELCE Change
Repository
Build BBN Model
2. Dependency Structure Matrix
DAES,
etc.
Scope
3. Assign
Conditional
Probabilities
2. Cause and Effect

Analysis.
Interdependency
1. Elicit
Change
Drivers and
Alternatives
2
2
1
2
2
1
1
1
1
XL
VL
6.20
5.07
7.07
5.48
7.80
4.96
4.05
5.65
4.38
6.24
0.49
0.60
2.12
1.59
1.43
1.62
1.33
1.30
1.43
0.83
0.95
0.87
1.26
1.12
1.10
1.14
PREC
FLEX
RESL
TEAM
PMAT
RCPX
RUSE
PDIF
PERS
PREX
FCIL
SCED
Stable
Users added
Additional
(foreign)
customer
defined
New condition
New mission
Capability
Definition
Stable
Addition
Funding
Schedule
Established
Funding delays tie up

resources {e.g.
operational test}
Stable
Joint service program

loses particpant
Advocacy
Change
Closing
Technical
Gaps (CBA)
Legend:
Selected Trade
studies are
sufficient
Technology does not

achieve satisfactory
performance
Technology is
too expensive
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
Complexity
Reduction
2.72
1.24
2.61
0.50
0.62
0.62
Technology not
New technology not
performing as
testing well
expected
~~~~
<X>
<X>
<X>
<X>
<X>
X
X
X
<X>
<X>
<X>
<X>
5. Monte Carlo with

Cost Estimation
Tools (e.g.,
COCOMO, SEERSEM
3. BBN Model
~~~~
Product Project
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
FFRDC ceiling Funding change for Funding spread
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
Senator did not Change in senior
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
XH
0.00
0.00
0.00
0.00
0.00
0
1
1. Driver State Matrix

Change Driver Nominal State
N
H
VH
Scale Factors
3.72 2.48 1.24
3.04 2.03 1.01
4.24 2.83 1.41
3.29 2.19 1.10
4.68 3.12 1.56
Effort Multipliers
1.00 1.33 1.91
1.00 1.07 1.15
1.00 1.29 1.81
1.00 0.83 0.63
1.00 0.87 0.74
1.00 0.87 0.73
1.00 1.00 1.00
~~~~
Modeling
Uncertainty
QUELCE October 2014
14
Use of Expert Judgment
Broad and diverse literature on eliciting expert judgment exists, especially in Psychology
and Statistics (see Selected References).
In this context, a great deal of information is generated by experts to propose a major

new system but only a small portion of that information is utilized in the cost estimation
process.
Estimates for budgeting, including lifecycle costs, are initially made when the system is
only a concept.
The minimum life cycle for major systems is several years and often spans decades.
Contextual uncertainties, particularly assumptions which can change over time, are not
captured by cost models.
Methods exist to help sharpen expert judgment, which also allow us to generate data to
incorporate the uncertainty associated with that judgment.
[expert] judgments of probability, however elicited, are just that judgments ... made in
response to the facilitators questions, not pre-formed quantifications of pre-analyzed
beliefs. - Uncertain Judgements: Eliciting Expert Probabilities, Anthony OHagan, et.al.,
Wiley, 2006.
QUELCE October 2014
15
Experts Tend to Be Over-Confident

Most people are significantly overconfident
about their estimates, especially educated
professionals
QUELCE October 2014
16
16
Reference Points to Help Calibrate Judgment for Estimation

Solution
Step 2: Iterate
through a series
of domain
specific tests
Step 3: Feedback on
test performance
Step 1: Virtual
training using
reference
points
Outcome: Expert
renders calibrated
estimate of size
DoD Domain-Specific
reference points
1)
Size of ground combat vehicle

targeting feature xyz in 2002
consisted of 25 KSLOC Ada
2)
Size of Army artillery firing

capability feature abc in 2007
consisted of 18 KSLOC C++
3)
Un-Calibrated
Calibrated
Calibrated = more
realistic size and
wider range to
reflect true expert
uncertainty
Estimate of SW Size
Used with permission from Douglas Hubbard Copyright HDR 2008 dwhubbard@hubbardresearch.com
QUELCE October 2014
17
Calibration Training of Experts

A series of training exercises
Typically day of 3 or 4 modules in sequence
Web-enabled version available
Each exercise includes:

A battery of factual questions
Most likely value; upper and lower bounds within which people are 90
percent certain the correct answer lies
True false questions where people provide their confidence in their
answers
Brief reviews of the correct answers
Group discussions of why the participants answered as they did
Guidance with heuristics about ways to explicitly consider
interdependencies among related factors ... that might affect the basis of
ones best judgments under uncertain circumstances
Group estimation experiments to be conducted this coming year.
QUELCE October 2014
18
Training Leads to Better Recognition of

Uncertainty
Generic Tests
Domain Specific
Tests
QUELCE October 2014

19
Experts Improved with Training
Test 2: Accurate & imprecise
Test 1: Inaccurate & imprecise
Test 3: Accurate & Precise
QUELCE October 2014

20
1: Experts Identify Change Drivers and States

1. Identify
Change
Drivers &
States
2.Reduce complexity
of Cause and Effect
relationships via
matrix techniques

Stable
Users added
As defined
New condition
Capability
Definition
Stable
Addition
Funding
Schedule
Established

resources {e.g.
operational test}
Stable

loses particpant
Selected Trade
studies are
sufficient
Technology does not

performance
Scope
Definition
Mission /
CONOPS
Advocacy
Change
Closing
Technical
Gaps (CBA)
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
3. Assign
Conditional
Probabilities to
BBN Model
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New mission
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Additional
(foreign)
customer
Technology is
too expensive
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
Technology not
New technology not
performing as
testing well
expected
~~~~
~~~~
QUELCE October 2014

21
Example:
Expert Data
From:
Obligations & Expenditures:
An Investigation into the Factors that Affect
OSD Goals, DoD Approved Survey:
Report Control Symbol DD-AT&L(OT)2513
Rob Tremaine, Donna Seligman, Shandy
Arwood, John Higbee
Briefing Presented to:
Honorable Katrina McFarland
05 Feb 2013
QUELCE October 2014

22
Examples of Expert data on frequency of change,

QUELCE October 2014
23
Addressing the Complexity

Experts specify the concepts we need to address, then judge the conceptual
interdependencies. We elicit likelihood ratings of cause and effect for change,
which are used to construct the Bayesian Belief Network and to assign
conditional probabilities.
Factor reduction
Sums of strength and error calculations help us to identify factors that
contribute little to the analysis. Experts decide on appropriateness.
Identify cycles
Cycles requires some detailed analysis or discussion with experts. The tool
identifies the cycles (typically 2-4) so that a manageable number remain. If
retained, the cycles can be recast as different variables.
In the end, we capture the experts judgments as to the likelihood of change and
the cascading consequences.
QUELCE October 2014

24
2: Reduce Complexity via Dependency Structure Matrix

1. Identify
Change
Drivers &
States
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
3. Assign
Conditional
Probabilities to
BBN Model
2. Reduce complexity
of Cause and Effect
relationships via
matrix techniques
5. Monte Carlo
Simulation to
Compute Cost
Distribution
2
2
1
2
1
2
2
1
1
1
1
1
1
1
3
2
1
1
1
1
1
0
1
2
1
2
2
2
1
1
2
2
1
2
1
1
2
1
1
2
2
1
2
2
2
1
1
1
1
2
1
1
1
1
2
1
1
1
1
2
2
2
1
1
1
1
3
3
1
2
2
1
1
1
1
2
1
1
1
1
2
2
2
Indicates remaining cycles that must be addressed

0
0
6
0
4
1
1
1
9
4
5
4
12
4
8
1
7
2
7
0
13
3
4
1
10
3
15
2
18
2
7
3
7
1
8
0
8
1
14
0
17
0
17
0
15
0
12
0
9
0
10
0
13
0
11
0
20
0
19
0
QUELCE October 2014

5
0
5
0
17
0
6
29
16
6
34
27
29
21
33
16
5
10
5
19
12
14
6
5
6
10
7
7
4
0
0
2
2
0
0
0
2
0
0
0
Number right of diagonal
Total
1
1
Product Challenge
2
2
Project Challenge
2
2
2
1
2
Size
1
1
1
3
2
Contractor Performance
1
1
1
1
1
2
0
Test & Evaluation
1
1
1
1
1
2
1
Cost Estimate
1
2
Industry Company Assessment
1
2
1
3
1
1
1
1
Data Ownership
2
2
2
1
2
1
Production Quantity
1
2
Contract Award
2
1
2
2
Sustainment Issues
2
2
2
1
1
1
2
PO Process Performance
1
1
1
Information sharing
2
0
2
2
3
Supply Chain Vulnerabilities
2
2
1
0
1
Standards/Certifications
2
2
1
0
1
3
3
Scope Responsibility
Prog Mgt Structure
Funding Schedule
Scope Definition
Functional Measures
Interdependency
Systems Design
Interoperability
Advocacy Change
3
3
Mission / CONOPS
Advocacy Change
Interoperability
Systems Design
Interdependency
Functional Measures
Scope Definition
Funding Schedule
Prog Mgt Structure
Scope Responsibility
Standards/Certifications
Supply Chain Vulnerabilities
Information sharing
PO Process Performance
Sustainment Issues
Contract Award
Production Quantity
Data Ownership
Industry Company Assessment
Cost Estimate
Test & Evaluation
Contractor Performance
Size
Project Challenge
Product Challenge
Totals
0
Below diagonal
0
Causes
Mission / CONOPS
Effects
Change Drivers - Cause & Effects Matrix
0
0
0
0
0
0
1
3
5
0
0
1
0
2
2
2
1
2
5
2
4
3
0
0
0
0
0
0
0
0
0
0
0
0
0
25
2: Reduce Complexity via Dependency Structure Matrix
SEI DSM Tool

Control Panel
All operations are done via the
DSM tools control panel.
Excel is used as a front end to
systematically bring in your
data, build a matrix, and
provide a selection of options
for matrix manipulation.
QUELCE October 2014

26
Reduction of the complexity of the network from

57 to 29 Program Change Drivers
QUELCE October 2014

27
DSM -> BBN

The DSM matrix captured the experts change probabilities for
one change driver affecting another change driver (all possible
pairings).
The BBN models the probabilistic relationships for the nominal
(expected) state, but experts then adjust the probabilities to
construct a set of critical scenarios to provide the basis for
program assessment based on cascading change.
For each scenario, the BBN produces probability distributions for
the output nodes which will then be used to assign probability
distributions to the input factors of the cost estimation models.
QUELCE October 2014

28
3: Assign Conditional Probabilities to BBN Model
QUELCE October 2014

29

1. Identify
Change
Drivers &
States
2.Reduce complexity
of Cause and Effect
relationships via
matrix techniques
3. Assign
Conditional
Probabilities to
BBN Model
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Truth table method
Capability Definition is
affected by CONOPS and
Strategic Vision
QUELCE October 2014

30

1. Identify
Change
Drivers &
States
2.Reduce complexity
of Cause and Effect
relationships via
matrix techniques
3. Assign
Conditional
Probabilities to
BBN Model
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Equation
method
a standard deviation of 0.08828 (8.9%)
Derived from
calibration
exercise
Engineering Change Proposal schedule delay = .56 + .0323 (waveform) + .0323 (Program Planning) + 0323 (Acquisition
Contracting) +.0215 (technology in motion) + .0323 (Contractor Program Mgt)
QUELCE October 2014

31
QUELCE October 2014

32
4: Calculate Cost Factor Distributions for Program

Execution Scenarios
1. Identify
Change
Drivers &
States
2. Reduce Cause and Effect

Relationships via
Dependency Structure
Matrix techniques
3. Assign
Conditional
Probabilities to
BBN Model
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
5. Monte Carlo
Simulation to
Compute Cost
Distribution
An example scenario
with 4 drivers in
nominal state
QUELCE October 2014

33
Example: BBN
Connection to the
COCOMO
estimation model.
Drivers
XL
VL
6.20
5.07
7.07
5.48
7.80
4.96
4.05
5.65
4.38
6.24
0.49
0.60
2.12
1.59
1.43
1.62
1.33
1.30
1.43
0.83
0.95
0.87
1.26
1.12
1.10
1.14
PREC
FLEX
RESL
TEAM
PMAT
RCPX
RUSE
PDIF
PERS
PREX
FCIL
SCED
N
H
VH
Scale Factors
3.72 2.48 1.24
3.04 2.03 1.01
4.24 2.83 1.41
3.29 2.19 1.10
4.68 3.12 1.56
Effort Multipliers
1.00 1.33 1.91
1.00 1.07 1.15
1.00 1.29 1.81
1.00 0.83 0.63
1.00 0.87 0.74
1.00 0.87 0.73
1.00 1.00 1.00
XH
0.00
0.00
0.00
0.00
0.00
2.72
1.24
2.61
0.50
0.62
0.62
QUELCE October 2014

Product Project
<X>
<X>
<X>
<X>
<X>
X
X
X
<X>
<X>
<X>
<X>
34
Connecting BBNs to Cost Estimation Models

1. Identify
Change
Drivers &
States
2.Reduce complexity
of Cause and Effect
relationships via
matrix techniques
Understand
and analyze cost
model input factors
COCOMO Parameter
Scale Factors
PREC
FLEX
RESL
TEAM
PMAT
Effort Multipliers
PERS
RCPX
PDIF
PREX
FCIL
RUSE
SCED
3. Assign
Conditional
Probabilities to
BBN Model
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
Group similar input factors

based on empirical analysis in
task 3.
Product Challenge factors (1=low5=high)

COCOMO Parameter
XL VL L N H VH EH
Scale Factors
PREC
1 3 5
FLEX
1 2 3 5
RESL
1 2 3 4 5
Effort Multipliers
RCPX
1 2 3 4 5
1 5
PDIF
RUSE
1 3 5
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Use empirical analysis

from Repository as basis
to map scale
(XL EH) of original
cost model input factors
to scale (15) of BBN
output factors
Project Challenge factors (1=low5=high)

COCOMO Parameter
XL VL L N H VH EH
Scale Factors
TEAM
1 3 5
PMAT
1 2 3 4 5
Effort Multipliers
PERS
1 3 5
PREX
1 2 3 4 5
FCIL
1 3 5
SCED
1 3 5
QUELCE October 2014

35
5: Monte Carlo Simulation to Compute Cost

Distribution
1. Identify
Change
Drivers &
States
2.Reduce complexity
of Cause and Effect
relationships via
matrix techniques
3. Assign
Conditional
Probabilities to
BBN Model
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Monte Carlo simulation for program change factor

distributions uses uncertainty on the input side to determine
the cost estimate distribution
BBN Outputs
4
Mapped
COCOMO
value
QUELCE October 2014
36
Summary
QUELCE includes the effects of uncertainty in the resulting estimate by:
Making visible the quantified uncertainties that exist in basic assumptions.
Calculating uncertainty of the input factors to the model rather than adjusting
the output factors.
Using scenario planning to calculate how specific changes might affect
outcomes.
The method utilizes subjective and objective data as input
Expert judgments are documented and made explicit.

Information typically not used for estimation purposes can be leveraged.
The method explicitly includes factors that have been documented as

sources of program failure in the past but are not typically captured by
cost models.
Visibility of potential change drivers to program management enables
quicker mitigation of emerging problems during the system lifecycle;
impacts can be quickly calculated.
QUELCE October 2014
37
Selected References
Air Force Cost Risk and Uncertainty Metrics Manual (CRUAMM) 2013
https://www.ncca.navy.mil/tools/csruh/CRUAMM_Printable_Version_16Nov2011_Edited_for_T
he_Joint_CSRUH_05Apr2013.pdf
Design structure matrix methods and applications, Steven D. Eppinger and Tyson R.
Browning, The MIT Press May 2012.
Enhanced Scenario-Based Method for Cost Risk Analysis: Theory, Application, and
Implementation, Paul R. Garvey, Brian Flynn, Peter Braxton, Richard Lee, Journal of Cost
Analysis and Parametrics, 5:98142, 2012
GAO Cost Estimating and Assessment Guide: http://www.gao.gov/assets/80/77175.pdf
Investigation into Risk and Uncertainty: Identifying Coefficient of Variation Benchmarks for Air
Force ACAT I Programs, Shaun T. Carney, Captain, USAF, AFIT-ENV-13-M-05, Air Force
Institute Of Technology, Wright-Patterson Air Force Base, Ohio, Mar 2013.
http://www.dtic.mil/dtic/tr/fulltext/u2/a579314.pdf
Joint Cost Schedule Risk and Uncertainty Handbook
https://www.ncca.navy.mil/tools/csruh/CSRUH_Printable_Version_16Jul2013.pdf
QUELCE October 2014

38
Selected References
Principles of Uncertainty, Joseph B. Kadane Chapman & Hall/CRC Texts in Statistical Science,
2011
Risk Assessment and Decision Analysis with Bayesian Networks, Norman Fenton, Martin Neil,
CRC Press, Nov 7, 2012
Software Development Cost Estimating Handbook Volume I, Naval Center for Cost Analysis
and Air Force Cost Analysis Agency, Sept 2008 https://acc.dau.mil/adl/enUS/323892/file/46968/SW%20Cost%20Est%20Manual%20Vol%20I%20rev%2010.pdf
Statistical Methods for Eliciting Probability Distributions, Paul H Garthwaite, Joseph B
Kadane & Anthony O'Hagan, J American Statistical Association
Volume 100, Issue 470, 2005 pp. 680-701.
The Correct Use of Subject Matter Experts in Cost Risk Analysis, Coleman, Richard L. ;
Braxton, Peter J. ; Druker, Eric R.,, 7th Annual Acquisition Research Symposium, May 2010
http://www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA530603
The Design Structure Matrix (DSM), www.dsmweb.org
Uncertain Judgments: Eliciting Expert Probabilities, Anthony OHagan, et.al., Wiley, 2006.
QUELCE October 2014
39
Extra
QUELCE October 2014

40
Building a Repository of Change
Two challenges
Collect the history and classify it
Assist experts to search and identify potential change
Collect and classify:

Initiate taxonomy from Naval POPS Guidebook v2
Expand to cover more detail
Assist the experts

Analyze group effects of calibration (probabilities)
Develop search tools so experts can identify possible
assumptions and decisions
Coded 200+ Artifacts

against Change
Taxonomy using Text
Analytic Tool
Information Cloud
Program Rpts:
Information
SARS,
DAES
Program Rpts:
SARS,
DAES
Program
Artifacts:
AoAs, ISPs, CBAs
Program Artifacts:
AoAs, ISPs, CBAs
MDAP Data
Sources
MDAP Data
Sources
DoD
Cloud
Repositories
DoD
Repositories
ARJ
Articles
DoD
Experts
ARJ
Articles
DoD
Experts
CAPE and
Service Cost
Centers
CAPE
and
Service Cost
Centers
Experiment to Test
Experiment
to Test &
Inter-Rater
Reliability
Inter-Rater
Reliability &
Value of Repository
Value1 of Repository
Group
Pretest
Use
DRR
Posttest
2
3
4
QUELCE October 2014

41
Sources of Data to Populate the Reference Point

Repository
Subject Matter Experts need DoD
MDAP data about uncertainty to
quantify relationships of program
change drivers and their impact on
program execution.
Why Hard? Empirical data need to be
identified, accessed, extracted and
analyzed from a myriad of sources.
Data about program change is not
structured nor quantified for use in
estimation.
DoD Need: Quantified information
about cost driver uncertainty should
inform estimates.
1. Identify
Change
Drivers &
States
3. Assign
Conditional
Probabilities to
BBN Model
2. Reduce Cause and

Effect Relationships
via Design Structure
Matrix techniques
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
5. Monte Carlo
Simulation to
Compute Cost
Distribution
Information Cloud
Program Rpts:
SARS, DAES
Program Artifacts:
AoAs, ISPs, CBAs
MDAP Data
Sources
DoD
Repositories
DoD
Experts
ARJ
Articles
CAPE and
Service Cost
Centers
QUELCE October 2014

42
Example Use of the Repository to Support Cost

Estimation - 1
Materiel Solution Analysis Phase Pre Milestone Estimate
2. Reduce Cause and
Effect Relationships via
Matrix techniques
1. Identify
Change
Drivers &
States
Program Change
Repository
Prog
DDG51
JTRS
F22
State
cond 1
Driver
CONOPS
cond 2
cond 3
cond 1
cond 2
cond 1
cond 2
cond 3
System De
CapDef
InterOpera
Prod uctio
Contract
Functional
CONOPS
3. Assign
Conditional
Probabilities to
BBN Model
Records show that Strategic Vision

changed in 45% of the MDAPS
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
Stable
Users added
Additional
(foreign)
customer
defined
New condition
New mission
Capability
Definition
Stable
Addition
Funding
Schedule
Established

resources {e.g.
operational test}
Stable

loses particpant
Advocacy
Change
Closing
Technical
Gaps (CBA)
Selected Trade
studies are
sufficient
Technology does not

performance
Technology is
too expensive
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
Technology not
New technology not
performing as
testing well
expected
~~~~
~~~~
5. Monte Carlo
Simulation to
Compute Cost
Distribution
The Materiel Solution of a

global network command and
control system anticipates a
possible change in Strategic
Vision which will include allied
participation.
For C2 systems,
how often does
Strategic Vision
change?
Driver State Matrix
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
Sharing information with allies

creates new encryption
requirements (a change in
Mission/CONOPs).
These changes lead to
changes in Capability
Definition.
Repository identifies probability of

change in MDAP cost drivers.
QUELCE October 2014
43

Estimation - 2
2. Reduce Cause and
Matrix techniques
1. Identify
Change
Drivers &
States
Program Change
Repository
70% of the time the

Mission/CONOPS changes
DSM Cause-Effect Matrix
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
Stable
Users added
Additional
(foreign)
customer
defined
New condition
New mission
Capability
Definition
Stable
Addition
Funding
Schedule
Established

resources {e.g.
operational test}
Stable

loses particpant
Advocacy
Change
Closing
Technical
Gaps (CBA)
Selected Trade
studies are
sufficient
Technology does not

performance
Technology is
too expensive
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
Technology not
New technology not
performing as
testing well
expected
~~~~
~~~~
Change Drivers - Caus
Effects
Causes
Mission / CONOPS
Advocacy Change
Interoperability
Systems Design
Interdependency
3
3
2
1
1
1
Interoperability
Driver State Matrix
System De
CapDef
InterOpera
Prod uctio
Contract
Functional
CONOPS
Advocacy Change
cond 2
cond 3
cond 1
cond 2
cond 1
cond 2
cond 3
F22
Driver
CONOPS
If Strategic Vision
changes, what
else changes?
JTRS
State
cond 1
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
5. Monte Carlo
Simulation to
Compute Cost
Distribution

participation.
Mission / CONOPS
Prog
DDG51
3. Assign
Conditional
Probabilities to
BBN Model

Mission/CONOPs).
Definition.
Repository
identifies
cascading effects of change

in MDAP cost drivers.
2
2
QUELCE October 2014

44

Estimation - 3
2. Reduce Cause and
Matrix techniques
1. Identify
Change
Drivers &
States
3. Assign
Conditional
Probabilities to
BBN Model
When
both Strategic Vision & Mission/CONOPs
experience change, the BBN calculates that
Capability Definition will also change
DSM Cause-Effect Matrix
Stable
Addition
Funding
Schedule
Established

resources {e.g.
operational test}
Stable

loses particpant
Advocacy
Change
Closing
Technical
Gaps (CBA)
Selected Trade
studies are
sufficient
Technology does not

performance
Technology is
too expensive
Selected solution
cannot achieve
desired outcome
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
~~~~
Technology not
New technology not
performing as
testing well
expected
~~~~
~~~~
Causes
Mission / CONOPS
Advocacy Change
Interoperability
Systems Design
Interdependency
3
3
BBN Model

Definition.
2
1
1
1

participation.
Interoperability
Capability
Definition
Effects
New mission
Additional
(foreign)
customer
New condition
Advocacy Change
Users added
defined
Stable
Alternative States
Additional
Production
Scope Reduction
deliverable (e.g.
downsized
(funding reduction)
training & manuals)
Program
New echelon
becomes Joint
Trade-offs
Subtraction
Variance
[performance vs
affordaility, etc.]
Obligated vs.
allocated funds
issue
end of year
out
shifted
Advocate
Service owner
requires change
different than
get re-elected
pentagon staff
in mission
CONOPS users
scope
Scope
Definition
Mission /
CONOPS
Mission / CONOPS
Change Drivers - Caus
5. Monte Carlo
Simulation to
Compute Cost
Distribution

Mission/CONOPs).
95% of the time.
Joint conditional
probabilities
Driver State Matrix
can be calculated for
downstream changes.
4. Calculate Cost
Factor
Distributions for
Program Execution
Scenarios
2
2
QUELCE October 2014

45
For More Information

QUELCE Technical Reports:
http://www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm
http://www.sei.cmu.edu/library/abstracts/reports/13tr001.cfm
SEI Webinar (recorded Oct 31, 2012)
http://www.sei.cmu.edu/library/abstracts/webinars/Quantifying-Uncertainty-in-EarlyLifecycle-Cost-Estimation.cfm
SEI Blog
http://blog.sei.cmu.edu
Improving the Accuracy of Early Cost Estimates for Software-Reliant Systems, First in
a Two-Part Series
A New Approach for Developing Cost Estimates in Software Reliant Systems, Second
in a Two-Part Series
Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE): An Update Journal

of Software Technology. http://journal.thedacs.com/issue/64/207
An Innovative Approach to Quantifying Uncertainty in Early Lifecycle Cost Estimation,
Proceedings,10th Acquisition Research Symposium, 2013
http://www.acquisitionresearch.net/files/FY2013/NPS-CE-13-C10P04R05-058.pdf
QUELCE October 2014

47
Contact Information
Presenters / Points of Contact
SEMA Cost Estimation Research
Group
Robert Ferguson
rwf@sei.cmu.edu
Dennis Goldenson
dg@sei.cmu.edu
Jim McCurley
jmccurle@sei.cmu.edu
Robert Stoddard
rws@sei.cmu.edu
Dave Zubrow
dz@sei.cmu.edu
U.S. Mail
Software Engineering Institute
Customer Relations
4500 Fifth Avenue
Pittsburgh, PA 15213-2612, USA
Web
www.sei.cmu.edu
www.sei.cmu.edu/contact.cfm
Customer Relations
Email: info@sei.cmu.edu
Telephone:
+1 412-268-5800
SEI Phone:
+1 412-268-5800
SEI Fax:
+1 412-268-6257
QUELCE October 2014
48

QUELCE - Quantifying Uncertainty For DoD Acquisition Programs 2014

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

QUELCE - Quantifying Uncertainty For DoD Acquisition Programs 2014

Uploaded by

Copyright:

Available Formats

QUELCE: Quantifying Uncertainty for DoD

QUELCE - October 2014

Copyright 2014 Carnegie Mellon University

QUELCE October 2014

Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE)

The Uncertainty Problem: DoD Context

Unrealistic estimates are caused by the invalidity of major cost-estimating assumptions,

QUELCE addresses the assumptions and decisions made

Early Estimation Lacks Dataand Change is

QUELCE October 2014

Changes in DODs 2011 Portfolio of Major Defense

Source: DEFENSE ACQUISITIONS: Assessments of Selected Weapon Programs,

QUELCE October 2014

Estimates are a Root Cause of Cost Overruns

DoD Programs Re-visit their Estimate Many Times

DoD 5000 with Naval POPS Gate Reviews

Motivation for QUELCE

Capture the probability of change and the change effects

Adjusting an estimate by probability of change

Include assessment of risk

QUELCE October 2014

Uncertainty in Product Development

QUELCE October 2014

MORS*: Affordability Analysis Workshop

Use a distribution when estimating (not point estimates)

*From Affordability Analysis Workshop Report Oct 2012

Reduce scale and complexity of the decision space

Multiple cost estimation relationships

Multiple scenarios of risk

Access to subject matter experts and other stakeholders is required.

QUELCE Inserts New Activity in Information Flows

Information from Analogous Programs/Systems

Proposed Material Solution & Analysis of Alternatives

Program Execution Change Drivers

Plans, Specifications, Assessments

QUELCE October 2014

Overview of the QUELCE Method

Manning at program office

Prog Mgt Structure

Functional Solution Criteria (measure)

Building Technical Capability & Capacity (CBA)

Closing Technical Gaps (CBA)

Project Social / Dev Env

Program Mgt - Contractor Relations

Change in Strategic Vision

4. Cost Factor Distributions by

Change Drivers - Cause & Effects Matrix

Build BBN Model

2. Dependency Structure Matrix

2. Cause and Effect

Funding delays tie up

Joint service program

Technology does not

5. Monte Carlo with

1. Driver State Matrix

Use of Expert Judgment

In this context, a great deal of information is generated by experts to propose a major

Experts Tend to Be Over-Confident

QUELCE October 2014

2014 Carnegie Mellon University

Reference Points to Help Calibrate Judgment for Estimation

Size of ground combat vehicle

Size of Army artillery firing