TECHNICAL PAPER

Decision & Risk Analysis

By: Justin Mao-Jones
Consultant
The Merrick Consultancy

Merrick & Company

2450 S. Peoria Street Aurora, CO 80014-5475
Tel: 303-751-0741 Fax: 303-751-2581
www.merrick.com

March 2012
Make Better R&D Decisions with Decision and Risk Analysis

TABLE OF CONTENTS
S ection Page
I ntroduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
U ncertainty, D ecision Criteria , and Weighted Averages 2
What D oes a D ecision P roblem L ook L ike in D&RA? 3
R isk A nalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Computer Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
O ther Useful Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Influence Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Value of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Utilit y Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
I ncorporating D&RA into Business and Strategic P lanning 8
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
R eferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Justin Mao-Jones The Merrick Consultancy i

Make Better R&D Decisions with Decision and Risk Analysis

Introduction
Decision and risk analysis (D&RA) is a field of study incorporating concepts and philosophies that are
designed to help organizations make better decisions. According to the Stanford Research Institute, D&RA is a
discipline that seeks to apply logical, mathematical, and scientific procedures to the decision problems of top
management1. At Merrick, D&RA has five basic steps:
1. Frame the decision problem
2. Model the frame
3. Quantify the risks and uncertainties
4. Analyze the model
5. Make decisions

Figure 1 - D&RA is a cost-effective way to help decision-makers identify which pieces of information are most important to project success,
thus facilitating decision efficiency.

In large part, D&RA is about understanding the uncertainties inherent in the decision problem by applying
probabilities to a range of possibilities for each potentially significant uncertainty. With the help of mathematical
and graphical tools, the practitioner can bring simplicity to complicated scenarios. As a project progresses,
valuable information comes to light and uncertainty is reduced. D&RA identifies those uncertainties that are
most important to project success, allowing decision-makers to optimize information gathering activities. This
increases the rate at which valuable information is accrued. Having more valuable information earlier on in the
project facilitates better project definition and avoids costly design changes down the road.

Figure 2 Value of D&RA over the project timeline. With better information earlier on, decision-makers can make good decisions sooner.
This promotes stronger project definition and avoids costly design changes down the road. As the project progresses, it becomes more
and more difficult to influence positive change to project value, and thus D&RA can be an important tool in front-end project development.

Justin Mao-Jones The Merrick Consultancy 1

Make Better R&D Decisions with Decision and Risk Analysis

D&RA has been highly successful in practical applications across industries and government. It has become
an industry standard in oil and gas exploration and pharmaceutical R&D. A Society of Petroleum Engineers
benchmarking effort found that petroleum companies introducing D&RA greatly improved their industry ranking
in just a few years2. According to the Society of Decision Professionals, a review of hundreds of projects across
industries suggests that decision consulting often yields value from 100 up to 1000 to 13. In fact, a 1993 decision
consulting effort at Smith-Kline Beecham returned an estimated 1300 to 1 value to cost (in-house analyst costs
and consulting fees) ratio3,4. Over the years, Merrick has helped several clients improve project value by up to
20%, shift R&D resources to higher potential initiatives, identify major risks (often hidden beneath the surface),
cut project development time by several months, and save millions in conceptual engineering costs. Needless to
say, D&RA has high value potential.

Uncertainty, Decision Criteria, and Weighted Averages

Every D&RA should follow a few simple rules. They are as follows:
Rule #1: Uncertainty should be modeled with probability distributions (a range of possibilities combined with
probabilities assigned to each of those possibilities). In general, the simpler the probability distribution the
better, but sometimes more complicated distributions are appropriate, such as a curve fitted to an empirical
data set.
The goal of defining an uncertain variable is to capture a reasonable representation of its possible values. It
is not to portray a perfect depiction of the uncertain variable. Generally speaking, such thinking is inherently
flawed, because in the future uncertain variables will ultimately have a single point value. For example, we may
put an uncertainty range on the capital cost of a project, but the actual money spent will be a fixed dollar value.
A common method for defining uncertain variables is to use the P10/50/90 framework. It is conveniently simple
when eliciting information from subject matter experts. Mathematically speaking, the P10, P50, and P90
represent the 10th, 50th, and 90th percentiles of the ranges, respectively. Figure 3 shows an example of how the
P10/50/90 framework is used to model uncertainty. Conceptually, the P10 is lowest value that the expert thinks
that the uncertain variable could be. The P50 is the most likely value. The P90 is the highest value that the
expert thinks the variable could be. Room is left below the P10 and above the P90 for highly unlikely scenarios,
such as Acts of God.

Figure 3 An example of how a P10/50/90 range is used to model uncertainty.

Justin Mao-Jones The Merrick Consultancy 2

Make Better R&D Decisions with Decision and Risk Analysis

Rule #2: To decide among viable options, the decision-maker must have a single, meaningful metric (i.e. a
decision criterion). A meaningful metric is not always obvious. Many business decision-makers commonly
use financial metrics such as net present value and internal rate of return. In complicated decision problems,
return on investment metrics may be insufficient to handle the nuances of the problem frame. Important criteria
may be difficult to quantify and incorporate into an overall decision criterion, such as weighting safety and
functionality considerations against each other. Methods such as multi-attribute utility theory are used in these
instances to derive a utility function that serves as the decision criterion. Deriving the right decision criterion in
the framing session is one of the most important exercises in the D&RA process.
Rule #3: The decision-maker should choose the option with the best expected decision criterion (i.e. the high-
est or lowest weighted average decision criterion). For example, a frame using net present value (NPV) as the
decision criterion would choose an alternative with the highest expected NPV. Conversely, a frame may choose
the alternative that maximizes the probability of an acceptable NPV. As an extension of this rule, when model-
ing future decisions the practitioner should assume that the decision-makers will be making good decisions,
unless special circumstances of the frame logically dictate otherwise.
It is important to note that a good decision can lead to a bad outcome, an unlucky scenario. Conversely, a bad
decision can result in a good outcome, a lucky scenario. Good decisions serve to maximize the likelihood of
good outcomes. Bad outcomes do not imply bad decision-making, unless they reveal poor framing, faulty logic,
or inappropriate use of resources1. With the right frame, appropriate use of available information, and a model
that adequately incorporates the nuances of the decision problem, a decision-maker should expect that he or she
is making the best decision possible.

What Does a Decision Problem Look Like in D&R A?

At D&RA conferences, one will often hear practitioners extol the virtues of getting the frame right. Thats because
after years of experience, practitioners have discovered that the most important and often overlooked part of the
process is the frame. With the right frame, you will analyze the right issues, and come to the right decisions. So it
goes, with the wrong frame you will get the wrong answer.
The frame should encompass the following:
Objectives: what decision(s) do we need to make?
Major issues: what do we need to worry about, such as major uncertainties?
Scenarios: what could our long-term environment look like?
Alternatives: what are the distinctly different options at each decisional node?
Decision criterion: how will we measure the effectiveness of decisions?
In R&D projects, decision-makers must decide which ideas are worth pursuing and how to pursue them. Merrick
recently worked with a biofuels client that was deciding between three reactor concepts, four ways of utilizing
each reactor, four energy integration schemes, and three distinctly different product profiles, for a total of 144
alternatives. The objective was to choose the alternatives that had the highest potential for long-term success
and to provide clarity on project risks and opportunities. Major issues included lack of development (high
uncertainty) in particular areas. Capital costs were a major concern and development of the reactors was in a
state of high flux. The decision criterion was the maximum affordable capital cost that would produce a specified
internal rate of return.
The example above focused on technology selection. A different kind of frame could focus on decisions that
lead up to technology selection. R&D focuses on preliminary studies, research campaigns, bench and pilot scale
testing, etc., to find good ideas, flush out bad ones, and develop commercially viable technology. In other words,
a frame could optimize resource allocation.
Many R&D frames have the following general issues:
1. Need for long-term growth and success
2. Numerous alternatives
3. Limited funding
4. Limited time frames to produce results
5. High uncertainty

Justin Mao-Jones The Merrick Consultancy 3

Make Better R&D Decisions with Decision and Risk Analysis

To illustrate the application of D&RA to an R&D decision problem, consider the following example of an R&D
decision at a hypothetical biofuels company. Suppose that the company has devised a methodology for
improving yield values of algae strains for the production of algae oil, which can be converted to biodiesel. The
company has spent the last two years developing a particular strain, Strain A, with moderate success. It has also
identified a new, though relatively unknown, Strain B that has the potential for even higher yields. The team feels
confident that, because of the strain development knowledge and skills acquired over the past two years, it could
develop Strain B at a greatly accelerated pace.
The manager, however, is not convinced that a mid-project audible is a good idea. If the company cannot show
results within the next two years, future fundraising will become very difficult. Work on Strain B would mean
partially or completely diverting resources from work on Strain A. The team is already on track to produce
moderate, but acceptable results with Strain A. And there is always the possibility that Strain B may not meet the
high expectations or may even turn out to be a dud.
The manager could decide to table Strain B for now and continue to devote all resources towards Strain A. He
is confident that his team can prove moderate yields within one to two years. Alternatively, he could decide to
divert his entire team towards Strain B. Strain B could prove to have moderate to high yields, but may not be able
to be grown beyond low concentrations. This would necessitate much higher capital and operating costs in a
commercial facility to produce the same amount of algae as Strain A and thus be a major disadvantage. It would
take about 6 months to determine whether or not Strain B can be grown at high enough concentrations. If it turns
out that it can, he estimates that it will take about one to two years to prove the yields for Strain B. If it turns out
otherwise, he can divert his team back towards Strain A development.

Figure 4 Decision tree of R&D direction based on time to complete research.

Within these two scenarios are three main variables: time to complete research, Strain B productivity, and Strain
B concentration constraint. One way to frame this problem is to look at it from the standpoint of whether or
not sufficient yields will be proven in time to show the investors. In this frame, the manager would choose the
alternative that maximizes the chance of research completion within two years. Figure 4 illustrates this frame
with a decision tree. Focusing on Strain A yields 100% certainty that the research will be completed on time.
Focusing on Strain B, however, yields an 84% probability that research will be completed on time.
If the team determines that Strain B can grow in high concentrations, there is 100% probability that the yields can
be proven within the two year time frame, assuming that Strain B development is concurrent with the six month
study. If not, the team will have lost six months to work on Strain A, leaving only 1.5 years to finish the work. The
team estimates that there is an equal chance that the work will take 1, 1.5, or 2 years to complete. This means
that there is only a 67% chance that they will be done on time. If the team assumes ahead of time that there is a
50% chance that Strain B can grow in high concentrations, then the probability of completing the work on time is
84%.

Justin Mao-Jones The Merrick Consultancy 4

Make Better R&D Decisions with Decision and Risk Analysis

This analysis clearly shows that continuing work on Strain A is the best path forward, because it ensures that
research on Strain A will be complete within two years. However, with a longer-term frame, this answer could
easily change.
Consider that the manager must also take into account the long-term future of the company. He decides to
measure this by looking at the investment potential, i.e. projected NPV, of a commercial facility utilizing either
strain. In this frame, Strain B gains an advantage in that higher yields would mean a higher NPV.

Figure 5 Decision tree on research direction based on NPV of future commercial facility.

Figure 5 illustrates this new frame with a decision tree. Based on financial projection analysis, the team
concludes that a commercial facility using Strain A would be worth $100 million in NPV. Using Strain B, however,
NPV could range from $120 up to $150 million. The team estimates a 50% chance that Strain B yields will be
high versus moderate, though always better than Strain As. If growth in high concentrations is possible, then
the expected NPV of a Strain B commercial facility is $135 million. If not, then the company must look at the
investment potential of Strain A, given that it only has 1.5 years left, having already wasted 6 months. Under the
scenario that the team does not finish the research in time, the yields of Strain A will be lower than target yields.
The team estimates that with these lower yields, the commercial facility will only be worth $80 million in NPV.
Thus, the expected NPV of the commercial facility will then be $93 million.
Assuming again a 50% chance that Strain B can grow in high concentrations, the expected NPV of the
commercial facility by following Strain B research path is $114 million. Therefore, the manager should decide to
divert resources to Strain B to achieve an expected increase in NPV of $14 million.
Keep in mind that this frame chooses the alternative that will yield the highest expected NPV after the two years
are up. The question it answers is: Given the amount of time I have available, what research pathway will yield
the highest potential for long-term success? In essence, it is a resource allocation problem.
This example is, of course, oversimplified. In many cases, deadlines take a backseat to funding constraints. The
frame could have read Given the amount of funding I have available, what research pathway will yield the highest
potential for long-term success? In real-life problems, there will likely be other decisional nodes and tens of
uncertainties. Furthermore, there may be other alternatives. For example, the manager could have evaluated the
option of diverting only 50% of his team to Strain B.
As the number of alternatives, decisional nodes, and uncertainties increase, the complexity of the problem
grows exponentially. For example, assume that the manager was also deciding between three algae oil recovery
technologies. Also assume that there are 10 uncertainties, each with three outcomes. With two research
pathways for algae strain development, three recovery technology options, and thirty points of uncertainty, there
are 180 scenarios to consider. Toss in a few more research studies and uncertainties and that number rises
above a thousand!

Justin Mao-Jones The Merrick Consultancy 5

Make Better R&D Decisions with Decision and Risk Analysis

Risk Analysis
Risk evaluation is a natural part of the decision problem. Uncertainty is essentially a lack of information and risk
is the potential impact of making decisions without that information. Risk analysis investigates this relationship
and is often illustrated using a tornado diagram, as illustrated in Figure 6. The tornado diagram depicts
uncertainties ranked by potential impact to the decision criteria. The larger the range of impact to decision
criteria, the larger the combined risk and potential upside. The uncertainties at the top of the tornado have the
highest potential impact. Each bar of the tornado is generated by ranging the corresponding input variable
across its uncertainty range while keeping all other variables at their 50th percentile values (the P50s). Typically,
inputs are ranged from their 10th percentile value (the P10) to their 90th percentile value (the P90).

Figure 6 Tornado diagram of example biofuels project. Each bar is generated by swinging its corresponding uncertainty to its P10 and
P90 values and calculating NPV while holding all other uncertainties at their P50 values. Uncertainties are ranked from top to bottom in
order of highest impact to NPV (i.e. largest dark green bar).

The point of the diagram is to very clearly show how important each uncertainty is relative to the others. It
generates a discussion on which variables warrant special attention such as risk management, an exercise in
reducing uncertainty.
Sometimes, reducing uncertainty can affect decisions. In Figure 6, for example, feedstock composition has
a very high impact to NPV. This suggests that the decision-makers should give feedstock selection high
importance and possibly spend money to investigate potential feedstock options. Indeed, the tornado diagram
can lead practitioners and decision-makers to explore new directions when uncertainties unexpectedly rise to the
top.
Just because an item does not appear on the tornado diagram or is ranked low does not mean that it is not
important. Instead, it means that the uncertainty is not large enough to make a difference. For example, efficient
operation of plant steam boilers may be important, but such a system is well understood across industries. The
efficiency will likely fall within a small range of values with minimal relative impact to overall project economics.

Computer Simulations
Computer simulation helps facilitates D&RA. These simulations are designed to process decision alternatives
and uncertainties, which are modeled with probability distributions. One way to do this is with decision trees, as
shown previously in the algae development example.

Justin Mao-Jones The Merrick Consultancy 6

Make Better R&D Decisions with Decision and Risk Analysis

Decision trees can be an essential tool in analysis and make presentation of analysis to the decision-makers
much easier for the practitioner. The tree is a hierarchy of decisions and uncertainties. Each node represents
a particular decision (typically represented by a square) or uncertainty (typically represented by a circle). Each
branch represents a possible decision, when stemming from a decision node, or a possible outcome, when
stemming from an uncertainty node. Figure 4 and Figure 5 are simple examples of a decision tree.
Monte Carlo analysis is another method for determining expected values of decision pathways and analyzing
risks. It is especially useful when dealing with highly complex problems. The analysis is performed through
repeated random sampling of all independent uncertain variables (inputs) in order to obtain a distribution
of values for each dependent variable of significance (outputs). In each Monte Carlo iteration, each input is
assigned a deterministic value randomly chosen from its probability distribution and the deterministic value of
each output is calculated. Multiple iterations yield a distribution of values for each output. As more and more
iterations are performed, each outputs distribution represents a closer and closer approximation of the outputs
actual distribution. With the Monte Carlo distributions, one can derive expected decision criteria values for
alternatives as well as uncertainty metrics such as P10/50/90 values. The uncertainty metrics are helpful for
understanding the overall uncertainty in each alternative. Commercially available software such as @Risk and
Crystal Ball are commonly used Monte Carlo simulators.

Other Useful Tools

Influence Diagrams
Influence diagrams are graphical depictions of how components in the decision problem affect each other. The
diagram is also useful as a conceptual basis for creating a mathematical model of the frame. In Figure 7, which
is a simplified version of an influence diagram, feedstock throughput is a function of feedstock composition and
feedstock availability. All arrows eventually lead to the decision criteria. It is important that the computer model
captures these relationships so that the D&RA results are based on a reasonable representation of reality.

Figure 7 Simplified influence diagram

Value of Information
Uncertainty exists when we lack information about the future. Consequently, information has value, because it
can change the future that we choose. The value of information (VOI) is calculated as the change in expected
value when new information is presented. This exercise is typically performed before investing in new

Justin Mao-Jones The Merrick Consultancy 7

Make Better R&D Decisions with Decision and Risk Analysis

information. Sometimes, information has no value, because it does not change the decision. Knowing this,
however, can be valuable by preventing unwarranted uses of resources.
Utility Functions
D&RA can be used to quantify anything, including soft or emotional issues. For example, a manufacturing
company might be choosing among a number of different sites for a new facility, but must wrestle with proximity
to highway and rail, workforce availability, traffic safety, noise ordinances, site buffering, land use impacts, etc. In
this case, the practitioner can derive scales to quantify and rank each siting attribute and weight each attribute
under an overall scale accordingly.
There are established methods for deriving utility scales such as SMART and Multi-Attribute Utility Theory5.
Different methods apply under different circumstances. Most are based on behavioral research and scientific
data.

Incorporating D&R A into Business and Strategic Planning

Many industry best practices for project development such as Front End Planning advocate that D&RA be applied
during conceptual phase planning, because it is a successful tool for evaluating various decision scenarios. The
conceptual phase succeeds the feasibility stage, in which rough numbers are estimated and then applied to
generate options and alternatives, and precedes the detailed scope phase, in which key design documents and
detailed project plans are developed. Important decisions are made during the conceptual phase and can have
significant and lasting impact to the success of a business or project, and thus this phase is ideal for D&RA.
Clearly, D&RA should not be used to make every decision. According to the Stanford Research Institute1, D&RA
is most appropriately applied to problems that have several of the following characteristics:
Uniqueness: the problem is one of a kind.
Importance: a significant portion of the organizations resources is involved.
Uncertainty: many key factors are known only imperfectly.
Long run implications: the organization will be affected by the results of the decision for many years.
Complex preferences: the desires of the decision-maker are not clearly formulated.
Strategic planning activities, which come before business planning, also have valuable uses for D&RA. A typical
strategy management process addresses five main questions: 1) where are we today?; 2) what does the future
look like?; 3) where do we want to be down the road?; 4) how do we get there?; 5) how do we measure our
progress? D&RA comes into play in identifying how to close the gap between the desired future state and the
current state of things. This exercise uses simpler models and is much less-involved in strategic planning than in
business planning activities, because strategic planning is a higher level exercise.

Summary
D&RA is a decision-making tool. It is a construct for defining and understanding decision problems and is ideally
suited to problems of high complexity, high importance, and high uncertainty. At Merrick, we often find that the
D&RA journey is as important as the result. Every part of the process provides the D&RA team with a much
better understanding of the problems facing decision-makers. With D&RA, decision-makers can expect to make
better decisions and manage those uncertainties that are most impactful to project success.

Justin Mao-Jones The Merrick Consultancy 8

Make Better R&D Decisions with Decision and Risk Analysis

References
1. Matheson, J. E., and Howard, R. A. An Introduction to Decision Analysis, Stanford Research Institute, Palo
Alto, CA, 1968. Reprinted in Howard and Matheson (Eds.), The Principles and Applications of Decisions
Analysis, SDG, Inc., Menlo Park, CA (1983), 17-55.
2. Jonkman, R. M., Bos, C. F. M., Breunese, J. N., Morgan, D. T. K., Spencer, J. A., and Snden, E. Best
Practices and Methods in Hydrocarbon Resource Estimation, Production and Emissions Forecasting,
Uncertainty Evaluation and Decision Making, Paper SPE 65144 presented at the 2000 SPE European
Petroleum Conference, Paris, France, October 24-25.
3. Menke, M. M., Spetzler, C., Keelin, T. The Value of DA/DQ: Building a Compelling Case for Decision Makers,
Presentation through Society of Decision Professionals Learning Exchange, February 2, 2011.
4. Sharpe, P., Keelin, T. How SmithKline Beecham Makes Better Resource-Allocation Decisions, Harvard
Business Review March-April 1998. Reprint 98210.
5. Goodwin, P., Wright, G. Decision Analysis for Management Judgment, 4th Edition, West Sussex, United
Kingdom: John Wiley & Sons Ltd., 2009.

Justin Mao-Jones The Merrick Consultancy 9