You are on page 1of 4

SPE 108523

Spreadsheet Simulators That Demonstrate How Statistics Can Lie About Safety
Performance
Carl Veley, SPE, vPSI Group

Copyright 2007, Society of Petroleum Engineers


This paper was prepared for presentation at the SPE Asia Pacific Health, Safety, Security and
Environment Conference and Exhibition held in Bangkok, Thailand, 1012 September 2007.
This paper was selected for presentation by an SPE Program Committee following review of
information contained in an abstract submitted by the author(s). Contents of the paper, as
presented, have not been reviewed by the Society of Petroleum Engineers and are subject to
correction by the author(s). The material, as presented, does not necessarily reflect any
position of the Society of Petroleum Engineers, its officers, or members. Papers presented at
SPE meetings are subject to publication review by Editorial Committees of the Society of
Petroleum Engineers. Electronic reproduction, distribution, or storage of any part of this paper
for commercial purposes without the written consent of the Society of Petroleum Engineers is
prohibited. Permission to reproduce in print is restricted to an abstract of not more than
300 words; illustrations may not be copied. The abstract must contain conspicuous
acknowledgment of where and by whom the paper was presented. Write Librarian, SPE, P.O.
Box 833836, Richardson, Texas 75083-3836 U.S.A., fax 01-972-952-9435.

Abstract
The science of statistical analysis uses advanced
mathematics beyond the capabilities of most people, but the
principles behind those high level equations are essential in
understanding the role of chance in conventional safety
performance metrics. A series of spreadsheets has been
developed to illustrate some of the essential principles,
without utilizing the daunting rigorous mathematical proofs.
Readers who experiment with these spreadsheet simulators
may be surprised to find some of their routine or basic
assumptions about safety statistics are actually scientifically
indefensible. Studying the principles illustrated by these
simulators may suggest needed modifications to common
industrial safety programs.
Introduction
There are many reasons for wanting to know the likelihood
of future accidents. Insurance companies must guess how
many accidents a client will experience in the next year, then
estimate how much those accidents will cost and calculate
premiums accordingly. Consequences of guessing wrong may
be very costly. Large companies or organizations often
reward departments, managers, or individual employees who
have the least chance of future accidents, and mistaken
rewards can also have costly consequences.
Ultimately these are two examples of the same process
using trailing indicators to forecast future performance and
issuing rewards based on that forecast. The objective of this
study was to illustrate why this practice is unreliable and often
has damaging consequences.
The phenomena described in this paper are easily
explained with high level mathematics, but most managers
making business decisions based on accident probabilities are
not skilled in such specialty mathematics. There is a need for
easily understood ways of illustrating the underlying

principles, without resorting to complex equations, and that is


the purpose of the simulators described in the following pages.
Probability of a given group of people having an accident
depends on three things:
1. What they do,
2. How they do it, and
3. Luck
Of course group members can change their accident
probability by changing what they do or how they do it, but if
they always do the same thing in the same way, then the
probability is only a matter of chance, or just plain luck. That
is the basis for all simulators described in this paper. We will
use various means of deciding what the overall probability is
that an accident will occur, then we will use a random number
generator to play the role of luck in determining if and when
accidents occur.
These are hands-on instructions for building computer
simulations, and readers should open a new workbook in
Microsoft Excel and follow the step-by-step procedures. The
same process applies in other spreadsheet computer
applications, but there may be minor differences in the way
statements or functions are entered.
Simulation Basics
All simulator models illustrated in this paper use a
standard random number feature to let pure chance determine
if something occurs. Here is how it works.
Open a new spreadsheet in Microsoft Excel.
1. Enter the number 10 in cell A1.
2. In cell B1 type:
=RAND() .
then press the enter key.
This statement produces a random number between 0.0
and 1.0 every time you press the F9 (re-calculate) key.
The number may be very small, such as 0.000001, but it
will never be zero, and it may be very close to one, say
0.999999, but it will never be 1.0. Press the F9 key
several times to see how this works.
3. Replace the statement in cell B1 with:
= INT($A$1*RAND())
and press the enter key.
This causes the computer to pick a random number
between zero and one, multiply it by the value in cell
A1, but display only the integer part of the product. The
numbers seen in cell B1 will be whole numbers ranging
from 0 to 9, and they will appear in random order every

www.petroman.ir

SPE 108523

time the F9 (recalculate) key is pressed. Each of the


numbers are equally likely to appear. For example,
pressing the F9 key is just as likely to produce a 3 as it
is to produce a 6 or any of the other possibilities.
4. Change the 10 in cell A1 to 15, then press the F9 key
several times. Now the number in cell B1 will vary
randomly between 0 and 14, with each of the 15 possible
numbers as likely to occur as any other.
5. Put the number 10 in A1 again.
The number in cell A1 is the total possibilities, but the
numbers that appear will always range from zero to one
less than the possibilities. If A1 contains 5 then there
will be one of 5 possible numbers in cell B1, namely 0,
1, 2, 3, or 4. If A1 contains the number 1000, there will
be 1000 possible numbers in B1, but they will include
zero and stop at 999. The number 0 is just as likely to
appear in B1 as 17, 499, 921, or any other of the 1000
possibilities. Zero is the only number that will always be
one of the possibilities, regardless of the number in A1,
and it is always just as likely to appear as any other
possibility.
6. Click in cell C1 of the spreadsheet and enter:
=If((B1=0),1,"")
and press the Enter key.
Press the F9 key repeatedly and notice that cell C1 is
always blank except when a 0 appears in cell B1, and
then it contains a 1.
7. Simplify, by combining all these steps into one cell.
Select column C and delete it completely, then replace
the contents of cell B1 with:
=IF((INT($A$1*RAND())=0),1,"")
Press the F9 key repeatedly to make sure cell B1 is blank
most of the time but occasionally contains a 1.
8. Select cell B1, then on the toolbar, click Format;
Conditional Formatting; and set conditions to cause cell
B1 to have a red font and a red background pattern when
the cell contains the value 1. Now press F9 repeatedly
and note that cell B1 is usually blank but occasionally
turns solid red.
9. Select cell B1, put the cursor on the lower right hand
corner and drag it down to cell B52. This makes 52
copies of cell B1, one for each of the 52 weeks in a year.
10. Click cell B53 and enter:
=SUM(B1:B52)
This is now a working simulator. Later we will add
features to make it more versatile, but a little experimentation
helps understand how the simulation works.
Using this basic Simulator
Imagine a company that has had the same 100 employees,
doing the same jobs, in the same way, for the past 5 years.
Conveniently, each of those employees worked exactly 40
hours per week for 50 weeks, giving the company a total of
200,000 employee hours per year and making the arithmetic
easy. Figure 1 shows recordable injuries officially reported by
this company for a five year period.
Often, people look at numbers like these and conclude:
1. We must have done something right in year 2 because
TRIR declined 50%.

2. That 67% increase between years 2 and 3 shows we got


careless.
3. We have been improving steadily since year 3 with a
50% decrease last year and a 60% decrease from year 3.
4. If we plot these numbers, as in Figure 2, draw a best fit
straight line through them, we have been improving
since Year 1, next year we should expect our TRIR to be
about 2, and our goal of zero injuries is reachable.
5. The 67% decrease from Year 1 means our safety
program works well.
We can use the basic simulator to check if these
conclusions are valid or if the TRIR values can be explained
by chance. The logic is simple if random variation around
the average TRIR, with no change in probability, can produce
the sequence of TRIR values, then the recorded TRIR values
neither prove nor disprove any of the listed conclusions.
The arithmetic average is 4 recordable injuries per year (52
weeks), and that equals 1 injury per 13 weeks. If they occur at
random, then we can pick any week out of the year and the
odds will be 1 out of 13 that an injury will occur in that week.
In real life it is possible that all four injuries could occur in the
same week or even all on the same day. We will refine our
simulator later to handle this possibility, but for now this basic
simulator will assume that the injuries result from separate
events and no two events occur in the same week.
Put the number 13 in cell A1. Now, each time the F9 key
is pressed, chances are 1 out of 13 that any given cell in the
range B1 through B52 will turn red. When one of the cells
turns red that signifies a recordable injury occurred that week,
by chance. Watch cell B53 to see the years total number of
recordable injuries that occurred when the probability was 1
out of 13 for each week.
Each time the F9 key is pressed the spreadsheet simulates
another year, randomly varying but with exactly the same
overall injury probability, and the annual totals are frequently
6 or higher, and 2 or lower. This tells us the TRIR values in
the example table could easily be a simple random variation
around an average value of 4, but we can make that point more
clearly if we upgrade the simulation slightly.
Multi-year Simulation
1. Select cells B1:B53, put the cursor in the lower right
hand corner and drag it across to cell K53.
2. Insert a new row above row 1. This moves everything
down one row and the simulated injury sum now appears
in cell B54 instead of B53.
3. Type Year 1 in cell B1 and Year 2 in cell C1.
(Without the marks.)
4. Select cells B1 and C1 together, put the cursor in the
lower right hand corner, and drag across to cell K1.
Cells B1 through K1 should now read Year 1, Year 2,
Year 3Year 10.
Press the F9 key several times and watch row 54. Cells
B54 through K54 now display the total number of recordable
injuries that occurred by random chance in each of 10 years,
when the probability was exactly 1 out of 13 for each of the
520 weeks. The numbers change each time you press the F9
key, but with only rare exception, at least one of the ten years
will have 6 or more random injuries, and at least one year will
have 2 or fewer simulated injuries. This means the example

www.petroman.ir

SPE 108523

TRIR data table could easily be explained by ordinary random


variation, and it is extremely unlikely there are any real trends
in the data, including the downward slope in Figure 2. If the
next year produces 6 recordable injuries, it will be totally
normal and will not prove a change in injury probability, even
though it is a 300% increase from the previous year.
Other simulator refinements
Readers experienced with Excel or other spreadsheets
should be able to add features like those shown in Figure 3,
where users can input a TRIR and average number of
employees and let the computer calculate the number of weeks
per recordable injury, either year-by-year, or as an overall
average as shown here.
The example in Figure 3 shows variations that could be
expected in a hypothetical situation where 100 employees do
the same things in the same way for 10 years. If we input a
fixed probability of 4 recordable injuries per year, by pure
chance the actual recordables will range from 0 to 10 or more.
We can then let the computer calculate an observed TRIR
based on these randomly generated injury statistics and
compare it with the input probability, 4.0 in this example.
Next, use the spreadsheets chart feature to graphically
represent errors involved in assuming any one years TRIR is
the recordable injury probability. One such chart is given in
Figure 4. and it shows the results of the simulation in Figure 3.
If the chart is set up correctly, each time the F9 key is pressed
all ten years of the simulator will be recalculated, simple
chance will determine which of the 520 weeks yield
recordable injuries, and the graph will show percentage error
involved in assuming the TRIR equals a probability.
Readers who take the time to generate such a spreadsheet
simulator and experiment with it will soon conclude that TRIR
is completely unreliable for predicting future injury
probabilities, with the exception of very large populations.
Small changes in TRIR may be meaningful when derived from
work records for millions of people, but for typical companies
it is not possible to determine if a TRIR change is due to a
change in activities or simply normal random variation. It is
not uncommon to find errors of 200% or more when
simulating small companies with 500 or fewer employees, or
organizations with low TRIR values. No manager would
accept such uncertainty in financial, production, or personnel
forecasts, but major decisions often hinge on safety
performance forecasts no more reliable than those in this
simulator.
Punishment and reward programs
Many companies offer bonuses for work units (vendors,
contractors, work crews, or individual employees) reporting
zero accidents over some specified time period. Also, large
companies often set performance standards for vendors or
service companies (in terms of a maximum injury frequency)
as a condition for either obtaining or retaining contracts. This
simulation process is ideally suited for illustrating the
consequences of such policies.
Figure 5 shows a variation of the same basic simulation
process using an ordinary spreadsheet. In this example we
assume an operating company has a policy that any service
company, however large or small, will be fired if they

experience two or more on-the-job fatalities in any 10 year


period. We will arbitrarily consider 20 service groups and
assume that two of them are high risk organizations, being five
times more likely to suffer a fatality than the other 18 groups.
Set up the simulator as shown in Figure 5 and press the F9
key repetitively. Each re-calculation simulates a normal
distribution of fatalities within the probability ranges,
arbitrarily set at one chance out of 10 for the two high risk
groups and one chance out of 50 for the 18 low risk groups. A
typical normal distribution is shown in Figure 5, but there are
2200 possible combinations of fatalities for 20 companies over
10 years, so the odds against producing exactly the same
distribution as shown here are about 1 out of 1060.
Fortunately you will not need to press the F9 key that many
times to understand the significance of this model.
Here are the principle conclusions derived from this
variation of the simulation model.
1. When we use a test procedure that is even slightly less
than 100% accurate, and we look for something that is
rare in the population tested, a positive test result is
likely to be wrong.
In this case, the high risk units are 5 times more likely
to have a fatality, but there are 9 times more of the low
risk units. Consequently, over time, for every 2 fatalities
among the high risk units, there will be 3.6 fatalities
among the low risk units. In the example of Figure 5 we
unjustly fired five low risk units and fired only one
unit that was actually a high risk unit. (Note that
probability of high and low risk units having two
fatalities in one year is not the same as the ratio of total
fatalities occurring in high and low risk groups.)
2. If rewards are based on uncertain test procedures,
mistakes are highly likely.
For example, if we reward units having zero fatalities
we cannot tell the difference between Unit 2, a high risk
unit, and low risk Units 5, 6, 13, 15, or 20. If we intend
to reward only the low risk units we would in this
example be improperly rewarding one out of six bonus
recipients, but as the simulator will show with a few F9
key presses, it is easily possible to have a much higher
error rate than this example. Issuing rewards to units
(individuals, crews, contractors) who do not deserve
them can have big negative consequences.
3. Applying intrinsically unfair policies will eventually be
counter-productive.
In this example, if we rigorously apply our policy we
will fire some of our best performers, and if we make
exceptions we create morale problems by demonstrating
that our policies are arbitrary or capricious and not really
policies at all, demeaning all other policies. Also, if
these 20 example contractors are competitors, firing 5 of
the good ones will increase the work going to one of the
bad ones and therefore increase probability of future
fatalities, exactly opposite of our intentions.
Dangerous self-deceptions
A common trap for managers occurs when they send
orders down the organizational ladder that subordinate
managers must reach their goals or face dire consequences.

www.petroman.ir

SPE 108523

Here is a way to use another modification of this simulator to


illustrate the dangers of such management practices.
A large company had unfavorable publicity over a
catastrophic accident, but they had for years been boasting of
their successful safety program. Data on their website
indicated their DAFWCF (Days Away From Work Case
Frequency) had been steadily declining for 17 years. The
companys top management assumed that data proved their
HSE programs were working as intended. We can use a
variation of the simulator spreadsheet to check the reliability
of this assumption.
Figure 6 shows the DAFWCF data posted on the company
website. Notice there is not a single year where the DAFWCF
is higher than the previous year. Analysis of these data show
there is almost exactly a 15% per year decrease in DAFWCF
over the 17 year period. Figure 7 shows the format of the
spreadsheet simulator we used to evaluate reliability of the
reported data.
In this case, we used 20 columns of 52 rows to simulate
the weeks in a 20 year period. As before, it assumes the
DAFWCF is the probability of a Day Away From Work Case
occurring, but it starts with a frequency of 2.0 for the first year
then reduces that probability by 15% for each subsequent year.
We built features into this spreadsheet to allow inputting the
number of employees, starting value for DAFWCF, and the
decline rate expressed as an annual percentage.
The simulator shown keeps the same number of employees
for each of the 20 years, but it is easily possible to use the
actual number of employees for each year for a more precise
calculation of the expected weeks between DAFWC events.
For large numbers of employees the simulation must be
improved by calculating in daily or even hourly increments
instead of weekly, but that means 365 or 8760 rows instead of
52 and the spreadsheet gets too large to show here.
Figure 7 shows a typical random output from the simulator
spreadsheet. (There are over 10313 possibilities, so it is highly
unlikely a similar spreadsheet will randomly produce exactly
this same DAFWCF array.) Adding a chart, like in Figure 8,
makes it easier to compare the reported data with hypothetical
data generated by the random number simulation. The
reported data does not change, but the computed data changes
randomly within the described probability constraints.
This simulator demonstrates it is likely that the reported
DAFWCF data was somehow prevented from varying in the
normal fashion. Yes, it is possible that not once in 17 years
would any DAFWCF for one year be higher than the previous
year, but the odds against that happening are comparable to
consecutive lottery wins. Reproducing that situation on this
simulator steady decline without a single year being higher
than the previous year might wear out several F9 keys. In
countless tries, this model never produced a data set without at
least one year having a higher DAFWCF than the previous
year, even though we imposed a 15% decrease in probability
each year.
A more likely explanation for the steady decline shown in
Figures 6 and 7 is that top managers told subordinates
something to the effect, Either you will reduce DAFWCF
each year or your replacement will. Mid level managers
faced with such an ultimatum can become very resourceful in
whatever creative accounting might be necessary to produce

the assigned goal. The net effect for top managers is that their
own policies shield them from problems they need to correct
but never see.
We cannot draw firm conclusions from the example data
because we do not know how many employees were involved
each year, and the possibility of a steady decline is greater
with larger numbers of employees. However, the principle
remains the same, and for companies with only a few thousand
employees it is easily possible to construct a spreadsheet
simulator modeled after this example and check believability
of data. Remember the old adage, When something seems
too good to be true it probably is.
What can we do instead?
The only quantitative measurement that can be directly
related to future performance is based on systematic
evaluation of reported corrective actions. This is a system1,2
that shows great promise and deserves careful investigation by
anyone concerned by the use and misuse of conventional
safety statistics.
Conclusions
The mathematics of statistical analysis are so daunting that
important principles tend to be overlooked, misunderstood, or
ignored by people in charge of safety or accident prevention
programs. Spreadsheets based on standard random number
algorithms can demonstrate, illustrate, or help explain the
following points pertinent to industrial safety management:
1. Conventional safety statistics are trailing indicators that
range from useless to highly misleading when used as
probabilities in forecasting future performance.
2. Programs that rely on conventional safety statistics for
issuing either punishments or rewards are unreliable and
may be counter-productive.
3. Standard measurements used by Governments and
regulatory agencies, such as the OSHA Recordable
Injury Rate, are scientifically indefensible when used for
forecasting purposes.
4. Managers who rely on conventional safety statistics may
be making serious mistakes about effectiveness of their
favorite programs and miss warning signals that changes
are needed. They can use spreadsheet simulators to get
an objective indication of how realistic their goals and
reported results really are.
1

Veley, C.D.: Applying a New HSE Measurement System, paper


SPE 74050 presented at the 2002 SPE International Conference
on Health, Safety, and Environment in Oil and Gas Exploration
and Production, Kuala Lumpur, Malaysia, 20-22 March.
2
Veley, C.D. et al.: A New Method of Measuring Safety
Performance Will Soon Affect the Whole Industry, paper SPE
86741 presented at The Seventh SPE International Conference
on Health, Safety, and Environment in Oil and Gas Exploration
and Production, Calgary, Alberta, Canada, 29-31 March.

www.petroman.ir

You might also like