Handbook All How To Measure Performance

How to Measure Performance
A Handbook of Techniques and Tools
Prepared by the
Training Resources and Data Exchange (TRADE)
Performance-Based Management Special Interest Group
for the
Special Project Group
Assistant Secretary for Defense Programs
and the
Office of Operating Experience, Analysis and Feedback
Assistant Secretary for Environment, Safety and Health
U.S. Department of Energy
October 1995
This document describes activities performed under Contract No. DE-AC05-76OR00033 between the U.S. Department
of Energy and Oak Ridge Associated Universities.
The Training Resources and Data Exchange (TRADE) network is managed by the Oak Ridge
Institute for Science and Education. The Oak Ridge Institute for Science and Education (ORISE) was
established by the U.S. Department of Energy to undertake national and international programs in
science and engineering education, training and management systems, energy and environment
systems, and medical sciences. ORISE and its programs are operated by Oak Ridge Associated
Universities (ORAU) through a management and operating contract with the U.S. Department of
Energy. Established in 1946, ORAU is a consortium of 88 colleges and universities.
The Training and Management Systems Division designs, delivers, and manages training programs for
the U.S. Department of Energy, other federal and state agencies, industries and industrial trade
groups, and the private sector; manages widespread dissemination of training systems and
information; conducts needs analyses; and finds solutions for training and human resource
management problems.
This material resulted from work developed under government contract no. DE-AC05-
76OR00033 and is subject to the following license: A paid-up nonexclusive, irrevocable,
worldwide license in such work to reproduce, prepare derivative works therefrom, distribute
copies to the public five years after October 1995, and perform or publicly display by or for
the government such works. Neither the United States Government nor the U.S. Department
of Energy, nor any of their employees, makes any warranty, expressed or implied, or assumes
any legal liability or responsibility for the accuracy, completeness, or usefulness of any
information, apparatus, product, or process disclosed, or represents that its use would not
infringe on privately owned rights.
©
October 1995
Oak Ridge Associated Universities
TABLE OF CONTENTS
Preface ......................................................................................................................................iii
Overview .....................................................................................................................................v
Section 1: Development Processes
Overview.......................................................................................................................... 1-1
1.1 Performance Measurement Process .......................................................................... 1-3
1.2 Developing Performance Measures–A Systematic Approach................................ 1-35
1.3 Developing Performance Metrics–University of California Approach.................. 1-53
1.4 Performance Indexes................................................................................................ 1-59
Section 2: Data Tools
2.0 Overview................................................................................................................... 2-1
2.1 Data Collection Techniques ...................................................................................... 2-7
2.2 Charts, Graphs, and Diagrams ................................................................................ 2-13
2.3 Presentation Approaches........................................................................................ 2-31
Appendix A: Glossary of Terms........................................................................................... A-1
Appendix B: Performance Measurement Process Case Study...............................................B-1
Appendix C: Sample Performance Measures.........................................................................C-1
Appendix D: Related References........................................................................................... D-1
i
R
TRADE
ii
R
TRADE
PREFACE
Introduction
The Training Resources and Data Exchange (TRADE) Performance-Based Management Special
Interest Group (PBM SIG) was chartered to foster continuous improvement and facilitate the use of
performance-based management techniques within the U.S. Department of Energy (DOE)
community. This handbook has been compiled by the PBM SIG to provide reference material to
assist in the development, utilization, evaluation, and interpretation of performance measurement
techniques and tools to support the efficient and effective management of operations.
Contributors
The following members of the TRADE PBM SIG served as the project team for the compilation of
this handbook:
• Joel Anderson, Los Alamos National Laboratory

• Will Artley, Oak Ridge Institute for Science and Education
• Amanda Denton, Lockheed Martin Energy Systems, Inc.
• Cynthia Eubanks, Lockheed Martin Energy Systems, Inc.
• Jean George, Lawrence Livermore National Laboratory
• Tammra Horning, Lockheed Martin Energy Systems, Inc.
• Bob Lyon, Lockheed Idaho Technologies Company
• Jeff Murphy, Lawrence Livermore National Laboratory
• Barbara Strack, Westinghouse Savannah River Corporation
• Kim Wilson, West Valley Nuclear Services
Other subject matter experts in performance-based management programs who provided information
or participated in the preparation or review of this handbook include Anne Roe, Westinghouse
Savannah River Corporation; Stan Love, retired, Sandia National Laboratories-Albuquerque; Buck
Koonce, University of California; Mary Ayles, Jack Ewing, and R. S. (Bud) Leete, Lockheed Martin
Energy Systems, Inc.; Megan Lohmann, DOE Nevada; TRADE PBM SIG Advisors Paul Krumpe
and Richard Day, DOE Headquarters.
iii
R
TRADE
iv
R
TRADE
O VERVIEW
"How to measure performance?" How often do you ask yourself this question? Once a week? Once
a month? Never? If you're a successful manager in a successful organization, you probably ask
yourself this question every single day. However, measuring performance often isn't easy.
In the performance measurement arena, you don't always (or even often) get the results that you
expect, want, or predict. After expending a great deal of energy collecting information, just when the
results look promising, you find that you're measuring the wrong things.
It doesn't have to be this way. Two key words, although they won't completely solve your
performance measurement problems, can put you on the path to success: disciplined approach. All
too often performance measurement programs, created with the best intentions, fail because they
were short sighted, ill conceived, and unfocused. Most of these ailments can be traced to one source:
the lack of a viable approach to performance measurement from the start.
This handbook offers three such disciplined, systematic approaches.
• The first approach, the Performance Measurement Process, was developed by the DOE
Nevada Family Quality Forum. This approach is quite detailed and outlines an 11-step
process for measuring performance.
• The second approach, Developing Performance Indicators . . . A Systematic Approach, was
used at Sandia National Laboratories. It is less detail-oriented than the first, and uses a
fictitious company, the Hackenstack Firewood Company, for anecdotal purposes.
• The third approach, Developing Performance Metrics-the University of California Approach,
was developed by the University of California. This method is broadest in scope.
Different organizations have different needs. Providing multiple approaches allows an organization
to pick and choose which approach, or combination of approaches, is right for it.
It is important to remember that the approaches previously outlined were developed independently;
they may use different terminology. For instance, what the first approach refers to as a performance
measure may be referred to as a performance indicator in the second approach, or a performance
metric in the third. All three approaches are referring to the same concept; however, each uses a
different nomenclature (in fact, each approach has its own glossary). Fortunately, this causes
problems only when comparing one approach to another, so be careful when you reach this stage.
A sound approach to performance measurement is a necessary ingredient for ensured success, but it
alone is not sufficient. You will also need to know what to do with performance measurement data
once it has been collected. The last few sections of this handbook provide some helpful hints on
proven methods of data analysis and management.
v
R
TRADE
Section 1
Development Processes
SECTION 1: DEVELOPMENT PROCESSES
1.0 OVERVIEW
1.0 Overview
The use of performance measures in business is hardly new. Companies have been measuring costs,
quality, quantity, cycle time, efficiency, productivity, etc., of products, services, and processes as
long as ways to measure those things have existed. What is new to some extent is having those who
do the work determine some of what should be measured in order that they might better control,
understand, and improve what they do.
This section contains information that can help an organization determine what kind of measures it
needs, provide some guidance on what should be measured, and show how to set up a measuring
system. Approaches used by three different sources are included. The basic fundamentals are the
same in each case; however, the specific methods are slightly different. These three approaches
address those who actually do the work in determining the appropriate performance measures.
The concepts introduced here apply anywhere in an organization, from the highest levels of a
company down to the area where a specific task is accomplished. The elements of continuous
improvement are built into the methodologies.
1-1
R
TRADE
1.0 OVERVIEW
1-2
R
TRADE
1.1 PERFORMANCE M EASUREMENT P ROCESS
1.1 Performance Measurement Process
Introduction
Performance measures are recognized as an important element of all Total Quality Management
programs. Managers and supervisors directing the efforts of an organization or a group have a
responsibility to know how, when, and where to institute a wide range of changes. These changes
cannot be sensibly implemented without knowledge of the appropriate information upon which they
are based. In addition, among many organizations within the Department of Energy (DOE) complex,
there is currently no standardized approach to developing and implementing performance
measurement systems. As a result, performance measures have not been fully adopted to gauge the
success of the various quality management programs practiced by members of the Department of
Energy Nevada Operations Office (DOE/NV) Family Quality Forum.
To address these issues, the steering committee members commissioned a work group to study the
development, implementation, and operation of performance measurement systems. This guidance
document, the product of the work group, provides a comprehensive, step-by-step explanation of
how to develop performance measurements at any level within an organization and how to evaluate
their effectiveness.
Appendix A contains a glossary of terms that may be used in this guidance document. The
accompanying Case Study (Appendix B) illustrates a practical example of how to put the concepts
of the guidance document to use. Appendix C contains examples of performance measurements that
can be considered.
The implementation of performance measurements for a specific process should involve as many
cognizant employees as possible to stimulate ideas and reinforce the notion that this is a team effort
requiring buy-in from all involved in order to succeed. Substantial benefits are realized by
organizations implementing performance measurement programs. These benefits are realized almost
immediately through an improved understanding of processes by all employees. Furthermore,
individuals get an opportunity to receive a broadened perspective of the organization's functions,
rather than the more limited perspective of their own immediate span of control.
As a process, performance measurement is not simply concerned with collecting data associated with
a predefined performance goal or standard. Performance measurement is better thought of as an
overall management system involving prevention and detection aimed at achieving conformance of the
work product or service to your customer's requirements. Additionally, it is concerned with process
optimization through increased efficiency and effectiveness of the process or product. These actions
occur in a continuous cycle, allowing options for expansion and improvement of the work process or
product as better techniques are discovered and implemented.
1-3
R
TRADE
Performance measurement is primarily managing outcome, and one of its main purposes is to reduce
or eliminate overall variation in the work product or process. The goal is to arrive at sound decisions
about actions affecting the product or process and its output.
This section, along with Appendix B, is reprinted by permission of the DOE/NV Family Quality
Forum.
What Are Performance Measures?

Performance measures quantitatively tell us something important about our products, services, and
the processes that produce them. They are a tool to help us understand, manage, and improve what
our organizations do. Performance measures let us know:
• how well we are doing

• if we are meeting our goals
• if our customers are satisfied
• if our processes are in statistical control
• if and where improvements are necessary.
They provide us with the information necessary to make intelligent decisions about what we do.
A performance measure is composed of a number and a unit of measure. The number gives us a
magnitude (how much) and the unit gives the number a meaning (what). Performance measures are
always tied to a goal or an objective (the target). Performance measures can be represented by single
dimensional units like hours, meters, nanoseconds, dollars, number of reports, number of errors,
number of CPR-certified employees, length of time to design hardware, etc. They can show the
variation in a process or deviation from design specifications. Single-dimensional units of measure
usually represent very basic and fundamental measures of some process or product.
More often, multidimensional units of measure are used. These are performance measures expressed
as ratios of two or more fundamental units. These may be units like miles per gallon (a performance
measure of fuel economy), number of accidents per million hours worked (a performance measure of
the companies safety program), or number of on-time vendor deliveries per total number of vendor
deliveries. Performance measures expressed this way almost always convey more information than
the single-dimensional or single-unit performance measures. Ideally, performance measures should be
expressed in units of measure that are the most meaningful to those who must use or make decisions
based on those measures.
1-4
R
TRADE
Most performance measures can be grouped into one of the following six general categories.
However, certain organizations may develop their own categories as appropriate depending on the
organization's mission:
1. Effectiveness: A process characteristic indicating the degree to which the process output
(work product) conforms to requirements.(Are we doing the right things?)
2. Efficiency: A process characteristic indicating the degree to which the process produces the
required output at minimum resource cost. (Are we doing things right?)
3. Quality: The degree to which a product or service meets customer requirements and
expectations.
4. Timeliness: Measures whether a unit of work was done correctly and on time. Criteria must
be established to define what constitutes timeliness for a given unit of work. The criterion is
usually based on customer requirements.
5. Productivity: The value added by the process divided by the value of the labor and capital
consumed.
6. Safety: Measures the overall health of the organization and the working environment of its
employees.
The following reflect the attributes of an ideal unit of measure:
• Reflects the customer’s needs as well as our own

• Provides an agreed upon basis for decision making
• Is understandable
• Applies broadly
• May be interpreted uniformly
• Is compatible with existing sensors (a way to measure it exists)
• Is precise in interpreting the results
• Is economical to apply
Performance data must support the mission assignment(s) from the highest organizational level
downward to the performance level. Therefore, the measurements that are used must reflect the
assigned work at that level.
Within a system, units of measure should interconnect to form a pyramid (Figure 1.1).
Technological units start at the base. These are measures of individual units of products and of
individual elements of service.
1-5
R
TRADE
The next level of units serve to summarize the basic data (e.g., percent defective for specific
processes, documents, product components, service cycles, and persons.)
Next are units of measure that serve to express quality for entire departments, product lines, and
classes of service. In large organizations, there may be multiple layers of this category.
At the top are the financial and upper management units (measures, indexes, ratios, etc.), which serve
the needs of the highest levels in the organization: corporate, divisional, and functional.
Figure 1.1
Pyramid Used at All Levels of the Company
Managerial
sensors
consist mostly
Corporate
of composites
Reports:
of data expressed
money, ratios,
in such forms as
index
summaries, ratios,
Measures of broad indexes
matters as quality
Upper management sensors
compare to that of
to evaluate broad matters
competitors; time required
include data systems, reports,
to launch new products
audits, personal observations
Department summaries of
Measures that help to product and process performance,
establish departmental quality derived from inspections and test,
goals and to evaluate departmental reports of nonconformance, etc., and
performance against goals personal observations
Technological units of measure Numerous technological instruments to

for individual elements of measure technological product and
product, process, service process features
Units of Measure Sensors
1-6
R
TRADE
What Are the Benefits of Measurements?

Listed below are seven important benefits of measurements:
1. To identify whether we are meeting customer requirements. How do we know that we are
providing the services/products that our customers require?
2. To help us understand our processes. To confirm what we know or reveal what we don't
know. Do we know where the problems are?
3. To ensure decisions are based on fact, not on emotion. Are our decisions based upon well-
documented facts and figures or on intuition and gut feelings?
4. To show where improvements need to be made. Where can we do better? How can we
improve?
5. To show if improvements actually happened. Do we have a clear picture?
6. To reveal problems that bias, emotion, and longevity cover up. If we have been doing our
job for a long time without measurements, we might assume incorrectly that things are going
well. (They may or may not be, but without measurements there is no way to tell.)
7. To identify whether suppliers are meeting our requirements. Do our suppliers know if our
requirements are being met?
Why Do We Need to Measure?

If you cannot measure an activity, you cannot control it. If you cannot control it, you cannot
manage it. Without dependable measurements, intelligent decisions cannot be made. Measurements,
therefore, can be used for:
1. Control: Measurements help to reduce variation. For example, a typical control for DOE
contractor accountability measurement is the Work Authorization Directive System
(WADS) and Performance Evaluation Plan (PEP). Their purpose is to reduce expense
overruns so that agreed-to objectives can be achieved.
2. Self-Assessment: Measurements can be used to assess how well a process is doing,
including improvements that have been made.
3. Continuous Improvement: Measurements can be used to identify defect sources, process
trends, and defect prevention, and to determine process efficiency and effectiveness, as well
as opportunities for improvement.
4. Management Assessment: Without measurements there is no way to be certain we are
meeting value-added objectives or that we are being effective and efficient. The basic
concept of performance measurement involves (a) planning and meeting established
operating goals/standards; (b) detecting deviations from planned levels of performance; and
(c) restoring performance to the planned levels or achieving new levels of performance.
1-7
R
TRADE
What Is the Foundation for a Performance Measurement System?

Successful performance measurement systems adhere to the following principles:
1. Measure only what is important. Do not measure too much; measure things that impact
customer satisfaction.
2. Focus on customer needs. Ask our customers if they think this is what we should measure.
3. Involve employees (workers) in the design and implementation of the measurement system.
Give them a sense of ownership, which leads to improvements in the quality of the
measurement system.
The basic feedback loop shown in Figure 1.2 presents a systematic series of steps for maintaining
conformance to goals/standards by communicating performance data back to the responsible worker
and/or decision maker to take appropriate action(s).
Without the basic feedback loop, no performance measurement system will ever ensure an effective
and efficient operation, and, as a result, conformance to customers' requirements.
The message of the feedback loop is that to achieve the goal or standard, those responsible for
managing the critical activity(ies) must always be in a position to know (a) what is to be done; (b)
what is being done; (c) when to take corrective action; and (d) when to change the goal or standard.
The basic elements of the feedback loop and their interrelations are:
1. The Sensor evaluates actual performance.

2. The Sensor reports this performance to a Responsible Worker.
3. The Responsible Worker also receives information on what the goal or standard is.
4. The Responsible Worker compares actual performance to the goal. If the difference
warrants action, the worker reports to a Responsible Decision Maker. (This could signal a
need for corrective action.)
5. The Responsible Decision Maker verifies variance, determines if corrective action is
necessary, and, if so, makes the changes needed to bring performance back in line with the
goals.
1-8
R
TRADE
Figure 1.2
Basic Feedback Loop
Control Activity Sensor Goal/Standard

1 (desired performance
(what we intend (measures actual performance
to regulate) in terms of the unit in terms of the unit
of measure) of measure)
2 3
5
Responsible Worker(s)
Responsible Decision
4 (compare actual performance
Maker
with goal or standard;
(makes changes needed if difference warrants
to bring performance action, reports to
into line with goal/standard)
responsible decision maker)
Process Overview
Figure 1.3 shows a high level block diagram of the performance measurement process. It has been
separated into 11 discrete steps. This is a guideline, intended to show the process generically.
Different organizations who best know their own internal processes should feel free to adapt the
guidelines where necessary to best fit within their operations. Subcomponents within the steps may
need to be exchanged, or it may be necessary to revisit completed steps of the process based on new
information arising from latter steps.
A brief description of each of the process steps follows:
1. Identify the process flow. This is the first and perhaps most important step. If your
employees cannot agree on their process(es), how can they effectively measure them or
utilize the output of what they have measured?
2. Identify the critical activity to be measured. The critical activity is that culminating activity
where it makes the most sense to locate a sensor and define an individual performance
measure within a process.
1-9
R
TRADE
3. Establish performance goal(s) or standards. All performance measures should be tied to a

predefined goal or standard, even if the goal is at first somewhat subjective. Having goals
and standards is the only way to meaningfully interpret the results of your measurements
and gauge the success of your management systems.
4. Establish performance measurement(s). In this step, you continue to build the performance
measurement system by identifying individual measures.
5. Identify responsible party(s). A specific entity (as in a team or an individual) needs to be
assigned the responsibilities for each of the steps in the performance measurement process.
6. Collect data. In addition to writing down the numbers, the data need to be pre-analyzed in a
timely fashion to observe any early trends and confirm the adequacy of your data collection
system.
7. Analyze/report actual performance. In this step, the raw data are formally converted into
performance measures, displayed in an understandable form, and disseminated in the form
of a report.
8. Compare actual performance to goal(s). In this step, compare performance, as presented in
the report, to predetermined goals or standards and determine the variation (if any).
9. Are corrective actions necessary? Depending on the magnitude of the variation between
measurements and goals, some form of corrective action may be required.
10. Make changes to bring back in line with goal. This step only occurs if corrective action is
expected to be necessary. The actual determination of the corrective action is part of the
quality improvement process, not the performance measurement process. This step is
primarily concerned with improvement of your management system.
11. Are new goals needed? Even in successful systems, changes may need to be revised in order
to establish ones that challenge an organization’s resources, but do not overtax them. Goals
and standards need periodic evaluation to keep up with the latest organizational processes.
1 - 10
R
TRADE
Figure 1.3
Performance Measurement Process
High Level Block Diagram
Steps Responsibility Activities
Start
Identify
1 1
process
Identify
2 critical activity 2
to be measured
Establish
3 performance 3
goal(s)/or standards
4 Establish 4
performance measurement(s)
Identify
5 5
responsible party(ies)
6 Collect data Database 6
Analyze/report
7 7
actual performance
Compare actual
8 performance to goal(s) 8
Are
NO corrective
9 actions 9
necessary?
YES
YES Make changes to

10 bring back in 10
Are in line with goal
new NO
11 goals
needed? 11
1 - 11
R
TRADE
Step 1: Identify Process

In identifying the process, an understanding of what you want to measure is of critical importance.
Usually there are many processes and functions, each potentially needing performance measures. If
there are multiple processes, consider the business impacts, and select those processes that are most
important to the customer (both internal and external) to satisfy their requirements and/or those
processes with problem areas identified by management. These then become the key (or important)
processes.
A process needs to be manageable in size. A lot of effort can be wasted if you do not start with a
well-defined process. You should ask the following:
A. What product or service do we produce?

B. Who are our customer(s)?
C. What comprises our process?
• What do we do?
• How do we do it?
• What starts our process?
• What ends our process?
Before you try to control a process, you must understand it. A flow diagram is an invaluable tool
and the best way to understand a process. Flowcharting the entire process, down to the task level,
sets the stage for developing performance measures.
All parties who are involved in the process should participate in creating the flowcharts. In a team
environment, individuals will receive a new understanding of their processes. As participants, you
can count on their later support to make the performance measurement system work.
OUTPUT: A LIST OF PROCESSES, KEY PROCESSES, AND

FLOW DIAGRAMS FOR THESE KEY
PROCESSES.
1 - 12
R
TRADE
Step 2: Identify Critical Activity(ies) to be Measured

It is important to choose only the critical activity(ies) to be measured. We measure these activities
to control them. Controlling, or keeping things on course, is not something we do in the abstract.
Control is applied to a specific critical activity. When making your selection, focus on key areas and
processes rather than people.
Examine each activity in the process and identify those that are critical. Critical activities are those
that significantly impact total process efficiency, effectiveness, quality, timeliness, productivity, or
safety. At the management level, critical activities impact management priorities, organizational
goals, and external customer goals.
Ask the following: Does it relate, directly or indirectly, to the ultimate goal of customer satisfaction?
Every critical activity should. For example, on-time delivery is directly related to customer
satisfaction. Use quality tools such as the Pareto principle, brainstorming, or examining data to help
prioritize the critical activities.
Confirm that the activity is critical. Do all concerned agree that this activity needs to be watched
closely and acted on if its performance is less than desirable? Is it something that should be
continuously improved? Does the benefit exceed the cost of taking the measurement? If the answer
is "no" to any of these questions, you should reevaluate why you consider it critical.
Each critical activity becomes the hub around which a feedback loop is constructed. (Figure 1.2)
It is at this step you begin to think about what you want to know or understand about the critical
activity and/or process. Perhaps the most fundamental step in establishing any measurement system
is answering the question, “What do I want to know.” The key issue then becomes, "How do we
generate useful information?" Learning to ask the right questions is a key skill in effective data
collection. Accurate, precise data collected through an elaborately designed statistical sampling plan
is useless if it does not clearly address a question that someone cares about. It is crucial to be able to
state precisely what it is you want to know about the activity you are going to measure. Without
this knowledge, there is no basis for making measurements.
To generate useful information, planning for good data collection proceeds along the following lines:
• What question do we need to answer?

• How will we recognize and communicate the answers to the question?
• What data-analysis tools (Pareto diagram, histogram, bar graph, control charts, etc.) do we
envision using? How will we communicate the results?
• What type of data do the data analysis tools require?
1 - 13
R
TRADE
• Where in the process can we get these data?

• Who in the process can give us these data?
• How can we collect these data from people with minimum effort and chance of error?
• What additional information do we need to capture for future analysis, reference, and
tractability?
Notice how this planning process (Figure 1.4) essentially works backward through the model for
generating useful information. We start by defining the question. Then, rather than diving into the
details of data collection, we consider how we might communicate the answer to the question and
what types of analysis we will need to perform. This helps us define our data needs and clarifies
what characteristics are most important in the data. With this understanding as a foundation, we can
deal more coherently with the where, who, how, and what else issue of data collection.
OUTPUT: A LIST OF THE CRITICAL ACTIVITY AREAS FOR

THE KEY PROCESS.
Figure 1.4
Model for Generating Useful Information
Generating Information
Information Needs
Questions
Communication Data
Analysis
1 - 14
R
TRADE
Information generation begins and ends with questions. To generate information, we need to:
• Formulate precisely the question we are trying to answer.

• Collect the data and facts relating to that question.
• Analyze the data to determine the factual answer to the question.
• Present the data in a way that clearly communicates the answer to the question.
Step 3: Establish Performance Goal(s) or Standard(s)

Goals and standards are necessary; otherwise there is no logical basis for choosing what to measure,
what decisions to make, or what action to take. Goals can be a management directive or can be set in
response to customer needs or complaints. Know your customers and their expectations. For each
critical activity selected for measurement, it is necessary to establish a performance goal or standard.
This is an "aimed-at" target, an achievement toward which effort is expended. Standards often are
mandated by external sources (e.g., Occupational Safety and Health Administration [OSHA],
government regulations, etc.). Knowledge of performance is not enough; you must have a basis for
comparison before you can decide or act.
The concept of establishing performance goals/standards is not limited to numbered quantities, i.e.,
budget, deliveries. Neither is it limited to "things." The concept of standards extends to business
practices, routines, methods, and procedures as well.
Performance goals can be established for (1) the overall process output, and/or, (2) the critical
activities that produce the output. In any case, if this is the first set of goals or standards to be
established, and no basis for setting goals or standards exists, a baseline period of observation is
appropriate prior to establishing the goal or standard.
Good performance goals or standards are:
• Attainable: Should be met with reasonable effort under the conditions that are expected to
prevail.
• Economic: Cost of setting and administering should be low in relation to the activity
covered.
• Applicable: Should fit the conditions under which they are to be used. If conditions vary,
should contain built-in flexibility to meet these variables.
• Consistent: Should help to unify communication and operations throughout all functions of
the company.
1 - 15
R
TRADE
• All-inclusive: Should cover all interrelated activities. Failing this, standards will be met at the
expense of those activities for which standards have not been set.
• Understandable: Should be expressed in simple, clear terms, so as to avoid misinterpretation
or vagueness. Instructions for use should be specific and complete.
• Measurable: Should be able to communicate with precision.
• Stable: Should have a long enough life to provide predictability and to amortize the effort of
preparing them.
• Adaptable: Should be designed so that elements can be added, changed, and brought up to
date without redoing the entire structure.
• Legitimate: Should be officially approved.
• Equitable: Should be accepted as a fair basis for comparison by the people who have the job
of meeting the goal or standard.
• Customer Focus: Should address areas important to the customer (internal/external) such as
cycle time, quality, cost schedule performance, and customer satisfaction.
OUTPUT:A LIST OF GOALS FOR EACH CRITICAL

ACTIVITY WITHIN THE PROCESS
Step 4: Establish Performance Measurement(s)

This step involves performing several activities that will continue to build the performance
measurement system. Each performance measurement consists of a defined unit of measure (the
performance measure itself), a sensor to measure or record the raw data, and a frequency with which
the measurements are made. To develop a measure, the team performs the following activities:
• translates “what do I want to know” into a performance measure

• identifies the raw data that will generate the performance measure
• determines where to locate the raw data
• identifies the sensor or measurement instrument that will collect the data for the performance
measures
• determines how often to make the measurements
1 - 16
R
TRADE
At this point, your team has agreed upon which process to measure (Step 1), identified the critical
activities of your process with emphasis on those that impact quality, efficiency, timeliness,
customer satisfaction, etc. (Step 2), looked at goals for these activities, products, and services (where
they exist), and has quantified these goals where possible (Step 3). Your team should use the
knowledge gained from these previous steps to help state precisely what you want to know about
the critical activities or the process as a whole. Think of this step as one that will allow you to
generate useful information rather than just generating data. The purpose of this information is to
provide everyone involved with an agreed-upon basis for making sensible decisions about your
processes, products, and services. Don’t move on until the team agrees on what information you are
trying to extract from the measurements.
Translate into Performance Measures

Having identified precisely what you want to know or understand about your process, you must
now assemble this knowledge into a performance measure. Performance measures, and the data
necessary to generate them, should be chosen to answer the questions you have just posed above.
At this point, your team must decide how you will “say it in numbers.”
Performance measures are generally easiest to determine for activities or processes that have
established and quantified goals. In such cases, the performance measures are usually stated in the
same units as or similar units to the goals.
When no goals exist for an activity (or the process as a whole), the team should revisit the
fundamental question of what it is they wish to know. The performance measures should provide
quantitative answers to their questions in units that relate to those questions. The team may wish to
reread What Are Performance Measures? on Page 1 - 4 to reinforce the concept of a unit of
measure and what it should convey.
The following example of a vendor selection process should prove useful in illustrating how to turn a
question posed into one of a possible performance measure:
You are part of a work team within the procurement department of your company.
Over the years the number of vendors from which you make purchases has grown
astronomically and you need some basis upon which to help decide which vendors
perform the best. You have concluded that one of the more fundamental questions
you would like to answer is “how well do our vendors meet the contract delivery
dates?” Your team needs to choose a performance measure that will help answer this
question. After putting several possible performance measures on a flip chart and
examining what information each could convey, the team decided to use:
% on-time deliveries per month
1 - 17
R
TRADE
To ensure the team understood what this measure will provide them, they rewrote
this measure in terms of the units that are actually used to calculate it. The
performance measure then looks like this:
number of on-time deliveries per month X 100%

total number of deliveries per month
Both versions of the performance measure are essentially the same, but the second
actually conveys more information to the reader and provides an indication of what
data goes into the measurement. This performance measure should help the team
answer the question of how well vendors are meeting contract delivery dates. By
writing their performance measure in more fundamental units, the team will be better
prepared to move to the next activity, which is identifying the raw data needed.
A good way to “test” a team’s understanding of the performance measures they have chosen is to
have them describe how they would display their results graphically. Have the team explain what
type of graph they would use for each performance measure and how they would interpret the
results. Quite often, seeing a performance measure displayed graphically will help determine if it will
actually provide the information needed. Doing this simple step now will help ensure the team that
it has chosen the right performance measure.
In reality, many work teams may find that some of their performance measures do not really tell
them what they want to know. Don’t panic, even performance measures that don’t quite work may
help refocus the team on the real issues they hope to address. Introduce a new set of measures and
try again.
Identify the Raw Data

The purpose of this activity is to identify the raw data you will need to generate the performance
measures. It is difficult to perform a measurement if the needed data and data source have not been
identified. For very simple processes with straightforward performance measures, this step may
seem simple. However, very complex or high-level performance measures may require many raw
data from numerous sources. In general, performance measures are seldom generated directly in a
single measurement or from a single source. They usually (but not always) consist of some
combination of other raw data elements as in the example above. To illustrate the difference,
consider the following examples:
1. Your workgroup enters data from customer order forms into an electronic database. Your
group decided that the number of errors per day was a useful performance measure for that
process. The raw data for your measurement consist of counting the errors in the database
each day. In this case, the collection of raw data needed is the performance measure and it
has been measured directly.
1 - 18
R
TRADE
2. You are in the procurement department and your team has decided to use the percent of on-
time deliveries per month of key vendors as a performance measure. The raw data you need
consist of four sets. First, you need the delivery date on the contract that was awarded to
your vendors. Second, you need the date the delivery was made. Third, you must compute
if the delivery was on time and count how many deliveries there were for each vendor.
Fourth, the team will need the total number of deliveries made within the month for that
vendor. Unlike example 1, several elements of raw data are required to reconstruct the
performance measure.
3. Your management team considers the company’s overhead burden rate to be an excellent
high-level performance measure. This measure is very complex and is frequently performed
by the company’s accountants and budget analysts. Such measures require many raw data
elements that consist of facilities costs, human resource benefits, training costs, rework
costs, sales income, and so on. This performance measure requires that many lower level
measures are taken and “rolled up” into the higher level measure. Many data elements must
be collected along the way.
When the team completes this activity, it should have a list of the raw data elements needed to
generate the performance measures. In addition, the team should consider what, if any, computations
or calculations must be performed with or on the data.
Locate the Raw Data

The purpose of this activity is to determine if and where the data exist. Stated differently, it’s a
matter of locating at what step in a process to make a measurement, at what point in time, or at what
physical or geographical location. Quite often, this activity is performed concurrently with the
previous one.
In the simplest case you may find that your work group already has the raw data collected and you
need only retrieve in order to generate the associated performance measure. In other cases, the data
you need may have been collected by another department. For instance, in Example 2 above, the
delivery date was probably collected by the Shipping and Receiving Department. Examine the data
you need and determine if your own work group, department, or an external group is already
collecting it.
If the data do not presently exist, the team will have to determine where to find it. The process of
locating it is generally quite straightforward. This is particularly true if the team is measuring its own
process. The measurement point is usually located at or near each critical activity that was identified
in Step 2. This is generally the case if your performance measure is measuring an activity within the
process rather than the overall process itself. For performance measures that assess some overall
aspect of a process, the collection point usually occurs at the culmination of a process.
1 - 19
R
TRADE
More global performance measures generally require data from many sources as in Example 3 above.
Before proceeding, the team must determine where the data are located, or where in the process, at
what point in time, and at what physical location the data will be collected.
Continuing with the procurement example, we would probably find something like the following take
place:
The team has determined what raw data they will need to construct their performance
measure and now they must locate the data. In this example the process is rather
simple. The contract delivery date is recorded within the procurement department
itself on several documents and within a database so that retrieval will be trivial. The
second data element is recorded by the Shipping and Receiving department and is
likewise simple to extract. All that remains to reconstruct the performance measure
are the computations with the data.
Identify the Sensor

By this point, the team has determined what raw data they require, where it is located, and where it
will be collected. To proceed, they must determine how they will actually measure or collect what
they need. A sensor is required to accomplish the measurement.
A sensor is a device or person that is able to detect (sense) the presence or absence of some
phenomena and (generally) provide a reading of the intensity (how much) of that phenomena in a
quantifiable form with the appropriate units of measure. The sensor is what or who will do the
measuring or data collection for your measurement system.
Sensors take many forms depending on what they are designed to measure. For technical and
manufacturing processes, there are sensors that can accurately measure length (micrometer),
temperature (thermocouple), voltage (digital voltmeter or digitizer), and so on. For less technical
processes there are databases, log books, time cards, and checksheets. In some cases, the sensor
makes a measurement and a person records the results. In other cases, only a human is capable of
“sensing” some phenomena and some other device is used to record the result. Many inspection
activities can only be performed by humans. There are also automated data collection systems or
sensors that require no human intervention other than calibration or maintenance. Many
manufacturing processes employ such sensors to detect, measure, and record the presence of
nonstandard products.
1 - 20
R
TRADE
Choosing a sensor usually involves asking simple questions about the measurement you hope to
make:
1. What am I trying to measure; what kind of data are they?

2. Where will I make the measurement; where are the data?
3. Am I simply trying to measure the presence or absence of some feature? (Was the order
placed, was the report delivered? Did the computer help desk solve the problem?)
4. Do I need to sense the degree or magnitude of some feature or count how many?
5. How accurate and precise must my measurements be?
6. Do the measurements occur at a particular point in time or space?
In most cases, the sensor will be rather obvious, but the team should be prepared to give some
thought to how they will measure and collect their data. For instance, the need for accuracy and/or
precision may rule out certain sensors. If you rely on a human as a sensor, you must consider all the
possible biases that are inherent in human sensors. Step 6 discusses biases and their potential
solutions. Replacing human sensors with technological instruments may be the best solution if a
particularly critical measurement requires unusual accuracy or precision.
When the team completes this step, it should have a sensor identified for each raw data element and
should have determined where the sensor will be deployed.
The procurement team determined that the sensor for their first data element (contract
delivery date) would be the “buyer’s diary,” an electronic database maintained by
each buyer in company-supported software. The second sensor was determined to be
the “receiving log,” which was maintained at the receiving dock by a receiving clerk.
This sensor provided the actual delivery date for each order. Having identified the
sensors, the team could now acquire the necessary data.
Determine How Often to Make Measurements

In this last activity, the team will determine how often measurements should be made. In a sense,
there are two distinct types of measures taken when a performance measurement system is adopted.
One type of measure is the performance measure itself. This measure is generally taken (calculated)
and reported over some regular or repeating time interval. In the procurement example, the
performance measure is calculated and presumably reported at a frequency of once per month. Some
performance measures are used to observe real-time trends in a process and may be measured and
plotted daily. In general, the frequency of measurement for the performance measure is usually
determined when the performance measure itself is determined. Often the unit of measure chosen as
the performance measure contains or alludes to the frequency of measurement.
1 - 21
R
TRADE
The other measure that should be addressed is that of the raw data itself. The frequency with which
raw data are collected or measured may have a significant impact upon the interpretation of the
performance measure. For some performance measures, this amounts to asking how many data are
needed to make the measure valid or statistically significant. Each team or manager will have to
determine how often measurements must be made (data taken) to ensure statistical significance and
believable results.
Again, using the procurement example, the raw data for this measure are each time a buyer enters
contract data into the database and each time the receiving clerk logs in a delivery. It could be said
then that the frequency of measurement or data collection is continuous; that is data are rewarded
each time a transaction or delivery occurs.
Processes that are repeated numerous times per hour or may only require a sample measure of every
tenth event or so. Other events, like the procurement example, are measured or recorded each time
they happen. Teams should use their best judgment in choosing the frequency of data collection and
should consult the company’s statistician or quality consultant if there is some question.
OUTPUT: THE PERFORMANCE MEASURE AND ITS

COMPONENTS
Step 5: Identify Responsible Party(ies)

Steps 1 through 4 are primarily team activities. To continue the performance measurement process,
the responsible worker(s) and the responsible decision maker must be defined. (In some instances,
one person may be responsible for the entire system.) It is now appropriate to determine who
should:
• Collect the data

• Analyze/report actual performance
• Compare actual performance to goal/standard
• Determine if corrective action is necessary
• Make changes
1 - 22
R
TRADE
Ideally, responsibility should be assigned to individuals commensurate with authority. This means
that each responsible party should:
1. Know what the goals are

2. Know what the actual performance is
3. Have the authority to implement changes if performance does not conform to goals and
standards
To hold someone responsible in the absence of authority prevents them from performing their job
and creates the risk of unwarranted blame.
OUTPUT:A LIST OF PEOPLE AND THEIR AREAS OF

RESPONSIBILITY.
Step 6: Collect Data

The determination of conformance depends on meaningful and valid data. Before you start out to
collect a lot of new data, it is always wise to look at the data you already have to make certain you
have extracted all the information you can from it. In addition, you may wish to refer back to Step 2
and review planning for good data collection.
Information, as a term, comprises the answers to your questions. Data are a set of facts presented
in quantitative or descriptive form. Obviously, data must be specific enough to provide you with
relevant information. There are two basic kinds of data:
• Measured or variables data: Data that may take on any value within some range. This type
of data provides a more detailed history of your business process. This involves collecting
numeric values that quantify a measurement and therefore require small samples. If the data
set is potentially large, consider recording a representative sample for this type of data.
Examples:
◊ Cost of overnight mail

◊ Dollar value of stock
◊ Number of days it takes to solve a problem
◊ Diameter of a shaft
1 - 23
R
TRADE
◊ Number of hours to process an engineering change request

◊ Number of errors on a letter
• Counted or attribute data: Data that may take on only discrete values. Attribute data need
not be numeric. These kinds of data are counted, not measured, and generally require large
sample sizes to be useful. Counting methods include defective/nondefective; yes/no;
accept/reject.
Examples
◊ Was the letter typed with no errors?

◊ Did the meeting start on time?
◊ Was the phone answered by the second ring?
◊ Was the report turned in on schedule?
A system owner needs to supervise the data collection process to determine if the data is being
collected properly; if people are doing their assignments. Some form of preliminary analysis is
necessary during the data collection process. Is your measurement system functioning as designed?
Check the frequency of data collection. Is it often enough? Is it too often? Make adjustments as
necessary and provide feedback to the data collectors.
Data Collection Forms

There are two types of forms commonly used to aid in data collection. Often combinations of these
are used:
• Checksheet: A form specially designed so that the results can be readily interpreted from
the form itself. This form of data collection is ideal for capturing special (worker controlled)
cause of process variation since the worker can interpret the results and take corrective
actions immediately.
• Data Sheet: A form designed to collect data in a simple tabular or column format (often
related to time-dependent data). Specific bits of data–numbers, words, or marks–are entered
in spaces on the sheet. As a result, additional processing is typically required after the data
are collected in order to construct the tool needed for analysis. This form of data collection
is usually used for capturing common (manager controlled) causes of process variations.
1 - 24
R
TRADE
Data Collection System

This system ensures that all of our measurements are collected and stored. The type of data and
frequency of measurement will help you determine how to collect it. Some data fit well into
checksheets or data sheets that collect information in simple tabular or columnar format.
Other measurements lend themselves to easy entry into a computer database. Whatever system is
chosen should provide easy access and be understandable by those who are tasked with reviewing
the data. Those tasked with performing the data collection should understand the data collection
system, have the necessary forms at hand, be trained in the data collection, and have access to
instructions pertaining to the system.
The data collected needs to be accurate. Inaccurate data may give the wrong answer to our
information questions. One of the most troublesome sources of error is called bias. It is important
to understand bias and to allow for this during the development and implementation of any data
collection system. Design of data collection forms and processes can reduce bias.
Some types of biases that may occur:
• Exclusion–some part of the process or the data has been left out of the data collection
process
• Interaction–the data collection itself interferes with the process it is measuring
• Perception–the data collector biases (distorts) the data
• Operational–the data collection procedures were not followed or were specified incorrectly
or ambiguously
• Nonresponse–some of the data are missing or not obtained
• Estimation–statistical biases
• Collection time period–the time period or frequency selected for data collection distorts the
data, typically by missing significant events or cyclic occurrences.
OUTPUT:A GROWING LIST OF DATA. DATA SHOULD BE

MONITORED AS THEY ARE BEING COLLECTED.
1 - 25
R
TRADE
Step 7: Analyze/Report Actual Performance

Before drawing conclusions from the data, you should verify that the data collection process has met
the following requirements:
• Review the information questions that were originally asked. Do the data collected still
appear to answer those questions?
• Is there any evidence of bias in the collecting process?
• Is the number of observations collected the number specified? If not, why?
• Do you have enough data to draw meaningful conclusions?
Once the raw data are collected and verified, it is time for analysis. In most instances, your recorded
data are not necessarily the actual performance measurement. Performance measurements are usually
formulated based on one or more raw data inputs. Therefore, you need to assemble the raw data into
a performance measurement.
The next step in analyzing data is deciding how you are going to present or display the data. You
usually group the data in a form that makes it easier to draw conclusions. This grouping or
summarizing may take several forms: tabulation, graphs, or statistical comparisons. Sometimes,
single data grouping will suffice for the purposes of decision making. In more complex cases, and
especially where larger amounts of data must be dealt with, multiple groupings are essential for
creating a clear base for analysis.
After summarizing your data, you develop your report. A number of tools are available to assist
you. Below are some of the more widely used tools and concepts to help you in your reporting.
• Use spread sheets and databases as appropriate to organize and categorize the data and to
graphically show the trends. This will greatly improve the ease and quality of
interpretation. Some of the more common graphic presentations are histograms, bar charts,
pie charts, scatter diagrams, and control charts.
• Make the report comparative to the goals.
• Make use of summaries. The common purpose is to present a single important total rather
than many subtotals. Through this summary, the reader is able to understand enough to
judge whether to go into detail or to skip on to the next summary.
• Be aware of pitfalls in your data presentation. Averaging your data on a monthly basis
might shorten the amount of information presented, but could hide variations within the
monthly period. Choices of scales on graphs and plots could skew interpretation.
1 - 26
R
TRADE
• Standardize the calendar so that the month begins and ends uniformly for all reports. Failing
this, the relation of cause to effect is influenced by the fact that events tend to congest at the
end of the reporting period.
• Adopt a standard format. Use the same size of sheets or charts. As far as possible, use the
same scales and headings.
Reports may take many forms. However, at this stage, the report is intended to be a status transfer
of information to the responsible decision maker for the process. Therefore, the report will likely
consist of sets of tables or charts that track the performance measures, supplemented with basic
conclusions.
OUTPUT: A PRESENTATION OF THE DATA IN THE FORM

OF A REPORT.
Step 8: Compare Actual Performance to Goal/Standard

Within their span of control, responsible workers compare actual performance with the goal or
standard. If variance warrants action, a report is made to the responsible decision maker.
Once the comparison against the goal or standard is initially established, you have several alternatives
available for possible actions. You can decide to:
• Forget it. Variance is not significant.

• Fix it. (Step 9 and 10)
• Challenge the goal or standard. (Step 11)
If there is no significant variance, then continue the data collection cycle. If there is a variance
between the goal and the performance measure, look at the magnitude. If it is significant, report to
the decision maker. If a decision to implement a corrective action is warranted, go to Step 9.
OUTPUT: DECISION BASED ON PERFORMANCE VARIANCE.
1 - 27
R
TRADE
Step 9: Determine if Corrective Action(s) is Necessary

Step 9 is a decision step. You can either change the process or change the goal. If the variance is
large, you may have a problem with your process and will need to make corrections to bring the
performance back into line with the desired goal or standard. To address these potential problems,
you can form a quality improvement team or do a root cause analysis to evaluate. Consider, too, that
the goal may have been unrealistic.
If the variance is small, your process is probably in good shape. But, you should consider
reevaluating your goals to make them more challenging. In addition, if you do make changes to the
process, you will need to reevaluate goals to make sure they are still viable.
The key objectives of correction are:
1. To remove defects; in many cases this is worker-controllable.

2. To remove the cause of defects. Dependent upon the defect cause, this may be worker or
management controllable.
3. To attain a new state of process performance, one that will prevent defects from happening.
4. To maintain or enhance the efficiency and effectiveness of the process. This is an essential
condition for continuing process improvement and ultimately increasing the competitiveness
and profitability of the business itself.
OUTPUT: ACTION PLAN TO IMPLEMENT CHANGES OR

REEVALUATE GOALS (STEP 11).
Step 10: Make Changes to Bring Process Back in Line with Goal or Standard
This is the final step in closing the feedback loop: Making changes to bring the process back in line
with the goal or standard. Changes comprise a number of actions that are carried out to achieve one
or more of the correction objectives listed in Step 9.
The prime result of these corrective actions should be removal of all identified causes of defects
resulting in an improved or a new process.
OUTPUT: A SUCCESSFULLY IMPLEMENTED PLAN.
1 - 28
R
TRADE
Step 11: Determine if New Goals or Measures are Needed

The decision to create new performance measures or goals will depend on three major factors:
1. The degree of success in achieving previous objectives.

2. The extent of any change to the scope of the work processes.
3. The adequacy of current measures to communicate improvement status relative to critical
work processes.
Goals need to be challenging, but also realistically achievable. If previously set objectives were
attained with great difficulty, or not reached at all, then it may be reasonable to re-adjust
expectations. This also applies to the objectives that were too easily met. Extensive scope changes
to the work processes will also necessitate establishing new performance measures and goals.
Changes in performance measures and goals should be considered annually and integrated into
planning and budgeting activities.
OUTPUT: NEW GOALS, MEASURES, OR NO CHANGE.
GLOSSARY: These definitions apply to terminologies used in Section 1.0 Development Processes:
Accuracy: The closeness of a measurement to the accepted true value. The smaller the
difference between the measurement and the true value, the more accurate the measurement.
Attribute Data: Data that may take on only discrete values; they need not be numeric. These
kinds of data are counted, not measured, and generally require large sample sizes to be
useful.
Bias (of measurement): A tendency or inclination of outlook that is a troublesome source of
error in human sensing.
Checksheet: A form specially designed so that results can be readily interpreted from the form
itself.
Continuous Improvement: The ongoing improvement of products, services, and processes
through incremental and measurable enhancements.
Control: The set of activities employed to detect and correct variation in order to maintain or
restore a desired state of conformance with quality goals.
Corrective Action: Measures taken to rectify conditions adverse to quality and, where
necessary, to preclude repetition.
1 - 29
R
TRADE
Critical Activity: Activity(ies) that significantly impact total process efficiency, effectiveness,
quality, timeliness, productivity, or safety. At the management level, they impact
management priorities, organizational goals, and external customer goals.
Customer: An entity that receives products, services, or deliverables. Customers may be
either internal or external.
Data: Information or a set of facts presented in descriptive form. There are two basic kinds of
data: measured (also known as variables data) and counted (also known as attribute data).
Data Collection System: A broadly defined term indicating that set of equipment, log books,
data sheets, and personnel used to record and store the information required to generate the
performance measurements of a process.
Data Sheet: A form designed to collect data in a simple tabular or column format. Specific bits
of data–numbers, words, or marks–are entered in spaces on the sheet. Additional
processing is typically required after the data are collected in order to construct the tool
needed for analysis.
Defect: A nonconformance to the product quality goals; it leads to customer dissatisfaction.
DOE/NV Family: DOE/NV, DOE/NV Contractors, and users of DOE/NV facilities.
Effectiveness: A process characteristic indicating the degree to which the process output (work
product) conforms to requirements.
Efficiency: A process characteristic indicating the degree to which the process produces the
required output at minimum cost.
Feedback: Communication of quality performance to sources that can take appropriate action.
Feedback Loop: A systematic series of steps for maintaining conformance to quality goals by
feeding back performance data for evaluation and corrective action. This is the basic
mechanism for quality control.
Frequency: One of the components of a performance measurement that indicates how often
the measurement is made.
Goal: A statement of attainment/achievement that is proposed to be accomplished or attained
with an implication of sustained effort and energy.
Management Assessment: The determination of the appropriateness, thoroughness, and
effectiveness of management processes.
Optimum: A planned result that meets the needs of customer and supplier alike, meets
competition, and minimizes the customer’s and supplier’s combined costs.
Organization: Any program, facility, operation, or division.
Performance Measure: A generic term encompassing the quantitative basis by which
objectives are established and performance is assessed and gauged. Performance measures
include performance objectives and criteria (POCs), performance indicators, and any other
means that evaluate the success in achieving a specified goal.
1 - 30
R
TRADE
Performance Measurement Category: An organizationally dependent grouping of related

performance measures that convey a characteristic of a process, such as cycle time.
Performance Measurement System: The organized means of defining, collecting, analyzing,
reporting, and making decisions regarding all performance measures within a process.
Precision: The closeness of a group of repeated measurements, to their mean value. The
smaller the difference between the group of repeat measurements and the mean value, the
more precise the instrument. Precision is an indicator of the repeatability, or consistency,
of the measurement.
Process: Any activity or group of activities that takes an input, adds value to it, and provides
an output to a customer. The logical organization of people, materials, energy, equipment,
and procedures into work activities designed to produce a specified end result (work
product).
Productivity: The value added by the process divided by the value of the labor and capital
consumed.
Quality: The degree to which a product or service meets customer requirements and
expectations.
Raw Data: Data not processed or interpreted.
Safety: Measures the overall health of the organization and the working environment of its
employees.
Self Assessment: The continuous process of comparing performance with desired objectives
to identify opportunities for improvement. Assessments conducted by individuals, groups,
or organizations relating to their own work.
Sensor: A specialized detecting device designed to recognize the presence and intensity of
certain phenomena and to convert this sensed knowledge into information.
Timeliness: Measures whether a unit of work was done correctly and on time. Criteria must
be established to define what constitutes timeliness for a given unit of work. The criterion
is usually based on customer requirements.
Unit of Measure: A defined amount of some quality feature that permits evaluation of that
feature in numbers.
Validation: A determination that an improvement action is functioning as designed and has
eliminated the specific issue for which it was designed.
Variable Data: Data that may take on any value within some range. It provides a more
detailed history of a business process. This involves collecting numeric values that quantify
a measurement and therefore requires small samples.
Variance: In quality management terminology, any nonconformance to specification.
Verification: The determination that an improvement action has been implemented as
designed.
1 - 31
R
TRADE
Worker Controllable: A state in which the worker possesses: (1) the means of knowing what
is the quality goal; (2) the means of knowing what is the actual quality performance; and (3)
the means of changing performance in the event of nonconformance.
1 - 32
R
TRADE
References
1. Harrington, H. James. Business Process Improvement. New York, NY: McGraw-Hill, Inc.,
1991.
2. Juran, J. M. Juran on Leadership for Quality. New York, NY: The Free Press, 1989.
3. Juran, J. M. Juran on Planning for Quality. New York, NY: The Free Press, 1988.
4. Juran, J. M. Managerial Breakthrough. New York, NY: McGraw-Hill Book Company,
1964.
5. Juran, J. M. and Frank M. Gryna, Jr., Quality Planning and Analysis. New York, NY:
McGraw-Hill, Inc., 1980.
6. Juran Institute, Inc. "Business Process Quality Management." Wilton, CT: Juran
Institute, Inc., 1990.
7. Juran Institute, Inc. "Quality Improvement Tools." Wilton, CT: Juran Institute, Inc.,
1989.
8. Pall, Gabriel A. Quality Process Management. Englewood Cliffs, NJ: Prentice-Hall, Inc.,
1987.
9. Excellence the IBM Way, pp.. 95-106.
1 - 33
R
TRADE
1 - 34
R
TRADE
1.2 DEVELOPING P ERFORMANCE M EASURES –A SYSTEMATIC A PPROACH
1.2 Developing Performance Measures–A Systematic Approach

Overview
Change might be inevitable, but all too often it occurs like an unguided missile seeking an elusive
target at unpredictable speeds. For most activities, it is far better to manage change with a plan–one
that includes clear goals and useful indications of progress toward a desired objective. Participants in
an activity need to know what outcome is expected, where their work contributes to the overall goal,
how well things are progressing, and what to do if results are not occurring as they should. This
approach places performance measures right where they should be: integrated with the activity.
Such integration makes it possible for performance measures to be effective agents for change. If the
measures quantify results of an activity, one only needs to compare the measured data with desired
goals to know if actions are needed. In other words, the measures should carry the message.
Inappropriate measures are often the results of random selection methods. For example,
brainstorming exercises can get people thinking about what is possible and provide long lists of what
could be measured. Unfortunately, such efforts by themselves do not provide reliable lists of what
should be measured. Unless the measures are firmly connected to results from a defined process, it is
difficult to know what corrective actions to take and to predict with confidence what effects those
changes will have.
If you want to be able to identify effective corrective actions to improve products and services,
results of all key processes must be measured. In this way, one can identify specific processes that
need to change if progress is not satisfactory.
For example, suppose sales are not meeting goals. What actions could be taken? The answer should
depend on what is causing the problem. If poor technical service is causing customers to shy away,
it will do no good to change or add sales personnel. Also, replacing technicians won’t help if the
poor service is caused by a lack of replacement parts. Suppose the replacement parts are on hand,
but, unknown to the service personnel, the parts are defective? If proper measures are instituted for
each key process (purchasing, inventory control, service, etc.), the cause for substandard results can
be found quickly and corrected.
This section is reprinted by permission of Stan Love, former chair of the Performance-Based
Management Special Interest Group, retired from Sandia National Laboratories-Albuquerque.
1 - 35
R
TRADE
Structure and Terminology

A structured approach provides a rational defensible basis for selecting measures and, should it
become necessary, for making changes to the work processes. The viewpoint here is that achieving
goals depends on the performance of interrelated sets of activities and processes that form a system
to be managed. It follows that you can improve the selection of performance measures if they are
based on the desired outcomes of the system and the results expected of each process in the system.
In turn, this suggests a systematic approach to develop performance measures. This methodology
will be described in the following sections.
Before the procedure is described, some key words should be discussed. Some of the terms used
throughout this document could be interpreted differently by different readers; therefore, we are
providing a short glossary of terms as they are used in this section. These are meant to be useful
interpretations, not standard definitions.
A system is an interconnected set of processes, and a process is a set of activities that produce
products or services (results). Products and services are treated alike; that is the output of a
process might be a product (like computer boards) or a service (like training). Performance
measures are quantitative evaluations of the products or services of a process or system. Metrics
are standards of measurement (such as length, area, frequency, mass, and so on).
In addition, there are terms such as Performance Indicators and Indexes. Dealing with these gets
complicated because people use them in very different ways, and there is no one standard to which
we can appeal. Some use indicator and measure interchangeably, while others see indicators as
subsets of measures. Others see indicators as sets of related measures. Still others prefer indexes,
often thought of as sets of related measures (sometimes individually weighted) that track changes
compared to a reference. For example, the Consumer Price Index measures inflation by combining
the prices of selected goods and comparing the results over time. Other examples include an Index of
Indicators (Business Week) that report various areas of the economy (production, construction, etc.).
These more sophisticated concepts are important, but they are beyond the scope of this document.
What is relevant here is that the more data that are combined, the broader the actions that must be
taken to change the situation. The closer the measures are to the activity (i.e., less complex data), the
more focused the actions that can be taken. Regardless of the complexity of the system, however,
the development procedure to be described will apply.
1 - 36
R
TRADE
The Process
Developing performance measures has a definite relationship with Total Quality Management.
Consider a quality process at AT&T:
AT&T Management & Improvement Steps
A. Establish process management responsibilities.

B. Define process and identify customer requirements.
C. Define and establish measures.
D. Assess conformance to customer requirements.
E. Investigate the process for improvement opportunities.
F. Rank improvement opportunities and set objectives.
G. Improve process quality.
In this process, Tasks B, C, and D are the core of performance measure development. In fact, the
systematic method to be described amounts to an elaboration of these three tasks. These three have
been extended to the following six steps:
1. Decide the outcomes wanted.

2. Describe the major work processes involved.
3. Identify the key results needed.
4. Establish performance goals for the results.
5. Define measures for the goals.
6. Select appropriate metrics.
Each of these six steps will be discussed in the following sections. We will create a very small
company, Hackenstack Firewood, that we can use to help us get a working knowledge of this
systematic method for developing performance measures.
1 - 37
R
TRADE
Step 1: Describe the Outcomes Wanted
First Law of Performance: If you try to be the best at everything, you’ll be the best
at nothing.
Why are we doing this work? The answer is to achieve some outcome or objective (the words are
used interchangeably in this process). As used here, objectives might not seem very definitive.
However, they are very important because they set the direction for all processes in the system.
Essentially, objectives (or outcomes) are statements of the wants, needs, and expectations of
customers and other stakeholders. Objectives are the warm and somewhat fuzzy expressions that
should form the mindset for all who are involved in the system. Examples are:
• Supply good pizza with superior delivery service.

• Be the safest airline and offer the lowest fares.
• Produce user-friendly VCRs.
Realize that the desired outcome sets the strategic direction of an enterprise. Consequently, tactical
decisions about what the business does, how it is done, and what gets measured must relate to this
strategic statement. The outcome or objective statement is a driving force for the selection of
performance measures. In the end, what is done and measured somehow must connect with the
desired outcome. For instance, to achieve its objective, that airline will have to spend resources for
equipment and maintenance, not for in-flight meals and reserved seating. And, the measures should
relate to safety and costs that drive ticket prices.
The choices of outcomes should be limited and selected carefully. A major consideration is focus;
avoid the desire to be the best of everything. If you can pick something you are sure you would
succeed at, that choice probably should be your number one objective. In the case of Hackenstack
Firewood, the employees decided that the result of their effort should be to: “Deliver firewood
profitably at competitive prices.”
1 - 38
R
TRADE
Step 2: Describe the Major Work Processes Involved
Second Law of Performance: People are more important than the process, but a
good process is important to people.
What are we doing, and how are we (or should we be) doing it? Processes and their activities are the
means to achieve the outcomes–the end results–identified in Step 1. To improve the chances of
meeting objectives, be sure to understand the system, that is, the operational structure that underlies
the effort. This task is not so obvious. The work we all do usually is part of a larger assignment that
is, in turn, part of a larger job, and so on. Quite often, the work contributes to more than one
assignment or, as is the case with Environment, Safety, & Health (ES&H) initiatives, it is not always
clear which work responds to ES&H requirements and which tasks are unique to building the widget.
Further, the interconnections between functions are not clearly defined or understood. Such
complexities make it even more important to describe carefully the system you want to measure.
Often, the system already is in place, and with luck, it is documented (more or less). In any case, it
is helpful to start with a simplified chart similar to the one in Figure 1.5. A system starts and ends
with customers. In between are identifiable processes that transform inputs (like money or raw
materials) into progressively more useful items (such as thread, then cloth, then clothing). Some
interim products might be enablers, such as operating permits and instructions.
Products/ Products/
Results Results
Process 1 Process 2 Process 3
Input: Products and

results data
Output: Corrective actions
Requirements End Results:

& Money Evaluate & Objective/Outcomes
Correct
Customers
Figure 1.5 A Generic System Diagram
1 - 39
R
TRADE
Notice in Figure 1.6 the two-headed arrows between the Customers and Managers, and next to the
“Evaluate & Correct” process. These indicate important give-and-take interactions, implying that
the inputs and outputs involved are dynamic and subject to negotiations. For example, the price that
customers are willing to pay is variable, as are the requirements for customer satisfaction. These
factors will influence the measures to be selected later.
This part of the procedure is similar to benchmarking and should include interviews with the people
doing the work. People often achieve desired results in spite of, not because of, the process that
exists. Thus, examining work processes usually leads to discovery of some that can and should be
improved. After examining their operations, the employees at Hackenstack Firewood decided that all
of their activities could be placed in the four process blocks shown in Figure 1.6.
Produce &
Manage Mobilize
Deliver Firewood
Evaluate &
Correct
Customers
Figure 1.6 Hackenstack’s operations
Next, the products (results) of the individual processes have to be identified so that measures can be
developed. Note that after the six-step procedure is completed for the system, it is repeated for each
process (Manage, Mobilize, Produce, and Deliver Firewood).
1 - 40
R
TRADE
Step 3: Identify the Key Results Needed
Third Law of Performance: If you can’t describe it, you can’t improve it.
What is produced? The “products” are the outputs or results of each process in the system. That
is, the purpose of the activities in each process is to produce some result (a product or service) that
is needed by other processes. Products of any given process are inputs to other connected
processes in the system. Ultimately, the final products of the system are those that meet the
strategic results–the objective–desired by the company.
For example, in the Hackenstack system, customers provide their requirements (such as amount and
type of firewood) and money to Hackenstack management. These are identified as the most
important products, or results, from customers, and so they will be included in the list of what is to
be measured. In turn, the function of the “Manage” process is to provide the necessary paperwork
and funding that is needed by the mobilization crew. Thus, the products of the first process are
Funding, Permits, and Orders. They will be added to the list of items to be measured.
Next the mobilization crew prepares the field equipment (trucks, saws, etc.) and operating
paperwork (maps, orders, permits, etc.) for the Field Crew who actually cut, stack, and deliver the
firewood. Figure 1.7 shows the completed diagram, and Table 1.1 summarizes the processes and
associated products.
Funding,
Permits, Equipment,
Orders Documents
Produce &
Manage Mobilize Deliver Firewood
Input: Costs, customer satisfaction

Output: Corrective Actions
Requirements
Delivered
Evaluate & Firewood
Customers Correct
Figure 1.7 Hackenstack’s operations and products/results
1 - 41
R
TRADE
Element/Process Products/Results
Customers Income, Requirements
Manage Funding, Permits, Orders
Mobilize Equipment, Documentation
Produce & Deliver Firewood Delivered firewood
Evaluate & Correct Costs, Customer Satisfaction
Table 1.1. Hackenstack Firewood’s Products/Results
Notice the “Evaluate & Correct” process. This function needs to be included in all systems if they
are to be effective and efficient. This is the process that evaluates the performance data and
prescribes changes that might be needed to meet the goals and objectives of the enterprise. This
process is unique because, in addition to the products already mentioned, this process will examine
customer satisfaction as well as costs throughout the system. While this is shown as a separate
block, remember that the activities might very well be performed by the managers, mobilizers, and
field crews. It is just the process, not the performer, that is identified here.
This discussion alludes to other issues, such as the politics and personal interactions that must
accompany any such effort. For purposes of this document, however, suffice it to say that at every
step, it is critical to obtain active endorsement and approval from all the involved and affected
personnel. Otherwise, when goals aren’t being met, the time-honored practice of finger pointing will
commence, and the problems won’t get resolved.
Step 4: Establish Performance Goals for the Results
Fourth Law of Performance: If you don’t have a goal, you can’t score.
How will I know when I get there? And an acid test: What will be done if progress isn’t satisfactory?
There are many questions to ask about this step. Setting goals is very important because you can
spend a lot of resources trying to meet them. You’re familiar with “no pain, no gain”? The PAIN is
worth it if the goals are:
• Profitable (Is it worthwhile to improve this? Favorable Benefit/Cost?)

• Achievable (Can it be improved? How? Who will do it?)
• Important (Does it matter to anyone?)
• Numerical (Without a number, you won’t know when you get there.)
1 - 42
R
TRADE
The GAIN is in reaching the goals, because: Goals Are Improvement Numbers.
There are various ways to determine goals. One of the best methods is to ask the customer for each
of the product(s) you listed in the previous step. In the system diagram, the “customer” is the group
or individual receiving the product. Determine the Customer’s Satisfaction Factors (or Critical
Success Factors, CSFs) that relate to each product. CSFs are the few key things that must be right
for the process to be successful in the customer’s view. Section 2.1.2.7 includes a form that can be
helpful when interviewing stakeholders to negotiate goals.
The goals should be stated in simple terms using numbers, such as “Deliver 500 completed manuals
with fewer than three errors by the end of the month,” or “Improve the average student grade to a
minimum of 93 within six months.”
Be sure to differentiate among lofty goals, stretch goals, and realistic goals. It is best to establish
realistic goals–those you have a decent chance to reach–and after reaching them, establish new ones.
Here are some other considerations that might help to set goals:
• People involved in the process should be able to evaluate their contributions

• Is there an intrinsic limit?
• Is there a safety limit?
• Is there a required level?
• Watch out for diminishing returns; at some point, the return will no longer be worth the
investment.
• Try benchmarking. Find out how well others are doing with similar processes.
Hackenstack’s Committee of the Whole developed the goals listed in Table 1.2. For example, after
researching available wood supplies and performing a market survey, they decided that it would be
reasonable for them to capture one fourth of the regional market, and that if the price is right,
customers would purchase twice the amount of Hackenstack firewood in the coming year. To
achieve these goals and meet their prime objective, they also found they would have to sell an average
of 50 cords of wood each day at $90 a cord. This, they believe, would keep customers and
themselves happy. The remaining goals are the corresponding improvements needed in the products
of the enabling processes to help meet the primary goals.
1 - 43
R
TRADE
Process Products Goals

Customers Income 25% of the regional market
(Output) Requirements Twice last year’s volume
Manage Funding +10% - 0% of budget request

Permits 100% timely availability
Orders 98% error free
Mobilize Equipment 97% uptime

Documentation 99% error free
Produce & Deliver Delivered Firewood Customer cost £ $ 90 cord

Firewood 50 cords per day
Evaluate & Correct Corrective Actions 100% on-time completion

Surveys Positive perception ≥ 80%
Costs +0%, -10% of budget
Customer Satisfaction See Customer output goals
Table 1.2 Hackenstack Firewood’s Annual Goals

Step 5: Define Measures for the Goals
Fifth Law of Performance: Measuring the activity usually improves the activity, but not
the result.
What can you use to track progress? Measures are descriptions of the items to be monitored. At
this stage, measures should be described with relative terms like “percentage of the market” and
“average prices.”
While there is no specific formula for selecting performance measures, there are some characteristics
that are typical of the good ones. They:
• reflect results, not the activities used to produce results

• relate directly to a performance goal
• are based on measurable data
1 - 44
R
TRADE
• contain normalized metrics for benchmarking

• are practical and easily understood by all
• provide a continual self assessment
• provide a benefit that exceeds the cost
• are accepted and have owners
The first criterion is important because it is very tempting to select measures that are easy, while the
right measures can be difficult. For example, the effects of training can be very hard to assess. It can
be time consuming to interview employees and managers to find out if the training has improved
worker abilities on the job. Rather than spend the time to do the interviews, people often change the
measure to something like “the number of class hours completed” (essentially a useless measure
unless it is accompanied by some measure of acquired skills).
Find appropriate measures by examining all the goals listed in the previous step. For example,
Hackenstack’s goal to keep the selling price of firewood at or below $90 a cord clearly implies that a
measure of customer prices must be included. This simplicity will always be the case if the goals are
chosen well and stated numerically. Difficulties occur when goals are poorly or incompletely stated,
such as “satisfy the customer.” This would be difficult to measure because this goal doesn’t make it
clear what will satisfy the customer! On the other hand, as the Hackenstack team observed, they
must be satisfying the customer if sales and income are progressing toward or beyond the goal.
There are other measures for customer satisfaction, of course. But to be useful, they must relate to
some numerical goal.
While selecting measures, it is wise to remember that the idea is to be able to track progress and to be
able to change processes (or activities, or the system) as needed to improve results. So, it will help
to ask if the system measures you choose will be adequate to identify which process needs fixing.
Also, consider what practical actions could be taken if any of the products are not progressing
toward the goal fast enough. This thought experiment will help you select the right measures you
need to help you make decisions later.
Finally, there must be a balance to the number of measures; after all, you can’t and shouldn’t measure
more than you have time to evaluate.
Appendix C contains an extensive list of performance measures for various processes that may be
helpful in determining measures for your situation.
Table 1.3 shows the measures selected by the Hackenstack Firewood team.
1 - 45
R
TRADE
Process Products Goals Measures

Customers Income 25% of the regional market Income compared
with regional markets
Requirements Twice last year’s volume Volume delivered

compared with last year
Manage Funding +10% - 0% of budget Available funds compared
request with budget request
Permits 100% timely availability Permits available

when needed
Orders 98% error free Proportion of error

free orders
Mobilize Equipment 97% uptime Proportion of available
equipment time
Documentation 99% error free Proportion of error

free documents
Produce & Delivered Firewood Selling price £ $90 cord Average selling price
Deliver
Firewood
50 cords per day Average daily volume
produced
Evaluate & Corrective Actions 100% on-time completion Proportion of corrective
Correct actions completed on time
Surveys Positive perception ≥ 80% Perception scores
Costs +0%, -10% of budget Spending

(90% to 100% of budget)
Table 1.3 Hackenstack Firewood Team’s Measures
Step 6: Identify the Required Metrics
Sixth Law of Performance: If you know the score, you should be able to predict the
outcome.
What specific things do I measure? Metrics–the actual measurements to make–should be fairly

obvious from the descriptions of the measures composed in the previous step. Examine the
measures statements and the goals to identify the units required for each term.
1 - 46
R
TRADE
For example, to support Hackenstack Firewood’s goal to capture 25% of the regional market, the
measure is the Company’s income compared with the total dollars spent by customers in the region.
This translates into dollars received by Hackenstack divided by the total dollars received by all
firewood companies in the region during the same time period (perhaps monthly or weekly if
possible). Notice that this fraction is dimensionless, expressing a percentage. The comparison, or
ratio of two meaningful dollar figures creates a normalized measure that can be used for tracking and
comparing with other similar businesses. Another way to normalize is to use a ration of actual
versus planned results. Metrics for a measure of yield quality might be the number of acceptable
units produced divided by the total number of units produced.
Of course there are more issues that might need attention. Can Hackenstack find out the total income
of their competitors? If not, what other measure can they consider? Perhaps the data only are
available annually; that will not help because by the time they get the data, it will be too late to take
any action. In this case, estimates based on projections from last year’s regional income could be
useful.
The complete summary of the Hackenstack Firewood system is tabulated in Table 1.4.
1 - 47
R
TRADE
Process Products Goals Measures Metrics *

Customers Income 25% of the regional Income compared $ income
market with regional markets $ regional
Requirements Twice last year’s Volume delivered # cords this year

volume compared with last year # cords last year
Manage Funding +10% - 0% of Available funds $ funding

budget request compared $ requested
with budget request
Permits 100% timely Permits available # permits available

availability when needed # permits needed
Orders 98% error free Proportion of error- # error-free orders
free orders # orders issued
Mobilize Equipment 97% uptime Proportion of available Equipment hrs. avail.

equipment time Equipment hrs. needed
Documentation 99% error free Proportion of error- # error-free documents

free documents # documents issued
Produce & Delivered Selling price £ $ 90 Average selling price $ income

Deliver Firewood cord # cords sold
Firewood
50 cords per day Average daily volume # cords stocked
produced days
Evaluate & Corrective 100% on-time Proportion of corrective # completed on time

Correct Actions completion actions completed on time # scheduled
Surveys Positive perception ≥ Perception scores Total actual score

80% Total possible score
Costs +0%, -10% of Spending $ spent

budget $ budgeted
* The time intervals for each metric will be the same but may differ between measures.
Table 1.4 Hackenstack Firewood Company System Summary
Supporting Information
This section contains additional forms, worksheets, and a glossary that will help you use the process
described above.
1 - 48
R
TRADE
Glossary
Activities: Actions that change resources from one form to another.
Goal: A specific, numerical result that is to be achieved by the process or system.
Indexes: Sets of related measures or indicators (sometimes individually weighted) that track
changes compared to a reference.
Metrics: Standards of measure (such as length, area, frequency, etc.).
Normalize: Adjust metrics to allow comparisons with a reference or standard (usually done by
using rates or percentages).
Objective: A statement of the general condition to be achieved (e.g., “work safely”).
Performance Indicator: Pointers comprising of related performance measures that reveal
changes compared to a reference; that is, an indicator is composed of one or more measures
Performance Measure: Quantitative descriptions of the quality of products or services of a
process or system.
Process: A set of activities that produce products or services.
Product: A tangible result of a process or system.
Service: Work done for others; also a result of a process or system.
System: A logical, interconnected set of processes.
1 - 49
R
TRADE
System Verification & Performance Goals
Interview Questions to Validate the System Definition
1. Verify the accuracy of each process block with the owners/representatives:

Question: Are these correct?
Input Yes No
Process Yes No
Products Yes No
Results desired Yes No
If any changes are suggested, specify them:
2. Identify the product goals:

Question: What are the most important expectations of the products; that is,
what do you want to have happen as a result of these outputs?
3. Determine if there are existing performance measures of value:

Question: Have you identified any specific, quantitative feedback that you
want or require?
1 - 50
R
TRADE
System Flowchart
System Summary Table
PROCESS PRODUCT GOAL MEASURE METRIC
1 - 51
R
TRADE
1 - 52
R
TRADE
1.3 DEVELOPING P ERFORMANCE M ETRICS – UNIVERSITY OF C ALIFORNIA A PPROACH
1.3 Developing Performance Metrics – University of California Approach
Introduction
Performance metrics should be constructed to encourage performance improvement, effectiveness,
efficiency, and appropriate levels of internal controls. They should incorporate "best practices"
related to the performance being measured and cost/risk/benefit analysis, where appropriate.
Performance measurement is an important cornerstone of the contracts between the University of
California and the Department of Energy for the operation of the laboratories. This section discusses
the principles and concepts used in developing effective performance metrics for these contracts.
The Department of Energy has promulgated a set of Total Quality Management guidelines that
indicate that performance metrics should lead to a quantitative assessment of gains in:
• Customer satisfaction
• Organizational performance
• Workforce excellence
The key elements of the performance metrics to meet these guidelines should address the following
key elements:
• Alignment with organizational mission

• Quality of product
• Timely delivery
• Cost reduction and/or avoidance
• Cycle time reduction
• Customer satisfaction
• Meeting DOE requirements
• Meeting commitments
1 - 53
R
TRADE
The Process
The first step in developing performance metrics is to involve the people who are responsible for the
work to be measured because they are the most knowledgeable about the work. Once these people
are identified and involved, it is necessary to:
• Identify critical work processes and customer requirements.

• Identify critical results desired and align them to customer requirements.
• Develop measurements for the critical work processes or critical results.
• Establish performance goals, standards, or benchmarks.
The establishment of performance goals can best be specified when they are defined within three
primary levels:
• Objectives: Broad, general areas of review. These generally reflect the end goals based on
the mission of a function.
• Criteria: Specific areas of accomplishment that satisfy major divisions of responsibility
within a function.
• Measures: Metrics designed to drive improvement and characterize progress made under
each criteria. These are specific quantifiable goals based on individual expected work
outputs.
The SMART test is frequently used to provide a quick reference to determine the quality of a
particular performance metric:
• S = Specific: clear and focused to avoid misinterpretation. Should include measure

assumptions and definitions and be easily interpreted.
• M = Measurable: can be quantified and compared to other data. It should allow for
meaningful statistical analysis. Avoid "yes/no" measures except in limited cases, such as
start-up or systems-in-place situations.
• A = Attainable: achievable, reasonable, and credible under conditions expected.
• R = Realistic: fits into the organization's constraints and is cost-effective.
• T= Timely: doable within the time frame given.
1 - 54
R
TRADE
Types of Metrics
Quality performance metrics allow for the collection of meaningful data for trending and analysis of
rate-of-change over time. Examples are:
• Trending against known standards: the standards may come from either internal or external
sources and may include benchmarks.
• Trending with standards to be established: usually this type of metric is used in conjunction
with establishing a baseline.
• Milestones achieved.
"Yes/no" metrics are used in certain situations, usually involving establishing trends, baselines, or
targets, or in start-up cases. Because there is no valid calibration of the level of performance for this
type of measure, they should be used sparingly. Examples are:
• Establish/implement a system.
• System is in place (without regard to effectiveness).
• Analysis performed (without criteria).
• Reporting achieved ( without analyses).
• Threshold achieved (arbitrary standards).
Classification of Performance Metrics
Measure of... Measures... Expressed as ratio of...

Efficiency Ability of an organization to perform a Actual input/
task planned input
Effectiveness Ability of an organization to plan for Actual output/ planned
output from its processes output
Quality Whether a unit of work was done Number of units produced
correctly. Criteria to define correctly/total number of
“correctness” are established by the units produced.
customer(s).
Timeliness Whether a unit of work was done on Number of units produced
time. Criteria to define “on-time” are on time/total number of
established by the customer(s). units produced.
Productivity The amount of a resource used to Outputs/inputs
produce a unit of work
1 - 55
R
TRADE
The following questions serve as a checklist to determine the quality of the performance metrics that
have been defined:
• Is the metric objectively measurable?

• Does the metric include a clear statement of the end results expected?
• Does the metric support customer requirements, including compliance issues where
appropriate? (Keep in mind that in some areas compliance is performance; e.g., ES&H.)
• Does the metric focus on the effectiveness and/or efficiency of the system being measured?
• Does the metric allow for meaningful trend or statistical analysis?
• Have appropriate industry or other external standards been applied?
• Does the metric include milestones and or indicators to express qualitative criteria?
• Are the metrics challenging but at the same time attainable?
• Are assumptions and definitions specified for what constitutes satisfactory performance?
• Have those who are responsible for the performance being measured been fully involved in
the development of this metric?
• Has the metric been mutually agreed upon by you and your customers?
Common Terms with Performance Metrics

Commonly used terms concerning performance metrics are:
Baselining: The process of establishing a reference set of data that reflect the current state of a
process, system, or product.
Benchmark: A standard or point of reference for measurement. By providing ranges or
averages, benchmarks enable an organization to compare performance in certain key areas
with other organizations.
Benchmarking: A method of measuring a process, system, or outcome within an organization
against those of a recognized leader. The purpose of benchmarking is to provide a target for
improved performance.
Best in class: Leader or top performer in relation to a particular performance goal as identified
through a benchmark.
Effectiveness: The ability to accomplish a desired result or to fulfill a purpose or intent.
Efficiency: The quality or degree of effective operations as measured against cost, resources,
and time.
Goal: A target level of performance expressed as a tangible, measurable objective against which
actual achievement can be compared.
1 - 56
R
TRADE
Lower control limit: The lower line on a control chart below which variation is not expected.
This is mathematically represented by the average minus three standard deviations.
1-10-100 Rule: The rule that states that if a problem is not fixed in a timely manner when first
discovered, it will be more costly to fix later (in terms of both time and money). The rule
recognizes that it makes a difference when a problem is discovered and resolved.
Quality Grid: A quality improvement concept that divides quality work into what is done (
doing the right things ) and how its done (doing things the right way)-
How You Do It
Right Things Right Things

Wrong Right What you do
Wrong Things Wrong Things

Wrong Right
Re-engineering: A process of rethinking and redesigning work processes to achieve noticeable

improvements in service delivery responsive to customer needs and/or achieve significant
reductions in cost.
Standards: A prescribed set of rules, conditions, or requirements used to measure or define the
quality or quantity of particular performance elements.
Value-added: Process or steps that enhance an outcome.
Upper control limit: The upper line on a control chart above which variation is not expected.
This is mathematically represented by the average plus three standard deviations.
References
Three basic references useful in the development of performance metrics are “Total Quality
Management Guidelines,” U.S. Department of Energy, December 1993, and the annual “Malcom
Baldrige National Quality Award Criteria” and its companion volume, “The Handbook for the Board
of Examiners.” The latter two documents are published by the National Institute of Standards and
Technology, an agency of the U.S. Department of Commerce.
1 - 57
R
TRADE
1 - 58
R
TRADE
1.4 PERFORMANCE I NDEXES
1.4 Performance Indexes

Often it is necessary to present information from several related areas simultaneously. This is done
to provide a statistical measure of how performance changes over time. The performance index is a
management tool that allows multiple sets of information to be compiled into an overall measure.
This section provides examples on different approaches that can be taken to develop a performance
index.
Overview
Up until this point you’ve learned about developing and working with single performance measures.
Now it is time to branch out into the world of performance indicators and indexes. For the purpose
of this section, a performance indicator (PI) is defined as the result of the comparative analysis of a
performance measurement outcome to a corresponding performance goal. These measurements give
an indication of performance. However, when you have too many indications to consider,
performance indexing becomes a useful performance management tool. The philosophy behind using
performance indexes is simple: they condense a great deal of information into one number. We know
that when dealing with a small number of indicators, PI-related information is easy to assimilate. But
what happens when you’re not dealing with just one or two PIs? What happens if you have 10, or
15, or 20 separate but related indicators to review. With some increasing, and others decreasing,
while still others remain the same, how do you determine what is happening overall? The answer is
to use an index.
Consider this: if I handed you a newspaper and asked, “How’s the stock market doing?” would you
examine the trend associated with each of the 5000+ stocks listed on the financial pages before giving
me your answer? Hopefully, not. A quicker, simpler, and more efficient method would be to turn to
the financial pages, and look at one of many business indexes that appear there, say the Standard and
Poors 500 Index (S&P 500). The advantage of the S&P 500 Index is that it gives you a general
indication of trends in the stock market at a glance. The downside of the index is that it will not give
you specific information on any one particular stock.
So, what exactly is an index? Simply put, an index is a statistical measure of how a variable, or set of
variables, changes over time. The purpose of an index is to give a quick, overall picture of
performance.
The power of using indexes as management tools clearly resides in their ability to capture the
information contained in a large number of variables in one number. For instance, economists can use
one number, the Consumer Price Index (CPI), to capture pricing information on several hundred
different consumer products. Now, instead of having to track over 400 different prices, they only
need to track one number–the CPI. Economists place a lot of trust in this index; annual cost-of-living
adjustments and retirement benefits for over 50 million civil servants are directly linked to
fluctuations in the CPI.
1 - 59
R
TRADE
How do you create an index? This is not an easy question to answer because there is no one set
formula or algorithm for generating indexes. However, there are certain concepts that apply to all
indexes, the most important being that all indexes are designed for a particular purpose, and that the
design process involves choosing the correct (related) indicators and then combining them in a manner
that supports the intended purpose of the index.
Now, simply because there is no patent method for producing an index does not mean that creating
one has to be a complicated matter. In fact, it can be as simple as computing the ratio between two
numbers.
It is not the intent of this handbook to address each and every method that can be used to develop
index numbers, nor will it make you an expert in the statistics behind developing indexes (for those
interested in a more in-depth study of indexing methods, a list of references has been included).
What it will do, hopefully, is give you an appreciation of the power of using performance indexes as
a management tool, and provide you with a few examples of methods that are currently being used
throughout the DOE complex and private industry to create performance indexes. These methods
range from fairly simple to fairly complicated. They include:
• the DOE Occupational Injury/Illness Cost Index

• the Westinghouse Hanford Company Conduct of Operations Event Index
• the Eastman Kodak Company Safety Performance Index
• the Defense Programs Average of Performance Relatives
Each of these methods will be discussed in detail in the following section.
Examples
Example 1: The DOE Occupational Injury/Illness Cost Index

This index is the simplest of the three methods presented above. Essentially, the cost index is a
linear combination of weighted parameters:
Index = (W1 x P1) + (W2 x P2) + ... + (Wn x Pn)
where Ws are constant weighting factors, and Ps are individual measurable items. When determining
the weighting factors, the following could be taken into account:
• Dollar Cost Analysis

• Probability Risk Assessment
1 - 60
R
TRADE
• Cost/Benefit Analysis
• Expert Opinion
A strength of weighted linear combinations is that they can assist in determining how to allocate
limited resources. That is, if W1 = 3 and W2 = 1, then, given limited resources, addressing P1 provides
more benefit than addressing P2 and resources could be applied accordingly. Essentially, this is a
“tradeoff” where one unit of improvement in P1 is worth losing up to 3 units of P2.
Caution! People will make these tradeoffs! If you mix safety and production indicators together,
you may, unwittingly, be sending the signal that degradation in safety performance can be offset by
increased production.
The DOE Occupational Injury/Illness Cost Index combines the following indicators:
• Number of fatalities (D)

• Number of transfers or terminations due to injury or illness (T)
• Number of lost workday cases (LWC)
• Number of days away from work (WDL)
• Number of restricted workdays (WDLR)
• Number of non-fatal cases without days away from work or restricted workdays (NFC)
The weighting factors were determined using dollar cost analysis.
• Index = 100[(1,000,000)(D) + (500,000)(T) + (2,000)(LWC) + (1,000)(WDL) +

(400)(WDLR) + (2,000)(NFC)]/Total hours worked
Example 2: Westinghouse Hanford Conduct of Operations Event Index

Let’s look at another example of an index based on linear combinations: the Conduct of Operations
Event Index developed at the Westinghouse Hanford Company. This index is different from the
previous example in that it does not utilize weighting factors for the components. Basically, this
index measures the number of certain types of occurrence reports per 200,000 hours worked by an
organization. This index uses information that can be obtained from the Occurrence Reporting and
Processing System (ORPS). The following parameters are combined:
A. Skin and clothing contaminations: Number of Occurrence Reports with nature of

Occurrence of 4B (Personnel Contamination).
B. Violations of Procedures: Number of Occurrence Reports with a Root Cause of 3C
(Violations of Requirement or Procedure).
1 - 61
R
TRADE
C. Procedure Problem: Number of Occurrence Reports with Root Cause of 2A or 2B.

2A - Defective or inadequate procedure
2B - Lack of procedure
D. Training Deficiency: Number of Occurrence Reports with Root Causes 5A through 5E
5A - No training provided
5B - Insufficient practice or hands-on experience
5C - Inadequate content
5D - Insufficient refresher training
5E - Inadequate presentation or materials
E. Management Problem: Number of Occurrence Reports with Root Causes 6A through 6F.
6A - Inadequate administrative control
6B - Work organization/planning deficiency
6C - Inadequate supervision
6D - Improper resource allocation
6E - Policy not adequately defined, disseminated, or enforced
6F - Other management problem
F. Lockout/tagout errors: Number of Occurrence Reports judged to be lockout/tagout related.
G. Work control errors: Number of Occurrence Reports judged to be work control related.
Note that some of the Occurrence Reports could fall into multiple categories
and will be counted more than once.
H. Person-hours worked: Each facility determines which employees should be included in the
person-hours calculation.
The index calculation is based on the number of occurrences that happened during the time period (as
per the above criteria) divided by the opportunities for occurrences to happen (i.e., person-hours
worked):
Index Calculation: Index = (A + B + C + D + E + F + G)
(H/200,000)
1 - 62
R
TRADE
The use of person-hours assumes that the larger the operating force of a facility, the more
opportunity there is for Conduct of Operations type events. Dividing by person-hours worked is an
attempt to express the conduct of operations as a rate identical to the method used for Lost Work
Day Case Rate.
Example 3: Eastman Kodak Safety Performance Index

The Eastman Kodak company uses an interesting and effective methodology for creating
performance indexes. Basically, the method used by Kodak involves mapping the range of
performance for several metrics onto a fixed scale, applying a multiplier to the value extracted from
the scale, and adding the results together. An example of how this index is computed follows (this
example has been adopted from Kodak’ handbook for Safety Performance Indexing and modified
slightly for the purposes of this example).
Developing a Performance Matrix

The first step in the Kodak process involves developing a performance matrix that shows goals and
ranges of performance for several metrics. Figure 1.8 is an example of a performance matrix.
At Kodak, developing this matrix is a 10-step process:
Step 1: Select indicators that are related to and that measure progress in the area for which you intend
to develop an index. Kodak developed a Safety Performance Index. In this example, we will use the
Kodak method to develop a Conduct of Operations Index. Remember to ensure that the performance
indicators that are chosen are clearly defined. Once the appropriate performance indicators are
chosen, list them down the left column of the matrix (see Figure 1.8).
1 - 63
R
TRADE
Performance Level Calculations
Performance Value Level Wt. Score

1 2 3 4 5 6 7 8 9 10
Indicator
(PI)
Unplanned
Safety Function
Actuations
Violations of
Operating
Procedures
Unplanned
Shutdowns
Number of
Unusual
Occurrences
Figure 1.8
Example Performance Matrix
Step 2: For each of the component performance indicators, determine its relative importance and the
impact that it should have on the index. The total of the weight for the constituent performance
indicators must add up to 100%. Write the value of the weights in the “Wt.” column. (See Figure
1.9).
1 - 64
R
TRADE

1 2 3 4 5 6 7 8 9 10
Indicator
(PI)
Unplanned
Safety Function 20
Actuations
Violations of
Operating 30
Procedures
Unplanned
Shutdowns 30
Number of
Unusual 20
Occurrences
Figure 1.9
Example Performance Matrix with Weights
Step 3: Establish the baseline value for each performance indicator. In the matrix, level 7 represents
the baseline. A good baseline might be a four-quarter average.
Step 4: Determine a goal for each measure. In the matrix, performance level 3 represents the goal.
Step 5: Determine a “stretch goal” for each performance indicator. This goal should be attainable,
but only if your facility performs superbly. In Figure 1.10, the stretch goal is represented by level 1
in the matrix.
1 - 65
R
TRADE

1 2 3 4 5 6 7 8 9 10
Indicator
(PI)
Unplanned
Safety Function 15 30 50 20
Actuations
Violations of
Operating 8 25 45 30
Procedures
Unplanned
Shutdowns 10 20 30 30
Number of
Unusual 20 35 60 20
Occurrences
Figure 1.10
Example Performance Matrix with Baselines and Goals
Step 6: Establish intermediate goals for levels 4, 5, and 6 in the matrix. These may be specific
milestones determined by line management, or they may be simple numeric increments between the
baseline and the goal.
Step 7: Determine values for levels 8, 9, and 10. It is possible that performance can be worse than
the baseline. To account for this, set appropriate values for levels 8, 9, and 10.
Step 8: Assign a value to Level 2 PIs. You should now have all performance levels filled in as shown
in the example in Figure 1.11.
1 - 66
R
TRADE

1 2 3 4 5 6 7 8 9 10
Indicator
(PI)
Unplanned
Safety Function 15 20 30 35 40 45 50 60 70 80 20
Actuations
Violations of
Operating 8 15 25 30 35 40 45 50 55 60 30
Procedures
Unplanned
Shutdowns 10 15 20 23 25 27 30 34 38 42 30
Number of
Unusual 20 25 35 45 50 55 60 65 70 75 20
Occurrences
Figure 1.11
Example Performance Matrix with Completed Performance Levels
Step 9: Debug the matrix. Use stakeholder feedback to evaluate the initial selection of performance
indicators, the performance levels, assigned weights, and so on. Make necessary changes.
Step 10: Develop a system for scoring and displaying results. It is important to assign the
responsibility for collecting, calculating, plotting, and disseminating performance index information.
It is equally important to set up a mechanism for the periodic review and updating of each
performance matrix.
Remember, for each of the PIs chosen in this example, an increase in value represents a decrease in
performance. This may not always be the case. For this reason, it is important to understand how
increases and decreases in each indicator relate to performance, and to determine the baseline values,
goals, and stretch goals accordingly.
Calculating the Performance Index

The first step in calculating the index is to measure the current value for each performance indicator.
Then, using the matrix, determine the corresponding performance levels. In situations where the value
for a performance indicator falls between performance levels, choose the next higher level. For
instance, say that during the last measuring period there were 53 unusual occurrence reports (UORs).
Since there is no performance level that corresponds exactly to 53 UORs, you would choose the next
higher level, or 55, which corresponds to performance level 6.
1 - 67
R
TRADE
For illustrative purposes, assume that measuring the performance indicators that make up our
hypothetical Conduct of Operations Index yields the following results:
• Unplanned Safety Function Actuations  38 events

• Violations of Operating Procedures  50 events
• Unplanned Shutdowns  23 events
• Number of UORs  53 events
Based on the above values, the completed performance matrix would appear as it does in Figure 1.12.

1 2 3 4 5 6 7 8 9 10
Indicator
(PI)
Unplanned
Safety Function 15 20 30 35 40 45 50 60 70 80 38 5 20 100
Actuations
Violations of
Operating 8 15 25 30 35 40 45 50 55 60 50 8 30 240
Procedures
Unplanned
Shutdowns 10 15 20 23 25 27 30 34 38 42 23 4 30 120
Number of
Unusual 20 25 35 45 50 55 60 65 70 75 53 6 20 120
Occurrences
Figure 1.12
Example Completed Performance Matrix
The score for each performance indicator is determined by multiplying the level times the weight.
Once this is done, the scores are added together to determine the composite results. In this case, it
yields a value of (100 + 240 + 120 + 120) = 580 for the index. This could be compared to a baseline
value for the index of 700 (performance level 7 for all indicators), and a goal of 300 (performance
level 3 for all indicators). Ideally, values for this index would be calculated every month, quarter, or
whatever time period is chosen, and tracked over time.
1 - 68
R
TRADE
Example 4: Defense Programs Average of Performance Relatives

This method is based on determining incremental changes in a number of indicators, relative to a
baseline period in time, and then averaging these changes or ratios. It is similar to methods used to
compute certain economic indexes. The incremental changes are referred to as “performance
relatives” (i.e., performance during a given period relative to performance during a set baseline
period). Remember that this index it is not an absolute indicator. It is only meaningful when tracked
over time and compared to past performance.
Performance Relative
A performance relative is a number that compares the value for a certain measure in a given period to
the value for the same measure at some fixed period in the past. The best way to demonstrate this
idea is by example. This example refers to price relatives rather than performance relatives; however,
the two are conceptually identical.
• Example 1: Say you have three commodities, rice , barley, and oats, whose prices in 1970 were
$1.00/bushel, $1.25/bushel, and $1.50/bushel, respectively.
In 1980 the prices for these same three commodities were: rice, $1.25/bushel, barley,
$1.10/bushel, and oats, $1.75/bushel.
Using 1970 as the baseline year, the price relatives (PR) for these three items would be:
PRrice = Price1980/Price1970 = $1.25/$1.00= 1.25
PRbarley = Price1980/Price1970 = $1.10/$1.25= .88
PRoats = Price1980/Price1970 = $1.75/$1.50= 1.17
The price relatives computed in Example 1 give a good indication of how the prices for these
commodities fluctuated from 1970 to 1980  the prices for rice and oats increased, while the price
for barley decreased. This is a good indicator of what happened individually, but what about what
happened overall?
Average of Performance Relatives

The next step is to compute an average of performance. Many sources agree that the most balanced
method for doing this is to use a geometric mean. The reason behind using the geometric mean is
beyond the scope of this handbook. However, those who would like to gain a better understanding
of the statistics behind this method may consult the Suggested Reading on Page 71.
• Geometric mean of performance relatives:
1 - 69
R
TRADE
I = antilog(∑ log Pi /Po )

n
Pi = series performance indicator

Po = baseline value
n = number of performance indicators in the series
The formula above may seem a bit imposing at first; it isn’t. It is simply one way to compute an
average. Instead of adding (PRrice + PRbarley + PRoats) and dividing by 3, which would give you a
simple arithmetic mean, you are multiplying (PRrice) x (PRbarley) x (PRoats) and taking the nth root, or
cube root, which gives you a simple geometric mean.
• Example 1 (continued): Using the above formula, the index value for the three commodities in
1970 would be 100  this is our baseline value. The value for 1980 would then be:
Index = [(1.25 x 1.17 x .88)1/3] x 100 = 109 points
The index value in 1980, 109, is meaningless in absolute terms. It only takes on meaning when
compared to the baseline value for the index, or 100, and, as you can see, the value of the index
increased by 9 points from 1970 to 1980.
Next, let’s assume that twice as much rice is consumed as either barley or oats. It follows that rice
should contribute more heavily to the index, say twice as much. Here’s how this weighting factor
could be handled:
Index = [(1.25 x 1.25 x 1.17 x .88)1/4] x 100 = 113 points
The increased emphasis on rice (PR 1.25) increased the value of the index by 4 points.
1 - 70
R
TRADE
Suggested Reading
1. Clark, Charles T., and Schkade Lawrence W., Statistical Analysis for Administrative
Decisions, South-Western Publishing Co., 1979.
2. Fisher, Irving, The Making of Index Numbers, Houghton Mifflin Company, 1927.
3. Persons, Warren Milton, The Construction of Index Numbers, Houghton Mifflin Company,
1928.
4. Safety Performance Indexing: Metrics for Safety Performance Improvement Projects, Eastman
Kodak Company, 1994.
1 - 71
R
TRADE
1 - 72
R
TRADE
Section 2
Data Tools
SECTION 2: DATA TOOLS
2.0 OVERVIEW
2.0 Overview
Performance Measuring
Introduction
Performance measures are an important topic. People are pursuing excellence and are eager for
results to show their efforts are working. Having data is fundamental, and the old saying, "without
data, you're just another person with an opinion," has its wisdom. Still, the use of performance
measures is not as easy as it appears, and poor use can do more harm than good.
The following introduction is taken from “Grassroots Approach to Statistical Methods” and is
reprinted with the permission of R. S. (Bud) Leete, Lockheed Martin Energy Systems.
Why Do We Measure?
Measuring is the act of assigning numbers to properties or characteristics. We measure to quantify a
situation, to regulate, or to understand what affects things we see. Sometimes we measure with
gauges and instruments; sometimes, we simply count things. Performance measures can help you
understand and improve performance. It is exciting to measure, to benchmark, and to stretch to do
better.
Some Measures are More Direct than Others

A first step in deciding what to measure is to decide what you want to improve. Sometimes there is
a direct measure. For example, runners or swimmers who want to improve their performance in a
100-yard race can measure their times directly as a performance measure. In golf, measuring
performance by the score shot may seem appropriate. However, it is important to note that the
golfer's score is not as direct a performance measure as the swimmer's time, because factors like
course difficulty and playing conditions vary considerably. In tennis or figure skating, it is even
harder to arrive at a performance measure.
Similar difficulties arise at work. Suppose we want to improve morale. Surveys are a possibility.
We could ask people, "how is your morale?" and administer a survey periodically. The answer is
subjective, and people may tire of being asked this question periodically, especially if morale does
not improve. Eventually, we might devise a measure that is indirect, but easier to obtain, like
attendance. "If morale is high, people will come to work," will be our logic. A performance indicator
is born. Of course, other factors influence attendance, such as sickness, family situations, births,
deaths, and the weather.
2-1
R
TRADE
2.0 OVERVIEW
It is important, therefore, that performance measures be as direct as possible. To improve

attendance, measure attendance. To improve cycle times, measure cycle times. If you want to
improve morale, you may be better off deciding how to improve morale and measuring your efforts.
Rule 1: The more directly you can measure, the better.
Operational Definitions
Once you decide what to measure, carefully and thoughtfully determine how to take your
measurement. Likewise, if you are counting you need to know exactly what you should and should
not count. This is the process of setting operational definitions.
As an example, suppose you want to measure the level of beryllium in a work area as an industrial
safety performance measure. You'll be taking smear samples. Should you check places at random?
Should you instruct the technicians to look for the "worst" places or at "typical" places? Should you
select a set of specific places and check those same places each measuring period? Should you
average the values? What measuring method should be used in the laboratory? Should the lab
perform duplicate analyses and average them?
Questions like these are important to the long-range success of the indicator. Efforts to standardize
and communicate instructions are needed. Operational definitions are instructions and turn general
terms, such as "contamination" or "scrap," into specific actions, procedures, and computations.
Without them, we leave ourselves vulnerable to variations people will introduce in interpreting
incomplete instructions.
Rule 2: Define exactly how to collect the data for the indicator and how to make the computation.
Then, make sure everyone understands.
How Defined is the Underlying Process?

There are two types of studies: "enumerative" and "analytical" studies. Enumerative studies are
those that show how things are but have no value in predicting. Taking a census is an enumerative
study. Measuring corn yield is an enumerative process. The farming conditions of 1984 or 1987 or
whenever are gone forever. They will never be repeated. In this sense, there is no consistent
underlying process. Analytical studies are used to study a process and show what that process is
capable of doing in the future unless something changes that process.
Dr. Walter Shewhart, a pioneer in statistical process control, said a process consists of equipment,
methods, materials, and people being blended to produce output in some environment. For example,
in farming, the environment is at times so dominant that the measure of yield may reveal little of the
farmer's processing methods, the quality of the soil, or seeds.
2-2
R
TRADE
2.0 OVERVIEW
In developing a performance measure, flowchart the process that produces the result you will
measure. If you can, you'll be able to use the performance measure as a tool for improving. If you
can't, the road to improving will be winding, at best.
Rule 3: If you can't flowchart the process you want to improve, a performance measure will offer
little insight into what you should do to improve.
Interpreting What You See

Variation is a fact of life. What we want to do with performance measures is to understand the
variation. We want to learn how to influence performance with elements of the process that we can
control. To interpret what we see, we need to know about variation and how observations vary. We
need to understand statistical methods.
Rule 4: Use the numbers to help people improve, not to judge people.
Statistical Models of a Process

Mathematicians have studied physical phenomena for years to understand and model how things
work. Consequently, statistical methods have been developed that everyone working with
performance measures should understand. Two main aspects of statistical methods deal with
statistical distributions and statistical control.
It is important for leaders, whether they be leaders of a company or key members of a team, to
understand statistical concepts on variation, including statistical distributions and statistical control.
They should also understand special causes, common causes, and control charting. In addition, they
should clearly understand the concept of "tampering" with a process.
Rule 5: Learn to understand variation. You'll be more effective.
Types of Variation
One way of visualizing variation is through a demonstration. Dr. W. Edwards Deming, often called
the father of modern statistics, wrote of a funnel experiment that helps explain variation. This
experiment requires a funnel, a marble, a stand to hold the funnel steady, a flat place to work, and
paper and pencil. On a level space, we arrange the funnel and its stand. We mark a spot on our
paper as the "target" and position the stand so we can drop the marble through the funnel and land it
on the target. The "process" consists of taking the marble, dropping it in a swirling motion into the
funnel, and letting it land on the paper. We use the pencil to mark where the marble comes to rest.
Marking lets us see the pattern of variation.
When the pattern of variation is observed after 30 or so drops one can see how the pencil marks on
the paper tend to form a natural pattern. This is the pattern of variation. The funnel, all set up for
the workers to use, is the "system." Workers perform the operations in the system. They are given
the funnel, the marble, and the procedures to follow. Managers are responsible for the system. If a
2-3
R
TRADE
2.0 OVERVIEW
change is required in funnel height, or composition, in the marble composition or diameter, or in the
flatness of the table, the change can only be made by managers. The local workers are those who
work inside the system but don't have the authority to change it. Managers are the workers who
have the authority to make changes to the system.
This natural pattern of variation shows what the process is capable of doing. When the marble
process continues to run, the pattern becomes predictable. We don't know where the next marble
will fall, but we can expect the pattern to be preserved. If it is not preserved, we look for a special
cause.
A key concept is that we must study the pattern of variation. If it is controlled, meaning
observations fall seemingly at random within some overall natural pattern, then the process is
performing at its natural capability. It is a management responsibility to improve a controlled
process. If it is uncontrolled or unstable, meaning the observations fall in patterns that seem to defy
the laws of probability, then special causes of variation are present. The local workers ought to be
able to identify these special causes because of their closeness to the process.
Rule 6: Only management can improve a controlled process.
Tampering
When people don't understand variation, many things can happen: (1) trends are identified when
there are no trends, (2) trends are not identified when there are trends, (3) employees blame or credit
others for things over which others have no control, (4) past performance can’t be understood, and
(5) future plans can’t be made.
Any time you see common cause variation and make a process adjustment to correct for this
variation, you increase the variation in your process. You make it worse. Adjusting a process when
you should not is called "tampering." Consider a process operator who works at a control station,
monitoring the moisture content of a powder being processed. Every 15 minutes, he samples the
powder. If it is below 5.4 percent, he turns a knob to raise the moisture content. If it is above 5.4
percent, he lowers the moisture content. He sounds like a conscientious worker. In truth, he is
unnecessarily adding variation to the process. Likewise, for a manager or supervisor to react to
normal process variation and expect explanations or corrective actions is wasteful. "Last month there
were six instances of machine breakdowns. This month there were eight. Why are breakdowns up
25 percent?"
Rule 7: Ignorance of variation is not bliss. It increases variation.
2-4
R
TRADE
2.0 OVERVIEW
The Use of Goals

Goals encourage us and challenge us to act. Worthwhile goals are to stay in business, to provide
jobs, to reduce costs, to communicate more with customers, to better understand our processes, and
to promote teamwork. In our excitement to meet our goals, we must remember tampering.
Suppose someone sets a goal that "we will not have more than 20 off-quality pieces in any month."
Typical actions that might come from this are: (1) soliciting reasons why some months have more
than 20 off-quality reasons, (2) comparing data on months with more than 20 off-quality pieces to
the data on the "good" months, (3) having celebrations for months where the goal is met, and (4)
taking disciplinary action on workers who produce off-quality work or on supervisors when goals
are missed. While the intention of producing less off-quality work is good, any of these actions can
be harmful. They can lead to fear, cover-up of off-quality work, and breakdown of teamwork.
Those who understand variation could show that this process is a controlled process, operating at a
level that is obviously unsatisfactory to those wishing it were better. Rule 6 tells us that
management must improve the underlying system.
Rule 8: We insist on driver’s education for those who want to drive; we should insist on statistical
education for those who want to set goals for others.
Statistical Methods of Looking at Performance Measures

What are the signals to look for in process data? How do we spot unusual variation in a performance
measure? There are several tests to use. Using statistical methods, control charts can be developed.
When control limits are computed and charted the probabilities of a value exceeding the control limits
are remote. Thus, values that fall outside the control limits are considered signals of special causes
and must be investigated.
Good performance measures are also assessed using statistical methods for run charts. When the
results are plotted, values are observed in relation to the center (median) line. Eight consecutive
points above or eight consecutive points below the center line signal the presence of a special cause
causing a shift in level. Six consecutive points, each of which larger (or each of which is smaller) than
its preceding point, are the statistical signal of a trend.
When signals of special causes are identified, the local workers need to be involved. When no signals
of special causes exist, the process is operating at its best. Dr. Deming wrote that workers are not
trained until they produce output in statistical control. Once the output is in statistical control, the
process is performing to its natural limits and can be improved only by system improvements, which
is a management function.
Rule 9: We learn when a curious person sees an unusual event and acts.
2-5
R
TRADE
2.0 OVERVIEW
A Performance Measure Illustration

Figure 2.1 shows a performance measure that Baldrige Award Examiners use as a model. While they
are not prescriptive to suggest that we should be measuring the percent of shipments on time, the
style is one to copy because it: (1) has significance to the organization; (2) shows trend data for a
significant period of time; and (3) shows benchmark results from other organizations.
96 World Class from

Another Industry
94
92 Best Competitor
Percent
90 Average of Group of
Good Competitors
88
86 Industry Average
84
87 88 89 90 91 92 93 94
Year
Figure 2.1
Percent of Shipments on Time for Product Line A
Note also that while the summary is given on a year-by-year basis as an organizational measure,
there are supporting monthly results that can be assessed and studied for special causes.
Summary
Performance measures focus attention on data. There have been successes where merely the calling
of attention to expected results produced improvement. However, some methods of using data to
bring about improvement are more successful than others. Teamwork, cooperation, and openness are
assets to an organization working to improve. As Ralph Stayer of Johnsonville Sausage said,
“Everyone wants to excel. Everyone wants to be part of a winning team.” Too much zeal for results
and accountability can lead to tampering and exactly the opposite effect we wish to produce.
Few situations in the business world are entirely black and white. Education is required.
Organizations need to decide what measures they need to be competitive and work to establish
processes and systems that produce the desired results. The use of statistical methods is important.
So is the knowledge of how to work with and learn from others.
2-6
R
TRADE
2.1 DATA COLLECTION TECHNIQUES
2.1 Data Collection Techniques

At times, you may want to use information collected in one system or database in other formats.
This may be done to share data between locations, utilize another software package for specialized
data manipulation, or export data for use in reports or other documents.
Generally, to the extent possible, the creation of duplicate databases is discouraged. If the source
data are changing, the destination data will need to be updated periodically to ensure that current data
are being accessed. However, on other occasions, it is useful to take a "snapshot" of the data by
copying the desired records and fields into a separate file. This allows the user to manipulate the
data to obtain counts and trends without the data changing between operations.
The following sections will discuss items that should be considered in sharing data between systems
or software packages. The first section will discuss general considerations in downloading data. The
second section deals specifically with downloading data from the DOE Performance Indicator Data
System (PIDS).
Guide for Downloading Data

There are two different approaches to transferring data between systems. The first consists of
transferring formatted files, e.g., a .DBF file, while the second consists of transferring a formatted
ASCII text file that can be imported into the destination system. The preferred method depends on
the capability of the host and destination databases and the desired information to be transferred.
Many database and spreadsheet software packages will work with a number of different file formats.
In some cases, the file formats are directly compatible between software packages. For example,
FoxPro will directly load or accept a .DBF file created with dBASE. In other cases, a conversion
utility is required. For example, the Lotus 123 translate utility will convert a number of different
database and spreadsheet formats into a file that can be loaded directly into Lotus 123. In many
cases, these translation utilities are not directly available from the main application menu but must be
run from a higher level menu or directly from the DOS prompt. The most common formats that are
directly compatible with multiple software packages are the WKS format for spreadsheet programs
and the DBF format for databases.
If a format is not available that can be moved directly between software packages, or translated to
work with another package, most software packages will export and import a formatted ASCII data
file. The two most commonly used types of formatted ASCII data files used in transferring data
between applications are the standard data format (SDF) and the character delimited format (CDF).
2-7
R
TRADE
In the SDF format, each row contains one record and each field is a predefined size. No punctuation
is used in the records and each record ends with a carriage return and line feed. This format is
particularly useful for working with columnar data. An example of a file in SDF format is shown
below.
Smith Tom 25 234-43-5547
Jones John 41 442-78-4531
The CDF format is widely used in spreadsheets and databases. In this format, each row again
contains one record. However, fields are separated by a character, usually a comma, and character
data are enclosed by punctuation marks, usually a double quote. Leading or trailing blanks in the data
are trimmed off (i.e., fields may be varying lengths). Again, each record ends with a carriage return
and line feed.
The above file in CDF format, with comma separators and double quote delimiters, would be
captured as:
"Smith", "Tom", 25, "234-43-5547"
"Jones", "John", 41, "442-78-4531"
Although commas are most commonly used as separators and double quotes as delimiters, many
software packages allow the user to specify the characters that are used. In some cases, use of an
alternate character may be preferable. For example, FoxPro will output a tab delimited file that is
particularly useful for outputting data that will be imported into a word processor.
It should be noted that, in some software packages, the file extension is important when creating a
text file for importing into a software package. For example, dBASE requires a .TXT extension for
files that are being imported.
Some general hints on downloading data from one system:
1. Make a backup copy of the downloaded file and the file it will be imported into BEFORE you
do anything else.
2. Many database systems include some type of a unique identifying field for each record, e.g.,
an index number. If this field is accessible, including it in the download may be useful to
facilitate later downloads to update information or include additional fields from the host
database.
2-8
R
TRADE
3. When data are moved between software packages using formatted ASCII files, each field will
be imported into a different column in a spreadsheet or into a different field in a database. If
you are importing data into an existing database or spreadsheet, the order of the fields in the
downloaded file needs to match the order of the fields in the existing database. The field sizes
in the existing database must be at least as large as the corresponding fields in the downloaded
data.
4. "Layered" spreadsheets (e.g., multidimensional .WK3 worksheets) do not import well into
other programs. Frequently the layers are lost. It is better to save the spreadsheet in a
different format and then try to import it.
5. Print a sample of the downloaded file (use a small font). Even though you can view the file on
screen, some problems in data continuity are more apparent on the printed page.
6. If the file to be imported contains a large number of fields or extensive narrative data, keep in
mind that each line in the file will be treated as a separate record. Depending on the system or
the software package being used, there may be a limit to the line length. If this is the case, the
data for a single record may wrap to more than one line, and subsequent lines need to be
associated with the main record line during the import process.
Downloading Data from PIDs

The current design of DOE’s Performance Indicator Data System (PIDS) provides the capability of
downloading data in a formatted ASCII data file (CDF format). This data file is compatible with
most spreadsheet and database programs.
The delimited ASCII file download option from the PIDS report option automatically creates a file in
the same format as the upload file that is used for submitting data to PIDS. Each line is a separate
record. Fields are separated by commas, and all data are delimited with double quotes. All data in
PIDS are treated as character data. The format for a performance indicator (PI) data record in PIDS
is as follows:
"year-quarter", "facility or contractor", "PI", "PI value", "change flag", "PI narrative"
The characteristics of the fields are as follows:
Year-Quarter Character (4)

Facility Character (20)
PI number Character (8)
PI value Character (14)
Change flag Character (2)
Narrative Character (Unlimited)
2-9
R
TRADE
PI numbers or identifiers are stored without decimal points, e.g., PI 1.2 is stored as 12.
All PI values are stored in PIDS as character data. In some cases, data may not be available. In the
records where values are not available, the value is replaced by a code as follows:
-1 Currently unavailable (CU)

-2 Not available - security concerns (NAS)
-3 Not applicable (NA)
The format for a PIDS root cause (RC) data record is as follows:
"year-quarter", "facility or contractor", "RPI", "root cause", "RC value", "change flat", "PI
narrative".

RPI number Character (8)
Root cause Character (4)
RC value Character (14)
The RPI number is the PI number or identifier, preceded by the letter "R", e.g. R12.
When an error is detected after the data submission deadline, an errata form must be approved and
submitted in order for data to be changed in PIDS. The format for an errata record is as follows:
"year-quarter", "facility or contractor", "PI", "old PI value", "new PI value", "change flag",
"PI narrative", "errata basis".

PI number Character (8)
Old PI Value Character (14)
New PI value Character (14)
Errata basis Character (Unlimited)
2 - 10
R
TRADE
Use of the Internet

The use of the Internet has also become a valuable tool to collect, capture, and share information
from different sources. Internet provides many capabilities, including the capability to transfer data
files electronically. Large amounts of data can be transferred from one location to another in a matter
of seconds. This capability can improve the timeliness of obtaining information necessary to
support organizational performance measurement analyses. Many books and manuals are available
that provide information on use of Internet.
2 - 11
R
TRADE
2 - 12
R
TRADE
2.2 CHARTS, GRAPHS AND DIAGRAMS
2.2 Charts, Graphs and Diagrams
Control Charts
Statistical process control was developed as a feedback system that aids in preventing defects rather
than allowing defects to occur. One element of a process control system is control charts. Dr.
Walter Shewhart defined the concept of common and special cause variation during the 1920s at Bell
Laboratories. He developed a tool that he called the control chart, which could graphically depict
variation. This control chart, could also distinguish the two types of variation from each other, thus
allowing for the elimination of special causes and the reduction of common cause variation.
There are several types of variables data and attributes data control charts. This section will discuss
the different types of control charts, the applications of each control chart, and the interpretation of
the data.
Types of Control Charts

Variables data are quantitative data that can be measured. Some examples are the diameter of a
bearing or the thickness of a newly minted coin. Variables data are usually represented as X-bar and
R-charts and X-bar and s-charts.
Attributes data are qualitative data that can be counted. Some examples are a count of scratches per
item or a count of acceptability for a go/no-go gauge. Attributes data are usually represented as
nonconforming units and are analyzed by using p, np, c, or u control charts.
2 - 13
R
TRADE
First, determine what variable you will measure. Then gather data and chart the data accordingly
(Figure 2.2).
Attribute or
Variable Data
Attribute Data Variable Data

(Frequency Counts) (Measurable)
Number of nonconformities per Number of Small sample Large sample

inspection unit (defects unlimited) nonconformities via subgroups subgroups
yes/no, good/bad,
pass/fail (defects limited)
X-bar and X-bar and

c-chart u-chart p-chart np-chart R-chart s-chart
Figure 2.2
Sample Control Chart
X-bar and R-charts

The principal types of control charts used to analyze variables data are X-bar and R-charts. X-bar
and R-charts are used in conjunction with each other. The measurements describe a process
characteristic and are reported in small subgroups of constant sizes (usually two to five
measurements per subgroup). Construction and use of these types of charts typically involve the
following steps:
• Select the size, frequency, and number of subgroups.

• Assemble the data for the periods of interest.
• Calculate the average (X-bar) and the range R of each subgroup.
• Plot the averages and ranges on the control charts.
• Calculate the central line control limits; plot them on the control chart.
• Study the charts for stability and/or trends.
2 - 14
R
TRADE
X-bar and s-charts

X-bar and s-charts (sample standard deviation) are also used in conjunction with each other and are
measured from process characteristics. The sample standard deviation is a more efficient indicator of
process variability, especially with larger sample sizes. However, it is more difficult to calculate and
is less sensitive in detecting special causes of variation that cause only a single value in a subgroup to
be unusual. Construction and use of these types of charts typically involve the following steps:
• Select the size, frequency, and number of subgroups.

• Assemble the data for the periods of interest.
• Calculate s, s-bar (average), and the control limits for both the X-bar and s chart, and the X-
double bar chart.
• Plot the data on the control charts.
• Calculate the central line control limits; plot them on the control chart.
X-charts
Unlike X-bar and R-charts, which collect and evaluate subgroups of data, X-charts (sometimes
referred to as individuals charts) involve the analysis of individual measured quantities for indications
of process control or unusual variation. The standard deviation for X-charts is calculated using a
moving range.
The following process should be used in developing and analyzing X-charts:
• Assemble data for the periods of interest.

• Calculate the average of the individual values.
• Calculate the individual moving ranges (all ranges will be positive numbers).
• Average the ranges.
• Calculate the standard deviation and subsequent control limits for the individual values.
• Plot the average and limit lines for the individual values and analyze for trends.
2 - 15
R
TRADE
C-chart
The c-chart is the principal type of control chart used to analyze attributes data. C-charts
(sometimes referred to as "count" charts) are used in dealing with counts of a given event over
consecutive periods of time. The following process should be used in developing and analyzing c-
charts:
• Assemble data for the periods of interest.

• Calculate the data central line.
• Calculate the upper and lower control limits.
• Plot central line, and control limits.
U-chart
U-charts (sometimes referred to as "rate" charts) deal with event counts when the area of
opportunity is not constant during each period. The steps to follow for constructing a u-chart are
the same as a c-chart, except that the control limits are computed for each individual quarter because
the number of standard units varies.
P-chart
P-charts (sometimes referred to as "proportion" charts) are used to show the fraction nonconforming
of a nonstandard sample size over a constant area of opportunity (e.g., each period of interest). The
steps to follow for constructing a P-chart are the same as a c-chart, except that the control limits are
computed for each time period because the sample size varies.
NP-chart
Like p-charts, np-charts are used to analyze nonconforming items over a constant area of
opportunity; however, the np-chart focuses on the number of nonconforming items when the sample
size is constant. The steps to follow for constructing an np-chart are the same as for a p-chart.
Use of Control Charts

Control charts serve to direct management attention toward special causes of variation in a system
when they appear. Limit lines drawn on the charts provide guides for evaluation of performance.
These lines (called control lines) indicate the dispersion of data on a statistical basis and indicate if an
abnormal situation (e.g., the process is not in control or special causes are adversely influencing a
process in control) has occurred.
2 - 16
R
TRADE
In evaluating control charts, managers should look for the following indications:
• Outliers - Data that fall outside the control lines.

• Runs - Series of data points over or below the central line. A "run" of 7 consecutive points or
10 out of 11 points indicates an abnormality. Other approaches exist for identifying runs,
such as detecting two of the last three data points beyond two standard deviations (2-sigma)
and the more general CUSUM (cumulative sum discussed in Section 3.2.7) procedures, which
involve adding up standardized deviations from the calculated mean to detect abnormalities
(such as runs or trends) sooner.
• Trends - Continual rise or fall of data points. If seven data points rise or fall continuously, an
abnormality is considered to exist.
• Periodicity - Data show the same pattern of change over time. Also known as a cyclical
pattern.
In summary, the two main uses for control charts are: (1) to monitor whether the system is stable
and under control (to warn of changes), and (2) to substantiate results from changes introduced into
the system (to confirm results).
Treatment of "Outliers"
In constructing control charts, individual or groups of data points may appear near or beyond the
calculated control limit lines. Since these data appear to indicate that a system is or may not be in
control (e.g., stable), additional evaluation may be needed to ascertain if the data in question are the
result of common cause or special cause variation. If the data are clearly influenced by a one-time
aberration (e.g., special cause), there could be a basis for excluding the number or estimating what the
actual value should have been for the purpose of determining actual system control limits.
Treatment of "Rare" Events

For trending PI data using c- and u-charts, the average used to calculate control limits should be equal
to or greater than five. Where the limited nature of the data does not support the use of control
charts, the use of more sensitive trend tests may provide a better indication of actual trends. An
example would be multinomial likelihood ratio tests that involve comparing the likelihood of
postulated rates of data change (e.g., constant, increasing, or decreasing) assuming the data are
generated by a multinomial distribution.
Scaling of Control Charts

The following general criteria should be applied to the depiction of trend data on control charts: (1)
the scale should be set so that the chart can be quickly understood, and (2) the data together with the
limit lines should span at least half of the vertical axis.
2 - 17
R
TRADE
Histograms
A histogram is a bar chart. It is designed to show:
• a distribution or spread of data

• the numbers of times various measurement values occur
• groups of values so patterns of variation are easily identifiable
• the average as well as variability of a data set
Histograms are most often used to record frequency of events associated with measurements of time
or cost. Histograms may be used when the characteristics of one or more sets of data need to be
summarized, when checking for possible variations in incoming data, and when absolute values reveal
less information than trends and patterns would. The viewer can quickly and easily see the shape of
the distribution of a process, which may lead to further analysis of correlation.
Listed below are the elements of a histogram and an explanation of what each element displays:
• Horizontal axis: lists measurement values.

• Vertical axis: shows frequency or amount of values.
• Width of each bar: represents an arbitrary range of values.
• Height of each bar: represents the number of times the values within specified range are
observed.
• Pattern created by the bar heights: displays a graphic distribution.
How to Construct a Histogram

To construct a histogram, you should collect as many measurable data points as possible using a
frequency table. Spreadsheet software programs (e.g., Excel or Access) are the best way to collect
and store frequency tables. As a general statistical rule, at least 100 measurement values are required
for an accurate distribution picture and at least 30 data points are required for reliable regression
analysis.
2 - 18
R
TRADE
1. Determine the number of bars or intervals.

The number of intervals, bars, or columns is based on the total number of measurement values
collected. Example:
Number of Measurements Number of Bars

Under 50 5-7
50-100 6-10
100-249 7-12
Over 249 10-20
2. Determine the range of the bars.

Find the overall range of the data by subtracting the smallest value from the largest value. Then
divide the overall range by the number of bars to determine the range of each bar. Round your
answers up to a convenient value.
3. Determine the starting point of each interval and the bar limits.
Start with the smallest data point as the starting point of the first interval. Add the interval width
that was determined to be the range of each bar. The sum is the starting point for the next interval.
These are interval labels for the horizontal axis. It is customary to extend the scale one interval above
and below range.
4. Construct a histogram worksheet or document the spreadsheet.

Identify the purpose, date, names of data collector and data source, the sample size, time period, and
unit of measure. Record the limits for each bar, the tally of measurement values it represents, and the
total frequencies of those values.
5. Construct the histogram.

Label the axes. List the values for the bar limits from left to right on the horizontal axis. Label the
vertical axis with a scale large enough to accommodate the frequency of the tallest bar. Each data
point can appear in only one interval. To make sure that no measurement values fall on either edge
of a bar, add a logical decimal value to each limit.
Draw each bar to the height that corresponds to the frequency of the values it represents. A bar or
column is centered around the midpoint of its interval.
2 - 19
R
TRADE
6. Interpret the completed histogram.

Look for one of the patterns described below and consider the recommended actions to investigate
the causes of non-normal distributions. To make it easier to compare the distribution to a standard
or an average value, draw a dashed line for reference on the diagram. Figure 2.3 provides examples of
these different distribution types.
• Normal: Not caused by any identifiable variable. Desirable distribution, no action required.
• Saw-toothed: Range of values for each bar is not the same or certain values were excluded
from the range. Review the bar widths on the horizontal axis and/or check the source of the
data and the way the data were gathered.
• Bi-modal: Variation maybe the result of cyclical factors or two sources of data. If two
sources of data are combined, try to separate them. If cycles are expected, no action is
required.
• Skewed: Sometimes the pattern is expected, especially with measurement of time.
Normal Saw-toothed
Bi-modal Skewed
Figure 2.3
Types of Distributions
2 - 20
R
TRADE
How to Display a Histogram - Software

Excel and Access are common spreadsheet databases used for data collection and storage. They will
automatically graph frequency data into histogram format as desired. Histograms are most effective
for visual analysis of population data where correlations may be further investigated. When graphing
histograms, always ask if the picture is reasonable and logically correct; there is always the danger
that software programs can make graphically pleasing displays from illogical input.
The Pareto Principle

The Pareto Principle derives its name from Vilfredo Federico Domaso Pareto, an Italian-Swiss socio-
economist and trained, practicing engineer. As Chair of Economics in the Faculty of Law at
Lausanne University in 1892 and forerunner in the field of mathematical approaches to socio-
economics problems, Pareto conducted a study of the distribution of personal incomes of an entire
economy. This led him to postulate that:
• Personal income will be distributed among the populace along the same lines in all countries.
• The pattern of income distribution cannot be easily changed.
• The number of incomes was more concentrated among the lower income groups.
• The majority of the wealth was in the hand of a few individuals.
Pareto postulated that, in an entire economic population, only a few individuals controlled the
majority of wealth. This proved to be valid and become known as Pareto’s Law, Pareto’s Concept,
or Pareto’s Principle. Historically, the Pareto Principle has come to be more universally known as
the 80/20 rule, i.e., an 80% improvement in quality or performance can reasonably be expected by
eliminating 20% of the causes of unacceptable quality or performance.
Dr. Joseph M. Juran, Chairman Emeritus, Juran Institute, and world-renowned authority on quality
management, was the first to apply this concept to the industrial environment. He noted that in
many situations where a group of factors contribute to a common effect, only a vital few account for
the bulk of the effect, and the useful many in the population account for the rest. Many problems
associated with quality and performance adhere to this principle to identify and separate the vital
few from the useful many.
The Pareto Concept

The simplicity of the Pareto concept makes it prone to be underestimated and overlooked as a key
tool for quality improvement. Generally, individuals tend to think they know the important problem
areas requiring attention; if we know, why do the problem areas exist? The most benefit derived
from a Pareto analysis is to identify and define improvements for quality or performance
improvement.
2 - 21
R
TRADE
The idea is quite simple, but to gain a working knowledge of the Pareto Principle and its application,
it is necessary to understand these basic elements:
• Pareto Analysis - creating a pictorial array of representative sample data that ranks the parts
to the whole, with the objective to use the facts to find the highest concentration of quality
improvement potential in the fewest number of projects or remedies to achieve the highest
return for the investment.
• Pareto Diagram - the Category Contribution, the causes of whatever is being investigated,
are listed and a percentage is assigned for each (Relative Frequency) to total 100%. A vertical
bar chart is constructed, from left to right, in order of magnitude, using the percentages for
each category.
• Relative Frequency = [(Category Contribution) / (Total of all Categories)] x 100 expressed
in bar chart form.
• Cumulative Frequency = [(Relative Frequency of Category Contribution) + (Previous
Cumulative Frequency)] expressed as a line graph with points of the line determined from the
right edge of each bar or as a stacked bar chart.
• Break Point - the percentage point on the line graph for Cumulative Frequency at which
there is a significant decrease in the slope of the plotted line.
• Vital Few - Category Contributions that appear to the left of the Break Point account for the
bulk of the effect or those that account for the first 60 percent, or so, of the total.
• Trivial Many - Category Contributions that appear to the right of the Break Point, which
account for the least of the effect.
Pareto Diagram Analysis

Pareto analysis provides the mechanism to control and direct effort by fact, not by emotion. It helps
to clearly establish top priorities and to identify both profitable and unprofitable targets. Pareto
analysis is useful to:
• Prioritize problems, goals, and objectives

• Identify root causes
• Select and define key quality improvement programs
• Select key customer relations and service programs
• Select key employee relations improvement programs
• Select and define key performance improvement programs
• Address the Vital Few and the Trivial Many causes of nonconformance
• Maximize research and product development time
2 - 22
R
TRADE
• Verify operating procedures and manufacturing processes

• Product or services sales and distribution
• Allocate physical, financial and human resources
Pareto analysis may be applicable in the presentation of Performance Indicators data through
selection of representative process characteristics that truly determine or directly or indirectly
influence or confirm the desired quality or performance result or outcome. Typical observations
from Paretodiagram analyses might reveal:
• 80% of all warranty repairs of a product were attributed to 20% of its parts.
• 75% of quality defects result from 15% of operations within a process.
• 10% of the items inventoried represent 70% of the total cost of inventory.
Pareto diagram observations may show the following: (1) that the bars are roughly the same size; (2)
that it takes more than half of the categories to determine 60-80 percent; or (3) if the most frequent
problem is not the most important. STOP after the first problem is resolved and develop a more
discrete set of nonconformance cause parameters to survey, analyze, and diagram.
Process Capability
Process Capability is a determination of whether an existing process is capable of attaining the
specified (or desired) performance. Process Capability has a precise, limited definition in a
manufacturing context. This determination is based upon observing the history of the process
output data. Statistical Process Control is used to determine the expected bounds of the data. These
bounds (which are the three standard deviation control limits) are compared to the manufacturing
specification for the process. A more general utilization of this concept can be applicable to
Department of Energy facilities and processes.
Statistical Control
In order to assess process capability, the process must first be in statistical control. The data that
have been charted must be reviewed against their three standard deviation control limits. If no data
points are outside of the control limits, and no discernible trends are detected (using the criteria given
in the section on Control Charts beginning on Page 2-13), then the process is in control. The original
work by Dr. Shewhart emphasized strongly that a process should not be declared in control unless
the pattern of random variation has persisted for some time and for a sizable volume of output. He
recommended taking at least 25 samples (data points) prior to declaring a process is in control.
However, assessments using less data may be made. Using less data for the assessment takes some
experience and can become more of an art than a science.
2 - 23
R
TRADE
In some cases, a process with a single data point outside of the control limits may still be assessed as
in control. To do this requires investigation of the cause(s) for the single data point being unusual.
In short, if you can assign a reason for the value of the out-of-control point, and you can reasonably
state that such an event will not reoccur, then you should disregard this point, treat it as an outlier,
and not include it in the average or control limit calculations.
With the determination that the process is in control, you have also accepted the hypothesis that the
process data will continue to behave as the past data have. It will continue to have the same average,
and 99.7% of all future data points will fall between the upper and lower (three standard deviation)
control limits. All variations that occur in the future data can be assumed to be due to random
variation. This has important implications for management of the process, which will be explored
later in this section.
Process Capability Analysis

Once the process is in control, the upper and lower control limits may be compared to the
specification limits for the process. Generally, specification limits exist in manufacturing processes;
for example, the diameter of the bolt shall be 5 cm plus or minus .01 cm. If the upper and lower
control limits for bolt size were 5.01 and 4.98 cm, then we would fail the process capability
determination (4.98 cm is outside the lower specification limit of 4.99 cm). If we improved our bolt
manufacturing such that the new control limits were 4.993 and 5.007 cm, then we would have an
acceptable process capability. That is, it would be highly unlikely (less than .3 percent) that we
would manufacture a bolt that would fail the specification limits.
Note that even if our process has an acceptable process capability, we should still continue to
practice continuous improvement and strive to minimize the variation in bolt sizes to be as close to 5
cm as is economically feasible. Reducing the variation in component sizes does tend to reduce several
hidden costs (an undersize bolt paired with an oversize nut) when processes are highly variable. It
may be economically worthwhile to consistently provide bolts that are much better than the
specification.
If the control chart(s) are in statistical control, then calculate the process capability by comparing the
natural tolerance (NT) of the process to the engineering tolerance (ET) of the specifications:
• NT = Upper Control Limit - Lower Control Limit

• ET = Upper Specification Limit - Lower Specification Limit.
If NT £ ET, the process is capable of meeting specifications. If NT > ET, then the process is not
capable of meeting specifications.
2 - 24
R
TRADE
At this point, if you are studying variables data and your process is in control and capable, you may
want to investigate whether the process is in control with respect to the average. Depending on the
data, X-bar and R-charts, or X-bar and s-charts must be graphed. Methods to describe this charting
are described in detail in the Western Electric Company’s AT&T Statistical Quality Control
Handbook, 1984.
The same statistical rules listed in the section on Control Charts (beginning on Page 2-13) apply to
determine if the charts are in control. If the charts are in control, then calculate a Capability Ratio
(CR):
CR = (UCL - LCL) / (USL - LSL).
A good capability ratio should be less than one. Also, a Capability Index (CPK) must be calculated.
Cpk = (USL - X-bar)/3s or Cpk = (LSL - X-bar)/3s
Select whichever calculation yields the smaller index. A good capability index should be greater than
one.
Now, estimate the percent of product that will be produced out of specification by calculating a Z
score.
Zhigh = |(USL - X-bar)/s| or Zlow = |(LSL - X-bar)/s
If the absolute value of Z is >=3, then there are essentially zero defects. However, if the absolute
value of Z is < 3, there are defects. A comparison of the Z score value on a Z table will provide an
estimate of the percent of product that will be out of specification. Z tables can be found in many
statistics textbooks, including Statistical Quality Control, 1989, by Grant and Leavenworth.
Use of Process Capability at DOE Facilities

Generally, the DOE facilities are not involved in producing bolts that must meet a specified limit.
However, being aware of the process control limits is important for management of DOE facilities.
A primary goal of Statistical Process Control and Performance Indicators is to communicate the
capabilities of the process being analyzed. If the process is not in control, then efforts should be
taken to understand why the process is out of control.
If there are a number of outlying data points (outside of the control limits, also known as outliers),
the effort should be made to determine the cause(s) that drove the points out of control. These
causes should then be used as a basis for management action to bring the process into control. A
process that is forever fluctuating in an unknown manner is nearly impossible to manage. Bringing
the process “in control” allows for the use of the control limits to predict future behavior of the
process.
2 - 25
R
TRADE
If there is a discernible trend, then there is an ongoing process change. If this trend is a result of a
management action taken to improve the process and the trend is in the improving direction, then
promote the trend. Eventually the effect of the change will stabilize, and you can compute new
average and control limits once the process is again in control. The new average and control limits
may be compared to the previous average and control limits to assess the impact of the management
action. If the process data have a discernible trend, but there is no known action that caused it, then
the cause(s) of the trend must be investigated.
If a major process change has just been implemented, then one would not expect the process to be in
control. After these changes have been implemented, the process should eventually stabilize with a
new average and new control limits. If you discover that you are continually making major process
changes and the process data are becoming increasingly erratic, you may be experiencing process
tampering as previously mentioned in this handbook.
Management Philosophy
Use of Statistical Process Control with Performance Indicator data provides important information
for managers. Managers can use the process capability information in order to manage the processes
they are responsible for. The control limits provide a reasonable guarantee of the expected extreme
values of the process data. If these extreme values (and/or the average) are determined to be
unacceptable, then management must embark on process changes in order to improve the outcome of
the process.
Many management philosophies focus upon outcomes. There is justification in this, as it can be said
that without the bottom line positive outcomes and results, a company will not stay in business.
However, it must be realized that outcomes are the result of processes. In order to change existing
outcomes for the better, the process that created the outcomes in question must be understood, and
management must create changes to these processes.
Dr. Deming has stated:
• Management’s job is prediction and there is no prediction without theory.

• There are no data on the future. Data from the past must be used to form a base for
prediction.
• 94% of the changes required for improvement will require action by management.
2 - 26
R
TRADE
All of these management philosophy statements point to the importance of managers understanding
the processes that they control. In order to improve, the existing process capability must be known.
Statistical Process Control and Process Capability provide the tools to:
• Measure current performance.

• Describe the state of statistical control.
• Attain statistical control.
• Determine if process changes are needed.
• Determine the effects of process changes once implemented.
Scatter Diagram
The scatter diagram is a useful plot for identifying a potential relationship between two variables.
The data are collected in pairs on the two variables, say (xi, yi), for I=1,2 . . . n. The yi is plotted
against the corresponding xi. The shape of the scatter diagram often indicates what type of
relationship may be occurring between the two variables.
Scatter diagrams aid in the interpretation of correlation. The correlation coefficient, denoted by r, is a
measure of linear association between two variables. The values of r range from -1 to +1, inclusively.
A value of r close to zero signifies little, if any, linear association between the variables. Values of r
close to either -1 or +1 indicates a high degree of association. Positive values of r indicate that as one
variable increases, the other increases. Negative values of r indicate that as one variable increases, the
other decreases. A value of +1 (-1) indicates that the data fall on a positive (negative) straight line.
The correlation coefficient is a measure of linear association only. Scatter diagrams will indicate
curved relationships if they exist. In addition, the correlation coefficient will be badly distorted by
outlying data. The scatter diagram can be used to locate outliers to check for their validity. Scatter
diagrams and correlation coefficients are useful for identifying potential relationships. Designed
experiments must be used to verify causality.
2 - 27
R
TRADE
Several example scatter diagrams and their correlation coefficients are indicated below in Figure 2.4.
Y Y Y
X r = 0.90 X X r = 0.88
r = 0.91
Figure 2.4
Scatter Plots and Correlation Coefficients
Cumulative Sum Trend Analysis Technique

Cumulative sum trend analysis can be applied to a performance measure to detect trends more
quickly than standard control charting methods (i.e., Shewhart methods) are able to do. Cumulative
sum trend applications are contained in most of the widely used PC-based statistics packages.
Cumulative sum analysis is better at detecting small shifts (i.e., one standard deviation) of the
process mean than standard control charting and provides an earlier indication of whether
improvement efforts are succeeding or not, which is an important part of the evaluation of
performance measures used in continuous improvement efforts.
A cumulative sum chart is a plot of the cumulative sequential differences between each data point
and the process average over time. A positive slope of the graph indicates an average higher than the
process average; a flat slope indicates an average the same as the process average; and a negative
slope indicates an average less than the process average. Changes in the process averages are more
easily seen plotted as a cumulative sum than in a standard control chart format.
Detection of a shift in the process average can be accomplished graphically with a horizontal V mask
overlay on the plot of the cumulative sum data points. The user specifies the desired level of
confidence and also the power of the method for detecting a change in the process average; for
example, one standard deviation.
Determination of a trend shift is accomplished by centering the mask at a point of interest. The time
at which the shift occurred is indicated where previous points cross one of the arms.
2 - 28
R
TRADE
Suggested Reading
1. Achieving Excellence: Tools and Techniques, Technicomp, Inc., Cleveland, OH. 1993
2. American Society for Quality Control Statistics Division, Glossary and Tables for Statistical
Quality Control, 1983.
3. Burr, I. R. Elementary Statistical Quality Control, 1979.
4. Ford Motor Company, Statistical Methods Office, Continuing Process Control and Process
Capability Improvement, 1984.
5. Games, P. A. and Klare, G. R., Elementary Statistics. New York, NY. McGraw-Hill, Inc., 1967.
6. Grant, E. L. and Leavenworth, R. S., Statistical Quality Control, 1988.
7. Guidelines for Statistical Documentation of Process and Product Characterization, Control, and
Capability, Statistical Applications Subcommittee, Nuclear Weapons Complex Quality
Managers, October 30, 1990.
8. Ishikawa, K., Guide to Quality Control, 1982.
9. Leete, R. S. (Bud). Grassroots Approach to Statistical Methods, LMES, Statistical Applications
Department, Oak Ridge, TN, March 1994.
10. Mason, R. L., Gunst, R. F., and Hess, J. L. Statistical Design and Analysis of Experiments. New
York, NY: Wiley, 1989.
11. Montgomery, D. C., Introduction to Statistical Quality Control, 1985.
12. Neter, J. and Wasserman, W., and Whitmore, G. Applied Statistics. Boston, MA; Allyn and
Bacon, Inc., 1982.
13. Ryan, T. P., Statistical Methods for Quality Improvement. New York, NY: Wiley, 1989.
14. Scherkenbach, W. W., The Deming Route to Quality and Productivity, 1990.
15. Special Technical Publication 15D, ASTM Manual on Presentation of Data and Control Chart
Analysis, 1976.
16. Western Electric Company, AT&T Statistical Quality Control Handbook. Charlotte, NC; Delmar
Printing Company, 1984.
17. Wheeler, D. J. and Chambers, D. S., Understanding Statistical Process Control, 1986.
2 - 29
R
TRADE
2 - 30
R
TRADE
2.3 PRESENTATION APPROACHES
2.3 Presentation Approaches

Presentation and display of data resulting from measuring and monitoring a process or product
involves more than just drawing graphs. Effective data presentation includes understanding the type
of data to be utilized, who the intended audience is, and how the information will be used. This
section provides ideas to consider using when determining the most effective method to present your
information.
Effective Data Presentation

Before actually presenting any information, it is beneficial to evaluate and understand a few key
areas:
• Who is the audience?

• What is the intended use of the data? Will it be used to support decisions and take actions or
is it just to monitor performance?
• What is the basic message you want to communicate (here is where we are and how we are
doing)?
• What is the presentation format (report, brochure, oral presentation, etc.)?
• What is the underlying nature of the data and any assumptions?
A key point to keep in mind is that decisions should not be made based on graphs alone. No graph
can tell you everything you need to know. The purpose of presenting the data graphically is to
provide information to assist in decision making and to monitor activities or progress. Combine
graphs with narrative discussions to help the reader understand the data in the proper perspective
related to operations. Consider including the following:
• Explain what the values mean for your organization.

• Relate to historical performance.
• Identify influencing factors (nature of operations, seasonal changes, significant management
initiatives).
• Relate to management performance goals.
• Explain significant increases or decreases.
2 - 31
R
TRADE
Use the data to try to answer the following questions for the reader:
• Is there a trend over time?

• Should I take any action? What kind of action?
• What contributes the most to the total (focus on the vital few)?
• Are we focusing on the highest priority actions?
Figures 2.5 through 2.10 demonstrate how the chart type you choose can influence the message you
are trying to convey.
Overdue Corrective Actions
300
250
200
# Late
150
100
5 0
0
Jan Feb Mar Apr May Jun
Figure 2.5
Is there a trend over time?
A commonly used approach
We often look at information to determine if there is a trend over time. Figure 2.5 shows an example
of a graph commonly used to present data over a period of time. However, looking only at the total
late items over time provides limited information. A more meaningful approach might be to look at
the overdue rate, which allows you to factor in changes in both the number of open actions and the
number that are late. This could be represented by the number of late corrective actions divided by
the total number of open corrective actions, such as in Figure 2.6.
2 - 32
R
TRADE
% Overdue Corrective Actions
100
75
Percent
50
25
0
Org. 1 Org. 2 Org. 3
Figure 2.6
Is there a trend over time?
Same data, more information
Figure 2.6 uses the same data as Fig. 2.5, but provides more information by presenting the data
differently. The combination graph shows both trends in the overall rate and the individual
organizations' rates over time. Apply statistical analysis techniques to determine the presence and
significance of any trends. Evaluate individual components of the rate (numerator and denominator)
to determine which organization is influencing the overall rate the most.
2 - 33
R
TRADE
% Overdue Corrective Actions vs. Goal
100
75
Percent
50
25 Target
Actual
0
J F M A M J J A S O N D
Figure 2.7
Progress toward a defined target
If targets are used, they should be realistic and achievable. The graph shown by itself in Figure 2.7
infers that the target will be met.
A control chart, as shown in Figure 2.8 for example, may lead you to question the assumption that
the defined target will be met. Different types of actions are needed to influence the overall totals for
systems in control versus systems out of control.
Notice how axis scale selection can significantly influence the impression given. Additionally,
horizontal grid lines can make control lines difficult to read.
2 - 34
R
TRADE
400 target 1000
800
300
600
200
target
400
100
200
0 0
Jan Feb Mar Apr May Jun Jan Feb Mar Apr May Jun
Figure 2.8
Should I take any action??
What kind of action?
System stability must be considered
A chart such as the one shown in Figure 2.9 will help you determine where to focus attention to have
the most impact on the overall rate. In this chart, consideration is given to both components of the
overall rate to determine what contributes the most to the total.
Org. 1
Org. 3
19%
27%
Org. 1
40%
Org. 3
50%
Org. 2
31% Org. 2
33%
Figure 2.9
What contributes the most to the total?
Focus on the vital few
2 - 35
R
TRADE
The graph shown in Figure 2.10 tells you whether high and/or medium priority actions are
dominating the overdue items. It also shows historical trends for overdue items for each priority
level and for the total overdue. If your presentation is in color, you can choose traditional colors to
indicate priority levels (red showing danger for high priority overdue items, yellow showing warning
for medium priority, and green indicating low priority).
Overdue Corrective Actions - by

Priority Level
Number
High Med Low
Figure 2.10
Are we focusing on the highest priority actions?
Focus on the critical areas
Choosing the Right Chart Type

There are a few universal rules about which chart type best portrays any given set of data, and in
many cases the same data set can be shown many different ways. The hard part is determining
which chart type emphasizes the point you are trying to make or puts the right "spin" on the data.
The following provides some general comments on different chart types.
Vertical bar chart: Vertical bar charts are used to show how values change over time. They are
typically used for a limited time series (i.e., just a few years, quarters, months, or whatever time
period you are working with). Vertical bar charts are good for handling multiple series for
comparison purposes.
Stacked vertical bar chart: Stacked vertical bar charts convey the same information as ordinary
vertical bar charts, but allow you to display subelements which contribute to the overall bar.
This may be helpful in understanding changes from one period to another.
2 - 36
R
TRADE
Vertical line chart: Vertical line charts are best for showing changes in a group of values over
longer periods of time. They are also recommended if you are displaying several groups of data
simultaneously. Control limits are often included in vertical line charts to evaluate variability.
Horizontal bar chart: Horizontal bar charts are best for simple comparisons of different
individual values at one time. A vertical bar, line, area or 3-D riser chart would be more beneficial
if you want to evaluate change over time.
Pie chart: Pie charts are often the best way to portray the contribution of parts to a whole.
They are used to show a “snap-shot” at a specific time.
Scatter chart: Scatter charts show the correlation of two sets of numbers by plotting where the
variables intersect. Scatter charts are useful when the coordinates on the horizontal scale, often
time intervals, are irregular.
Histogram: Histograms show the frequency of the values in a set of data. Data is plotted in
increasing or decreasing order based on the frequency count for each data value.
Chart Design: A Few Tips and Hints

The charting area is the focal point of the chart. The graphical, dramatic representation of numbers
as bars, risers, lines, pies, and the like is what makes a chart so powerful. So make your charting area
as prominent as possible without squeezing other chart elements off the page. If you can still get
your point across without footnotes, axis titles, or legends, do so to make the charting area bigger.
However, remember that the document needs to communicate enough information to be a stand-alone
document. The following is a list of tips to keep in mind when designing your chart.
• Less is more: Do not try to put too many series in a chart. Line charts are especially
intolerant of overcrowding. More than three or four lines, particularly if the lines follow
much the same direction, is visually confusing. The only exception to this rule is creating a
line chart of several series that people would not expect to be similar.
• Group bars to show relationships: Group bars together tightly if you are trying to suggest
that they belong together. If you are showing a group of bars over a series of years, for
example, it makes sense to cluster the bars for each year and leave a little extra space between
years. If there is no need to do this with your chart data, put more space between your bars
and make them a little wider so they are easier to see.
• Add definition with black outlines: Give the bars in bar charts, the slices in pie charts, and
the risers in 3-D charts a little definition by making their outlines black, or a dark, brilliant
color. If you are making your chart into a slide, the people at the back of the room will
appreciate being able to distinguish the elements.
2 - 37
R
TRADE
• Use grids in moderation: When using grid lines in your charting area, use only as many as
are needed to get an approximate idea of the value of any given data point in the chart. Too
many grid lines create visual clutter. Balance horizontal and vertical grid lines so that the
rectangles they create are not too long and narrow or tall and narrow. Use soft colors, such as
gray, for grid lines. Once you have defined the color and weight of the grid lines, make sure
the chart frame (the frame around the charting area) is black or a dark, brilliant color and
heavier than the grid lines.
• Choose colors carefully: When choosing colors, use your company's corporate colors where
possible and appropriate. Failing that, you can use software-supplied templates or color
wheels. Also consider where your chart will appear. If it is going to be part of a computer
screen show or a slide presentation in a large room, use strong, coordinating colors that attract
attention and help the people at the back of the room distinguish the individual series.
However, if it is going in a publication where it will be examined at close range, keep the
colors softer so you do not overwhelm the reader.
• Limit use of typefaces: Use one typeface, or at most two, on each chart, and use the same
size and weight for similar elements such as the axes and legend text. A recommended setting
for these is in 12 to 18 points and bold. If you use the bold and italic fonts in a typeface, as
well as different sizes, you can generate enough typographic variety without going outside
that type family.
• Choose legible typefaces: Pick a typeface that looks clear in smaller sizes and in bold,
especially if your chart is to be printed in a small size in a publication, or if it will be viewed
by a large audience in a big room. If your title is big enough, you can use just about any
typeface for it, and it will be legible. However, for legend text, axes, footnotes and the like,
take more care. Use faces that are neither too light nor too heavy.
• Set type against an appropriate background: Be careful about the background behind your
type. Some color combinations, such as pink or violet type and a medium or dark blue
background, could make your audience feel a little dizzy. If you are using a dark background
color, your type must be bright enough to be readable; it should not look as if the background
is trying to "swallow it up." If you are using light type on a dark background, use a bold
weight, especially with smaller type sizes. Complex fill patterns in the background can also
make type hard to read, particularly smaller items like legend text and axis scales.
• Use pattern fills with moderation: Many charting software packages can create just about
any kind of color combination or fill pattern you can imagine. But do not become carried
away with color and patterns without thinking about your output device. Sophisticated fill
patterns take up more disk space and take longer to print on color printers.
2 - 38
R
TRADE
Sample Charts
Table 2.1 provides sample charts that may be helpful when choosing the appropriate chart type for
your presentation.
Table 2.1 Sample Charts
LINE
Shows trends or changes in data over a period of time;

similar to an area chart, but emphasizes time flow and
rate of change, rather than amount of change.
PIE
Shows the relationship or proportions of parts to a

whole (always contains just one data series); good for
highlighting a significant element.
XY (Scatter)
Shows the relationship or degree of relationship

between the numeric values in different groups of
data; useful for finding patterns or trends and for
determining whether variables are dependent upon or
affect one another.
COMBINATION
Shows related data that are measured in different units

(up to four axes can be used in a combination chart);
combines different data to show comparisons and
relationships that might not otherwise be recognized.
2 - 39
R
TRADE
3-D AREA
Shows a 3-D view of an area chart, emphasizes the

sum of plotted values, and separates data series into
distinct rows to show differences between the series.
3-D BAR
Shows a 3-D view of a bar chart; similar to a column

chart, but categories are organized vertically instead of
horizontally.
AREA
Shows the relative importance of values over a period

of time; similar to a line chart, but emphasizes the
amount of change (magnitude of values) rather than
the rate of change.
BAR
Shows individual figures at a specific time or draws

comparisons between items, but not to a whole;
similar to a column chart, but categories are organized
vertically instead of horizontally, placing less
emphasis on time flow.
COLUMN
Shows variation over a period of time or draws

comparison between items, but not to a whole;
horizontal orientation suggests time flow more than a
bar chart.
2 - 40
R
TRADE
3-D COLUMN
Shows a 3-D view of a column chart; emphasizes

comparison of data points along two axes - a category
axis and a data series axis - so that you can compare
data within a data series more easily and still view
data by category.
3-D LINE
Shows lines in a line chart as 3-D ribbons; makes

individual lines easier to view, particularly when they
cross, while still showing all data series in one chart
for comparison.
3-D PIE
Shows a pie chart with height to the slices; places

additional emphasis on data values that are in front.
Simplified Graph/Report Generation

Spreadsheet and database software can be used to generate pareto charts, bar charts, pie charts, and
scatter diagrams. The choice of which software to use is often based on personal preference or
company policy.
However, software for more complex analyses and presentation beyond that performed by common
spreadsheet and database software packages can be difficult to find. A comprehensive list of
software used for data acquisition, data presentation, statistical analysis, and other subjects related to
quality assurance and quality control is provided in the annual Quality Progress Software Directory
produced by the American Society for Quality Control (ASQC). The 1995 report, published in
March 1995, listed over 500 software packages. There are two parts to the annual Quality Progress
Software Directory:
• A two-dimensional matrix lists each software package and indicates its applicability across 19
categories, such as calibration, data acquisition, and management.
2 - 41
R
TRADE
• An index of each of the software packages (alphabetical by company) that includes a brief
description of the software, hardware requirements, and price. Included in the description are
company telephone and fax numbers and addresses, so the company can be contacted directly
for more information.
The annual ASQC Quality Progress Software Directory can be obtained by writing to: ASQC
Quality Press, P.O. Box 3005, Milwaukee, WI 53201-9488; or by telephoning 1-800-248-1946.
2 - 42
R
TRADE
Appendix A
Glossary of Terms
APPENDIX A
GLOSSARY OF TERMS
Glossary of Terms
Note: The following glossary was compiled from various documents used within the Department of
Energy. Where shown, the number in parentheses indicates the associated reference
document (see page A-15).
A
Acceptance
The decision that an item, process, or service conforms to specified characteristics defined in
codes, standards, or other requirement documents. (3)
Accountability
Responsibility for an activity, accompanied by rewards and recognition for good performance,
and adverse consequences for performance that is unreasonably poor. (2)
Activity
Actions taken by a program or an organization to achieve its objectives. (2)
Assessment
An all-inclusive term used to denote the act of determining, through a review of objective
evidence and witnessing the performance of activities, whether items, processes, or services
meet specified requirements. Assessments are conducted through implementation of the
following actions: audits, performance evaluations, management system reviews, peer reviews,
or surveillances, which are planned and documented by trained and qualified personnel. (5)
Assessment/Verification
The act of reviewing, inspecting, testing, checking, conducting surveillances, auditing, or

otherwise determining and documenting whether items, processes, or services meet specified
requirements. DOE Order 5700.6C uses the terms assessment and verification synonymously.
This order defines these terms by who is performing the work; assessments are performed by
or for senior management, and verifications are performed by the line organization. (9)
R
Appendix A - 1
TRADE
APPENDIX A
GLOSSARY OF TERMS
A
Audit
A planned and documented activity performed to determine by investigation, examination, or

evaluation of objective evidence the adequacy of and compliance with established procedures,
instructions, drawings, and other applicable documents and the effectiveness of implementation.
Audit should not be confused with surveillances or inspection activities performed for the sole
purpose of process control or product acceptance. (3)
Also, a systematic check to determine the quality of operation of some function or activity.
Audits may be of two basic types: (1) performance audits in which quantitative data are
independently obtained for comparison with routinely obtained data in a measurement system,
or (2) system audits of a qualitative nature that consist of an on-site review of a laboratory's
quality system and physical facilities for sampling, calibration, and measurement. (5)
B
Baseline
The current level of performance at which an organization, process, or function is operating. (2)
Benchmarking
To measure an organization's products or services against the best existing products or services
of the same type; the benchmark defines the 100 percent mark on the measurement scale. (2)
Also, the process of comparing and measuring an organization's own performance on a particular
process against the performance of organizations judged to be the best of a comparable industry.
(3)
Bottom Up
Starting with input from the people who actually do the work and consolidating that input
through successively higher levels of management. (2)
Appendix A - 2 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
C
C-Chart
Also referred to as “count” charts, these are used in dealing with counts of a given event over
consecutive periods of time. Many of the initial DOE performance indicators involve counts of
events for consecutive calendar year quarters, making c-chart analysis of these indicators
appropriate. (1)
Cascaded Down
Starting with a top level of management, communicated to successively lower levels of

management and employees.
Characteristics
Any property or attribute of an item, process, or service that is distinct, describable, and
measurable.
Common Causes of Variation
Indicated by statistical techniques, but the causes themselves need more detailed analysis to be
fully identified. Common causes of variation are usually the responsibility of management to
correct, although other people directly connected with the process sometimes are in a better
position to identify the causes and pass them on to management for correction. (1)
Continuous Improvement
The undying betterment of a process based on constant measurement and analysis of results
produced by the process and use of that analysis to modify the process. (2)
Also, where performance gains achieved are maintained and early identification of deteriorating
environmental, safety, and health conditions is accomplished. (1)
Control Charts
The two main uses for these charts are to monitor whether the system is stable and under
control (to warn of changes), and to substantiate results from changes introduced into the
system (to confirm positive results). (1)
R
Appendix A - 3
TRADE
APPENDIX A
GLOSSARY OF TERMS
C
Control Lines/Limits
The “limit lines” drawn on charts to provide guides for evaluation of performance indicate the
dispersion of data on a statistical basis and indicate if an abnormal situation (e.g., the process is
not in control or special causes are adversely influencing a process in control) has occurred. (1)
Also, two control limits are the statistical mean (average) plus three times the standard deviation
and the statistical mean minus three times the standard deviation. (5)
Corrective Action
Measures taken to rectify conditions adverse to quality and, where necessary, to preclude
repetition. (5)
Criteria
The rules or tests against which the quality of performance can be measured. They are most
effective when expressed quantitatively. Fundamental criteria are contained in policies and
objectives, as well as codes, standards, regulations, and recognized professional practices that
DOE and DOE contractors are required to observe. (3)
D
Data
Factual information, regardless of media and format, used as a basis for reasoning, discussion, or
calculation. (5)
Data Reduction
Any and all processes that change either the form of expression or quantity of data values or
numbers of data items. (5)
Appendix A - 4 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
D
Data Validation
The systematic effort to review data in order to ensure acceptable data quality. A systematic
process for reviewing a body of data against a set of criteria to provide assurance that the data
are adequate for their intended use. A systematic review process conducted to confirm the
degree of truth in an analytical measurement. The process includes the review of all pertinent
sample analysis and quality assurance/quality control (QA/QC) data compared to recognized
standards or criteria. Data validation consists of data editing, screening, checking, auditing,
verification, certification, and review. The "screening" process may be done by manual and/or
computer methods, and it may use any consistent techniques, such as sample limits, to screen
out impossible values or complicated acceptable relationships of the data with other data. (5)
Distribution Charts
Data are divided into categories of interest (e.g., root causes or reporting elements). It is then
graphed as a stacked bar chart to compare the relative contribution of each category to the total.
(1)
Distribution Diagram
A block diagram showing data in order of contribution to the total. The horizontal axis of the
Distribution Diagram lists the most frequent item in the performance indicator population on the
left and progresses in descending order to the least frequent item on the extreme right. The
cumulative total of the items is reflected above the block at each interval. By structuring the
data in this form, the Distribution Diagram provides a focus on the largest contributing items in
each performance indicator. (1)
G
Goal
The result that a program or organization aims to accomplish. (2)
Also, a statement of attainment/achievement, which is proposed to be accomplished or attained

with an implication of sustained effort and energy. (3)
R
Appendix A - 5
TRADE
APPENDIX A
GLOSSARY OF TERMS
G
Guideline
A suggested practice that is not mandatory in programs intended to comply with a standard.
The word "should" or "may" denotes a guideline; the word "shall" or "must" denotes a
requirement. (3)
I
Item
An all-inclusive term used in place of the following; appurtenance, sample, assembly,

component, equipment, material, module, part, structure, subassembly, subsystem, unit,
documented concepts, or data. (7,8)
L
Lessons Learned
A summary intended for the beneficial use of the receiver, of conditions detected at any facility
that may include techniques and actions employed to correct the condition. DOE Order
5000.3B suggests that facilities use the Occurrence Reporting and Processing System (ORPS) to
identify good practices and lessons learned. (8)
A “good work practice” or innovative approach that is captured and shared to promote repeat
application. A lesson learned may also be an adverse work practice or experience that is
captured and shared to avoid recurrence. (11)
Limit Lines
Lines drawn on charts to provide guides for evaluation of performance. (1)
Line Manager
Includes all managers in the chain of command from the first-line supervisors to the top manager.
(5)
Appendix A - 6 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
M
Management
All individuals directly responsible and accountable for planning, implementing, and assessing
work activities. (5)
Mean
The arithmetic average of a set of numbers. (5)
Measurement
The quantitative parameter used to ascertain the degree of performance. (2)
Metric
Used synonymously with measurement. (2)
O
Objective
A statement of the desired result to be achieved within a specified time. (3)
Occurrence
An unusual or unplanned event having programmatic significance such that it adversely affects
or potentially affects the performance, reliability, or safety of a facility. (3)
Outliers
Data that fall outside the control lines. (1)
R
Appendix A - 7
TRADE
APPENDIX A
GLOSSARY OF TERMS
P
Parameter
A quantity that describes a statistical population or any of a set of physical properties whose
values determine the characteristics or behavior of something. (13)
Pareto Analysis
A type of analysis (also known as “distribution diagram”) that focuses attention on areas that
have the most influence on the total, facilitating the assignment of resources in order to prioritize
improvement efforts. (1)
Performance Based
Being associated with the outcome rather than the process. (2)
Performance Goal
The target level of outcomes expressed as a tangible, measurable objective against which actual
achievement can be compared. (2)
Performance Indicator(s)
A parameter useful for determining the degree to which an organization has achieved its goals.
(2)
Also, a quantifiable expression used to observe and track the status of a process. (3)
Also, the operational information that is indicative of the performance or condition of a facility,
group of facilities, or site. (6)
Performance Measure(s)
Encompassing the quantitative basis by which objectives are established and performance is
assessed and gauged. Includes performance objectives and criteria (POCs), performance
indicators, and any other means that evaluate the success in achieving a specified goal. (3)
Also, the quantitative results used to gauge the degree to which an organization has achieved its
goals. (2)
Appendix A - 8 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
P
Performance Objectives and Criteria (POC)
The quantifiable goals and the basis by which the degree of success in achieving these goals is
established. (3)
Periodicity
Data that show the same pattern of change over time, frequently seen in cyclical data. (1)
Q
Quality
A degree to which a product or service meets customer requirements and expectations. (9)
Quality
Actions that provide confidence that quality is achieved. (9)
Quality Management
The management of a process to maximize customer satisfaction at the lowest cost. (2)
R
Root Cause
The basic reasons for conditions adverse to quality that, if corrected, will prevent occurrence or
recurrence. (3)
Root Cause Analysis
An analysis performed to determine the cause of part, system, and component failures. (10)
R
Appendix A - 9
TRADE
APPENDIX A
GLOSSARY OF TERMS
R
Runs
Series of data points above or below the central line. A “run” of seven consecutive points or 10
out of 11 points indicates an abnormality. (1)
S
Self-Assessment
A systematic evaluation of an organization's performance, with the objectives of finding

opportunities for improvement and exceptional practices; normally performed by the people
involved in the activity, but may also be performed by others within the organization with an
arms-length relationship to the work processes. (2)
Senior Management
The manager or managers responsible for mission accomplishment and overall operations. For
DOE, DOE Cognizant Secretarial Office, and Field/Operations Office Managers are responsible
for mission accomplishment and overall operations. For DOE management and operating
(M&O) contractors, the General Manager, or similar top position is responsible for mission
accomplishment and overall performance in accordance with the requirements of their contracts
or other agreements. (9)
A continuous process of comparing performance with desired objectives to identify

opportunities for improvement. Assessments are conducted by individuals, groups, or
organizations relating to their own work. (3)
Site
The area comprising or within a DOE laboratory or complex with one or more DOE facilities.
Situation Analysis
The assessment of trends, strengths, weaknesses, opportunities, and threats, giving a picture of
the organization's internal and external environment to determine the opportunities or obstacles
to achieving organizational goals; performed in preparation for strategic planning efforts. (2)
Appendix A - 10 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
S
Special Cause of Variation
Also known as assignable causes of variation. A cause that is specific to a group of workers, a
particular worker, a specific machine, or a specific local condition. Examples are water in a
gasoline tank or poor spark plugs. (12)
Stakeholder
Any group or individual who is affected by or who can affect the future of an organization-
customers, employees, suppliers, owners, other agencies, Congress, and critics. (2)
Standard Deviation
A statistic used as a measure of the dispersion in a distribution, the square root of the arithmetic
average of the squares of the deviations from the mean. (5)
Strategic Planning
A process for helping an organization envision what it hopes to accomplish in the future,
identify and understand obstacles and opportunities that affect the organization's ability to
achieve that vision, and set forth the plan of activities and resource use that will best enable the
achievement of the goals and objectives. (2)
Surveillance
An act of monitoring or observing a process or activity to verify conformance to specified

requirements.
R
Appendix A - 11
TRADE
APPENDIX A
GLOSSARY OF TERMS
T
Task
A well-defined unit of work having an identifiable beginning and end that is a measurable
component of the duties and responsibilities of a specific job.
Top Down
To start with the highest level of management in an organization and propagating through
successively lower levels of the organization. (2)
Total Quality Management (TQM)
A management philosophy that involves everyone in an organization in controlling and

continuously improving how work is done in order to meet customer expectations of quality. (2)
Also, the management practice of continuous improvement in quality that relies on active
participation of both management and employees using analytical tools and teamwork. (4)
Trend Analysis
A statistical methodology used to detect net changes or trends in levels over time. (5)
An analysis of parts, systems, component surveillances, performance, and operating histories to

determine such things as failure causes, operational effectiveness, cost-effectiveness, and other
attributes. (10)
Continual rise or fall of data points. If seven data points rise or fall continuously, an
abnormality is considered to exist. (1)
Appendix A - 12 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
U
U-Chart
Also referred to as “rate” charts, deal with event counts when the area of opportunity is not
constant during each period. The steps to follow for constructing a u-chart are the same as a c-
chart, except that the control limits are computed for each individual time period since the
number of standards varies. (1)
Unit of Measure
A defined amount of some quality feature that permits evaluation of that feature in numbers. (4)
V
Validation
A determination that an improvement action is functioning as designed and has eliminated the
specific issue for which it was designed. (4)
Also, to determine or test the truth or accuracy by comparison or reference. (5)
Verification
A determination that an improvement action has been implemented as designed. (4)
Also, the process of evaluating hardware, software, data, or information to ensure compliance
with stated requirements. The act of reviewing, inspecting, testing, checking, auditing, or
otherwise determining and documenting whether items, processes, services, or documents
conform to specified requirements. (5)
Vertical Axis Scaling
The following general criteria should be applied to the depiction of trend data on control charts:
(1) scale should be set so that the chart can be quickly understood, and (2) data together with the
limit lines should span at least half of the vertical axis. (1)
R
Appendix A - 13
TRADE
APPENDIX A
GLOSSARY OF TERMS
W
Work
A process of performing a defined task or activity; e.g., research and development, operations,
maintenance and repair, administration, software development and use. (5)
Also, the process of performing a defined task or activity, for example, research and
development, operations, maintenance and repair, administration, software development and
use, inspection, safeguards and security, data collection, and analysis. (9)
X
X-Chart
Involve the analysis of individual measured quantities for indications of process control or
unusual variation. The standard deviation for x-charts (also referred to as individual charts) is
calculated using a moving range. (1)
Appendix A - 14 R
TRADE
APPENDIX A
GLOSSARY OF TERMS
References
1. DOE-STD-1048-92, DOE Performance Indicators Guidance Document, U.S. Department of

Energy, Washington, DC, December 1992.
2 Total Quality Management Implementation Guidelines, Energy’s Quality Journey, United States
Department of Energy, Washington, DC, December 1993.
3. DOE/NV Family Quality Glossary, Revised 08/04/93.
4. Strategic Planning Definitions, source unknown.
5. DOE Richland Operations Glossary, DOE/RL-90-28, Rev. 2, November 1993, Pages G-3 through
G-23.
6. DOE Order 5480.26, Trending and Analysis of Operations Information Using Performance
Indicators, U.S. Department of Energy, Washington, DC, January 15, 1993.
7. DOE 5000.3B, Occurrence Reporting and Processing of Operations Information, January 19, 1993.
8. Westinghouse Occurrence Reporting and Processing of Operations Information Guidance, Current

Draft.
9. DOE Order 5700.6C, Quality Assurance, U.S. Department of Energy, Washington, DC, August 21,
1991.
10. DOE Order 4330.4A, Maintenance Management Program, U.S Department of Energy, Washington,
DC, October 17, 1990.
11. DOE-STD-7501-95, Development of DOE Lessons Learned Programs, U.S. Department of Energy,
Washington, DC, May 1995.
12. Quality, Productivity, and Competitive Position, W. Edwards Deming, Massachusetts Institute of
Technology, 1982.
13. Webster’s New Collegiate Dictionary, G&C Merriam Company, Springfield, Massachusetts, 1975.
R
Appendix A - 15
TRADE
APPENDIX A
GLOSSARY OF TERMS
Appendix A - 16 R
TRADE
Appendix B
Performance Measurement Process

Case Study
APPENDIX B
PERFORMANCE MEASUREMENT PROCESS CASE STUDY
Cominman Company
Company Background
The Communications & Information Management (Cominman) Company has been in business for 20
years, providing, on a national scale, communications and information management services. The
company’s warehouse, part of the Property Management Division, provides storage and excess
services for company property in the custody of 25 divisions. The warehouse department has a
staff of ten personnel: a warehouse supervisor, four property specialists, one property clerk, three
drivers, and one data entry clerk. The warehouse makes approximately 50 pickups per week at
company locations that include remote areas.
Process Description
To request services from the warehouse, a division customer telephones the warehouse property
clerk requesting a pick-up of property for storage or excess. The customer provides the clerk with
the property identification number or serial number for each piece of property to be picked up and
brought to the warehouse. There are typically one to twenty pieces of property per pick-up. If a
pick-up date is not requested by the customer, a date will be provided to the customer by the
property clerk. The property clerk completes a property transfer form, which reflects the date of
the call, customer’s name, division, location, property identification number and date scheduled for
pick-up. A goal of the warehouse is not to exceed three days from the date of the call to the time of
the pick-up, unless a special date has been requested by the customer. The warehouse receives
approximately ten calls per week for pick-ups on special dates. On the scheduled pick-up day, the
assigned driver takes the transfer form to the designated location. The driver is responsible for
ensuring each piece of property matches the property identification numbers or serial numbers listed
on the transfer form. After the truck is loaded, the driver obtains the customer’s signature on the
transfer form. The driver also signs the form and provides the customer with a copy acknowledging
receipt.
The driver returns to the warehouse, where a property specialist annotates the date on the transfer
form, unloads the truck, and provides the data entry clerk with the signed copies of the form. The
data entry clerk enters the information from the transfer form into the automated accountable
property system and the transfer forms are then filed. The data entered are intended to transfer
accountability from the division customer to the warehouse.
Appendix B - 1
R
TRADE
APPENDIX B
At the end of the month, division customers receive a computer-generated property list indicating the
accountable property in their location for which they are responsible. The customer reviews this
report for accuracy. If the customer records do not agree with this listing, the customer calls the
warehouse supervisor who logs the complaint with the following information: date of the call,
division name, property location, date of the property list, and discrepancies. The supervisor
assigns a property specialist to resolve these discrepancies.
Issue
The warehouse supervisor had recently attended a Quality Leadership Seminar during which time a
workshop was conducted on Performance Measurements. During a review of the telephone
complaint logbook, a supervisor realized that customer complaints were beginning to increase. The
supervisor felt that developing Performance Measurements for the warehouse process would be
beneficial. Why?
• To ensure that customer requirements are being met. How do we know that we are providing
the service that our customers require?
• To ensure an understanding of the process by all warehouse employees.
• To ensure we are meeting value-added objectives or that we are being effective and efficient.
• To ensure decisions are based on fact, not on emotion.
• To show where improvements need to be made. Where can we do better? How can we
improve?
The Quality Leadership Seminar stressed the value of a team-based approach when solving problems
or establishing performance measures. The supervisor, therefore, decided to involve her entire staff
in developing performance measurements for their process. The supervisor was the team leader; a
trained facilitator was requested to assist them; and the team elected the property clerk as the
secretary. They were ready to start.
The group is responsible for many processes, such as delivering property,

conducting inventory, etc. For purposes of simplicity, this case study only
addresses the process of picking up property or storage.
Appendix B - 2
R
TRADE
APPENDIX B

The supervisor thought, “Where do we begin? What is the very first thing we have to do?”
Well first, she thought, we need to define our current process so all my team members can share a
common understanding of what we do. The tools? Brainstorming and Flow Diagramming.
• Brainstorming is a group technique for generating new, useful ideas. It uses a few simple
rules for discussion that increase the chances for originality and innovation.
• Flow diagramming is a method of graphically describing the activities and sequence that we
perform to produce some output in a process. Before you try to control a process, you must
understand it. Flow diagramming is basic to understanding our work and the way we function
as a whole.
So the supervisor gathered the department together, and they began to document all the steps in their
work process. Post-it sheets were all over the wall! What started their work? A telephone call from
a customer. What ended their process? An accurate property list. They wrote down all the related
activities between these two boundaries (input/output) in the order in which they occurred. The
department realized that the flow diagramming session was certainly a time of “discovery.”
Contrary to what they thought, they did not proceed quickly and they did not proceed methodically
through their process from beginning to end, capturing every detail the first time through. A lot of
discussion took place. Finally, the department reviewed the completed diagram to see if they had
missed any activities or decision points and verified the accuracy of the flow diagram. Is this the
actual process? Yes, they all agreed. A lot of time was spent on this effort. However, the
supervisor was very pleased. “We have an invaluable tool; a map of our process,” she stated (Figure
B-1). “Now we can start thinking about performance measurements.”
Appendix B - 3
R
TRADE
APPENDIX B
Figure B-1
Department’s process for picking up property
Start
Receive call
from customer
Schedule pick-up Pick-Up

of property Property Process
Arrive at site
Pick-up
specific property
Return to
warehouse
Update property
Database
record
Updated property list Property

sent to customer list
Review by
customer
Yes Concurs with

Agree? End
list
No
Notify warehouse
management
Resolve
Revise list Database
End
Appendix B - 4
R
TRADE
APPENDIX B
Or can we? The supervisor thought for a moment. She learned in her workshop that performance
measurement is better thought of as an overall management system, involving prevention and
detection aimed at achieving conformance of the work product or service to our customer’s
requirements. Performance measurement is primarily managing outcome, and one of its main
purposes is to reduce or eliminate overall variation in the work product or process. The goal is to
arrive at sound decisions about actions affecting the product or process and its output.
So she asked her department, “What is our product? What is our output?” The department came up
with two outputs: (1) a property list for their customer, and (2) removal and storage of company
property.
She then told her department that measurements should focus on their customer’s needs. They
should measure only what is important: Things that impact customer satisfaction, goals given by
management, and their own internal objectives. Keeping the customer in mind, she asked her
department, “What is the objective of our two outputs?” They responded immediately. Their
objectives were (1) a current, accurate property list for our customers; and (2) timely pick up and
removal of property. (Refer to flow diagram Figure B-2).
Appendix B - 5
R
TRADE
APPENDIX B
Figure B-2
Step 1: Identify Process - Continued
Department’s process for picking up property
Start
Receive call
from customer

of property
Property Process
Output?
1. Property list for Arrive at site
customer
2. Moves property
Pick-up
specific property
Return to
warehouse
Update property
record Database

Review by
customer
Yes Concurs with

Agree? End
list
No
Objectives?
1. Current, accurate list Notify warehouse
2. Timely service management
Resolve
End
Appendix B - 6
R
TRADE
APPENDIX B
Step 2: Identify Critical Activity to be Measured

The next step is determining how objectives will be met.
One of the topics discussed in the Performance Measurement workshop was involving employees in
the design and implementation of the measurement system. This gives them a sense of ownership
and improves the quality of the measurement system.
The supervisor called her department together again. “We are now ready to identify specific critical
activities to set up our control points. Controlling, or keeping things on course, is not something we
do in the abstract. Control is applied to a specific critical activity.”
She continued to instruct her department that they should examine each activity in the process and
identify those that significantly impact total process efficiency and effectiveness. Then they should
establish measurements for these critical activities.”
Ask the following: Does it relate, directly or indirectly, to the ultimate goal of
customer satisfaction? Every critical activity should.
The department began to brainstorm. “Keep focused,” the supervisor reminded. “Keep looking at
our objectives. How can we accomplish these?” The supervisor stated that as they approached the
data collecting step, the key issue was not “how do we collect data?” Rather, the key issue is “How
do we generate useful information?” You must learn to ask the right question(s), the supervisor
cautioned. “It is crucial to be able to state precisely what it is you want to know about the activity
you are going to measure. Without this knowledge, there is no basis for making measurements.
The department thought about this and after some more discussion felt they needed the answers to
two questions:
1. How do we know that we are providing the service that our customers require?
2. Where can we do better or improve?
All parties finally agreed upon two sets of critical activities that needed to be watched closely and
acted on if performance is less than the desired goal. The reason these were considered critical is
they are the sets of activities that produce our outputs (refer back to Figure B-2). Control point 1 is
when the driver returns to the warehouse after pick-up. Control point 2 is when a discrepancy on
the property report is resolved (Refer to Figure B-3).
Appendix B - 7
R
TRADE
APPENDIX B
Figure B-3
Step 2 - Identify Critical Activity to be Measured
Start
Receive call
from customer

Activity of property Property Process
Set 1
Arrive at site
Pick-up
specific property
Return to
Critical Activity (Control Point 1)
warehouse
Update property
Database
record

Review by
Activity customer
Set 2
Yes Concurs with

Agree? End
list
No
Notify warehouse
management
Critical Activity Resolve (Control Point 2)
End
Appendix B - 8
R
TRADE
APPENDIX B
Step 3: Establish Performance Goals or Standards

The supervisor was very pleased. “We are moving along quite nicely. Now we are ready to
establish a performance goal or standard,” she stated. She continued and said that for each control
point selected for measurement, it would be necessary to establish a performance goal or standard.
She again referred back to her workshop notes. A goal or standard is:
an “aimed-at” target; an achievement toward which effort is expended. Without a

goal or standard, there is no logical basis for making a decision or taking action.
Knowledge of performance is not enough; you must have a basis for comparison
before you can decide or act.
Because this is the first time the department has ever considered formalizing measurements, they
would need to establish some sort of baseline to set goals. The basis for the initial goals chosen were
the informal observations made by the department. The department planned to reevaluate the goals
in six months.
The department looked at Critical Activity 1, Return to Warehouse. They reviewed their objectives
and came up with three goals: (1) three-day turnaround; (2) scheduling pick-up per customer
request; and (3) 95% on time pick-ups. For Critical Activity 2, Resolve Discrepancies, they did the
same thing and came up with two goals: (1) 98% property list accuracy and (2) no more than 5% of
their time-resolving discrepancies. The department was satisfied that these performance goals would
produce the output and their corresponding objectives (Refer to Figure B-4). They were now ready
to move on to the next major activity.
Appendix B - 9
R
TRADE
APPENDIX B
Figure B-4
Step 3: Establish Performance Goal(s) or Standards
Start
Receive call
from customer

Set 1
Arrive at site
Pick-up
specific property
Return to Control Point 1:

warehouse Goal: Three day turnaround
Goal: 100% per request
Goal: 95% on-time
Update property
Database
record
Updated Property List Property

Review by
Activity customer
Set 2
Yes Concurs with

Agree? End
list
No
Notify warehouse
management
Critical Activity Resolve (Control Point 2) Control Point 2:

Goal: 98% accuracy
Goal: 5% resolution time
End
Appendix B - 10
R
TRADE
APPENDIX B
Step 4: Establish Performance Measurement(s)

Again, the supervisor was satisfied with their progress. Now they needed to identify specific
performance measures for the two critical activities they identified. The department decided to do
some brainstorming to generate potential performance measures. This step took a considerable
amount of time, and the team was clearly frustrated.
The supervisor reminded her department that good performance measures exist to aid in
understanding how well a process or activity is working or how well a product or service is produced
and delivered. “Remember,” she said, “what we measure should help us control and manage our
work.” She also reminded them that in addition to identifying performance measures, they must also
determine what raw data they will need to collect, find its location, determine what sensors will
measure or record the raw data, and decide how often the data will be collected.
The team felt somewhat overwhelmed by what seemed like a difficult task. The supervisor quickly
pointed out that for the first time, they would have measurable data that they could track to
determine how well they were doing and identify areas for improvement.
The team frequently found themselves asking, “what is it that we really want to know about what
we do.” Their supervisor reminded them that since they already had quantifiable goals, they could
use these to help determine their performance measures.
Critical Activity 1:
Performance Measure A
One of the goals of the department had always been to perform all property pick-ups in three days
or less. The team decided that measuring the number of days elapsed from call to pick-up for all
routine requests would be a useful performance measure that could be compared directly to their
goal. The raw data needed to construct their performance measure was simply the date of each call
for pick-up services and the actual date of pick-up. The data could be retrieved from the Property
Transfer form and collected weekly. If 50 regular pick-ups are performed in a week, the team would
have 50 measures to plot. The team decided to plot this measure in the form of a histogram, which
could be used to display the results of one week’s worth of data. In this way they could display
how many deliveries took one day, two days, three days, and so on.
Appendix B - 11
R
TRADE
APPENDIX B
Performance Measure B
The department had already determined that it was very important to make all specially scheduled
pick-ups on time. They had already chosen a goal of 100% on-time pick-up for specially scheduled
jobs, so they needed a performance measure that would track their results. The team reasoned that a
simple performance measure would be the percent of on-time special pick-ups. Since a percent
results from the ratio of two numbers, this measure can be shown more clearly as follows:
Number on-time special pick-ups X 100

Number scheduled special pick-ups
Expressing this performance measure as a ratio gives an indication of what data are needed to actually
construct it. In this case, the team would need the number of special pick-ups scheduled for each
week and the number performed on time. The Property Transfer form records the scheduled pick-up
date and the actual pick-up date.
Performance Measure C
Since the department had a goal that 95% of all pick-ups (without regard to type) would be
performed on time, they needed a performance measure to make a comparison. This was rather
straightforward, and the team settled on Percent On-Time Pick-Ups. As in the previous
performance measure, this measure is the result of a ratio and can be written as:
Number on-time pick-ups X 100

Total number of pick-ups
In order to make this calculation, the team had to determine what raw data were needed. To calculate
the total number of pick-ups, the team noted that they needed to only count the total number of
property transfer forms completed each week as this would tell them how many pick-ups were
completed. For this performance measure, the number of on-time pick-ups includes both the regular
and specially scheduled pick-ups that are performed each week. The teams would have to calculate
how many regular pick-ups were on time and how many specially scheduled pick-ups were on time.
Again, the property transfer forms record the type of transaction and all dates needed.
Appendix B - 12
R
TRADE
APPENDIX B
Critical Activity 2
Performance Measure A
Another product of the department is the monthly property list, which consists of line items that
specify property type, owner, and location. The team had already decided that this list should be at
least 98% accurate, and now they needed a performance measure to go assess their work. In this
case, the performance measure is the percent of accuracy of the monthly property list. Since this
measure results from a ration, it makes sense to write it out the way the calculation is actually
performed. Percent accuracy is the percent done correctly error-free and is written as:
Number of error-free line items X 100

Total number of line items
Written this way, it was obvious to the team what raw data they needed to make this calculation.
First, they needed the total number of line items or entries from all of the property lists that they
generate each month. This was available in the database and was easily extractable each month.
Second, they needed the number of error-free entries for each month. To determine this number,
they needed to check the complaint log to find the number of discrepancies that were their fault. The
total number of entries minus the discrepancies are the number of error-free line items. Discrepancies
that are due to customer error (such as misplacing their own property) do not count against the
department.
Performance Measure B
For their final performance measure, the team had already set a goal that they would not spend more
than 5% of their time resolving problems resulting from the monthly property list. For their
performance measure they chose percent time spent resolving property list problems (time spent by
the four property specialists). Again, as a ratio it would be written as:
Total hours spent on resolutions X 100

Total hours worked per month
The raw data are already spelled out in the numerator and denominator of the performance measure.
They consist of the total number of hours the four property specialists spend on problem resolution
and the total number of hours they work each month. The sensor to record this did not exist, so the
supervisor had the payroll department provide them with a special job number to add to their time
card to track time spent resolving property list problems.
The team was pleased with their results (Figure B-5) and was ready to move to the next step.
Appendix B - 13
R
TRADE
APPENDIX B
Step 4: Establish Performance Measurement(s) Summary
Performance Measure A: Number of days from call to pick-up
Raw Data: Date of call for pick-up services, actual pick-up date
Sensor: Property Transfer Form
Frequency: Weekly
Performance Measure B: % on-time special pick-ups
Number on-time special pick-ups X 100
Number scheduled special pick-ups
Raw Data: Number of special pick-ups scheduled each week,

number on time
Frequency: Compiled weekly
Performance Measure C: % on-time pick-ups (for all pick-ups)
Number on-time pick-ups X 100
Total number of pick-ups
Raw Data: Total number of pick-ups completed, total number on time

Frequency: Compiled weekly
Performance Measure A: % accuracy of monthly report
Number of error-free line items X 100
Total number of line items
Raw Data: Total number of line items entries generated each month on
property list, number of errors detected (used to calculate
number error-free)
Sensor: Property List Database, Complaint Log
Frequency: Compiled monthly
Appendix B - 14
R
TRADE
APPENDIX B
Performance Measure B: % time spent resolving property list problems

Total hours spent on resolution X 100
Total hours worked per month
Raw Data: Total number of hours that the four property specialists spend
on problem resolution each month, total hours worked by the
property specialists each month
Sensor: Time card with special job number to track problem resolution
time
Frequency: Compiled monthly
Appendix B - 15
R
TRADE
APPENDIX B
Figure B-5
Step 4 - Establish Performance Measurement(s)
Start
Receive call
from customer

Set 1
Arrive at site
Pick-up
Control Point 1:
specific property • Performance Measure A:
Number of days elapsed
from call to pick-up
Return to Goal: Three day turnaround
warehouse
• Performance Measure B:
Number of on-time special
Update property pick-ups as a percent of
Database number of scheduled
record
special pick-ups
Updated property list Property • Performance Measure C:
sent to customer list Number of on-time pick-
ups as a percent of total
number of pick-ups
Review by Goal: 95% on-time
Activity Customer
Set 2
Yes Concurs with

Agree? End
list
No
Notify warehouse Control Point 2:

management • Performance Measure A:
Number of error-free line
items as a percent of total
number of line items
Critical Activity Resolve (Control Point 2) Goal: 98% accuracy
Revise list Database Total hours spent on
resolution as a percent of
total hours worked per
month
End Goal: 5% resolution time
Appendix B - 16
R
TRADE
APPENDIX B
Step 5: Identify Responsible Party(ies)

The team was glad they had completed identifying the performance measurements. The next step
was a fairly easy one for the team members. They needed to identify responsible parties for
collecting the data, analyzing/reporting actual performance, comparing actual performance to
goal/standard, determining if corrective actions are necessary, and making changes.
For Critical Activity 1, a property specialist #1 will be responsible for collecting, interpreting, and
providing feedback on the data. The warehouse supervisor will be responsible for making decisions
and taking action. (Refer to Figure B-6.)
Obviously, many people could be involved in collecting data; however, someone needs to be
responsible for compiling the data and comparing actual performance with the department goal. If a
difference warrants, they need to notify the decision maker.
For Critical Activity 2, a similar argument was used. The group selected another property specialist
#2. However, the same supervisor will be the responsible decision maker. (Refer to Figure B-6.)
Appendix B - 17
R
TRADE
APPENDIX B
Figure B-6
Step 5 - Identify Responsible Party(ies)
Start
Receive call
from customer

Set 1
Arrive at site
Pick-up
Control Point 1:
from call to pick-up
Return to Goal: Three-day turnaround
warehouse
Responsible
Party(ies) Number of on-time special
Property Specialist # 1 pick-ups as a percent of
Property Supervisor Update property Database
record number of scheduled
special pick-ups
sent to customer list Number of on-time pick-
number of pick-ups
Review by Goal: 95% on time
Activity customer
Set 2
Yes Concurs with

Agree? End
list
No
Notify warehouse Control Point 2:
number of line items
Critical Activity Resolve (Control Point 2) Goal: 98% accuracy
Responsible
Party(ies) • Performance Measure B:
Property Specialist # 2 Revise list Database Total hours spent on
Property Supervisor resolution as a percent of
month
End Goal: 5% resolution time
Appendix B - 18
R
TRADE
APPENDIX B
Step 6: Collect Data

At this step, the supervisor reviewed her notes again. She remembered from her seminar that even
the best of measurement systems have failed because of poor data collection.
As the system owner, she needed to be the one with the overall responsibility for supervising the
data collection process. Each employee was of course responsible for the quality of his/her own
work, but she needed to be sure the data were being collected properly and that people were doing
their assignments.
Data collection was much more than simply writing things down and then analyzing everything after
a period of time. She resolved to conduct several preliminary analyses to determine if the
measurement system was functioning as designed, that the frequency of data collection was
appropriate, and to provide feedback to the data collectors with respect to any adjustments in the
system.
In process step 2, schedule pick-up of equipment, the team identified two control points. The first
control point covers flow process activity numbers 2, 4, and 5. The second control point covers
activity numbers 11 and 12. (See Figure B-7).
For the first control point, the use of an existing property transfer form, already in use for recording
the data, was determined to be the most efficient means for collecting the necessary information:
Activity 2: Schedule pick up of equipment

The date the customer placed the request. The scheduled date for the pick-up.
Activities 4 and 5: Pick-up specific property and return to warehouse
The date the property was actually picked up and delivered to the warehouse.
Because of the variety of the raw data comprising the performance measures, the data gathering
approach at the second control point was somewhat more complex. A complaint logbook is used to
record a description of the problem, its resolution, and related dates. Time charging records of the
problem solving personnel could also be reviewed. The required information was:
Activity 11: Notify warehouse management

A description of the problem and the date the division notified the warehouse (complaint
logbook).
Appendix B - 19
R
TRADE
APPENDIX B
Activity 12: Resolve

A description of what was done to resolve the issue and the date action was taken (complaint
logbook).
The time spent by a property specialist in resolving the specific issue versus the total time
spent on all work activities during the issue resolution period (time card records).
The total number of reports distributed during the measurement interval (property reports).
The team gathered data for a five-month period. During the first month, the supervisor had a
preliminary look at the data they were collecting on a weekly basis. She continued her spot checks
each month until the full collection period was completed. Ultimately, there were no significant
changes to the measurement system or collection frequency. The supervisor felt the team had done
an excellent job in understanding their process and designing their system. They were now ready to
begin Step 7, which involved analyzing the data.
Appendix B - 20
R
TRADE
APPENDIX B
Figure B-7
Step 6 - Collect Data
Start
Receive call
1 from customer

Activity 2 of property Property Process
Set 1
3 Arrive at site
4 Pick-up Control Point 1:

from ca ll to pick-up
Return to Goa l: Three day turnaround
Critical Activity 5 warehouse
(Control Point 1)
Responsible Party(s)
Property Specialist # 1 Number of on-time special
Property Supervisor pick-ups as a percent o f
Update property
6 Database number o f scheduled
record
special pick-ups

7 sent to customer list Number of on-time pick-
number o f pick-ups
Review by Goa l: 95% on-time
Activity 8 customer
Set 2
10
Yes Concurs with
9 Agree?
list
End
No
11 Notify warehouse Control Point 2:

12 Resolve number o f line items
Critical Activity (Control Point 2) Goa l: 98% accuracy
Responsible Party(s)
Property Specialist # 2 • Performance Measure B:
Property Supervisor 13 Revise list Database Total hours spent on
resolution as a percent of
month
End Goa l: 5% resolution time
Appendix B - 21
R
TRADE
APPENDIX B
Step 7: Analyze/Report Actual Performance

After five months, the supervisor felt they had a good baseline on which to start their analysis. Just
what do these data mean? The team was reminded of the issue of customer complaints that started
them thinking about performance measures. The supervisor asked them, “What were the questions,
identified in Step 2, that we felt needed to be answered?” The team responded:
1. How do we know that we are providing the service that our customer requires?
2. Where can we do better or improve?
In this step, we will explore some of the possible ways to analyze and to display the results of these
performance measures to clearly communicate the answer to their questions.
They started with Control Point 1. Performance Measure A was: number of days elapsed from call
to pick-up. Their goal was pick-up within three days or less from date of call. They had collected
the following data:
1. The time a call to pick-up was received

2. The day it was picked up
The forms the team used recorded a lot of information. They revealed how many pick-ups actually
took longer than three days and were late, and how early or late each pick-up was. One way to look
at the data is to use a bar chart to plot the number of late pick-ups each week (the number of on-time
pick-ups could also be plotted). This will show the progress each week, and after several weeks or
months, some trends may appear.
Figure B-8 shows a simple bar chart reflecting the number of late pick-ups each week. It shows that
in week one, there were five late pick-ups out of 51, with 13 during week two, and so on. The next
step would be to investigate what happened during week two and analyze the process to see what
can be done to meet the goal.
The same set of data can be used to view the process from a different perspective. A frequency
chart, as in Figure B-9, can be created to show the variation in the process. This shows the results of
the second week in January (week two in Figure B-8); of the 13 late pick-ups, for eight of them it
took four days and for five of them it took five days. The goal is three days or less, and it was seen
that it usually takes three days. However, the data also showed that ten times it took one day and
nine times it took two days. This type of chart shows how capable the process actually is.
Appendix B - 22
R
TRADE
APPENDIX B
Figure B-8
# Late Pick-Ups Each Week
With total normal pick-ups shown
16
(53)
14
12
# Late Pick-Ups
10
(48)
8
(51)
6
(40) (42)
4
0
1 2 3 4 5
Week Number
Figure B-9
Variation in Pick-Up Time
Week 2
20
Number of pick-ups
15
10
0
1 2 3 4 5
Number of days to pick-up
Appendix B - 23
R
TRADE
APPENDIX B
Step 7: Analyze/Report Actual Performance (continued)

The team continued with Critical Activity 1, Performance Measure B: Number of on-time special
pick-ups. The goal was meeting the customer’s specific need 100% of the time. We collected the
following data:
1. The date the customer requested pick-up

2. The actual date picked-up
This measure looks at how well those customers who want pick-up service provided on a particular
day are served. The data could be plotted as late pick-ups as in Figure B-8, or could be plotted as
the number of on-time pick-ups each week. Since this performance measure results in a ratio, a bar
chart can be used, as in Figure B-10, to measure the percent on-time pick-ups performed each week.
The bar chart can show the progress week-by-week.
If it was found that the group was consistently missing their goal, this should be investigated to find
out why and some type of process improvement should be performed. A frequency chart, such as
Figure B-9, may be useful in the study of the process.
Figure B-10
Percent On-Time Special Pick-Ups
When customer requested specific date
100
80
% On-Time
60
40
20
0
1 2 3 4 5
Week Number
Appendix B - 24
R
TRADE
APPENDIX B

The last performance measure for Critical Activity 1 is C: the number of on-time pick-ups with a
goal to have at least 95% of the pick-ups on time.
The team collected the following data:

1. The total number of pick-ups in a week.
2. The number of pick-ups performed in three days or less when the customer specified no date.
3. The number of on-time pick-ups when the customer specified a special date.
This performance measure looks at how well the entire property pick-up process is working. It
counts all on-time pick-ups against the total number of pick-ups each week. Like Performance
Measure B, this measure also results in a ratio when the data are put in. Likewise, a simple bar
chart, such as in Figure B-11, can be used to track these over a period of some five weeks. Progress
can be seen at a glance.
Figure B-11
Percent Total On-Time Pick-Ups
With total number of pick-ups shown
100
(57) (48) (49) (55)
(63)
80
% On-Time
60
40
20
0
1 2 3 4 5
Week Number
Appendix B - 25
R
TRADE
APPENDIX B

The team now began to look at Critical Activity 2, Performance Measure A: the percent of error-free
line items with a goal of 98% or more of all line item entries to be error-free.
The team collected the following data:
1. The total number of line items processed each month.

2. The total number of line item errors processed each month.
This performance measure tells how well the information on the transfer forms is recorded and
transferred to the data base (Activity 6.) It also measures customer satisfaction indirectly because
every error made causes customers to investigate the source of any discrepancy. A simple way to
display these data is again to use a simple bar chart, Figure B-12, that plots the resulting ratio month
by month. It is easy to see if the 98% error-free goal is being met.
The transfer forms generate the data that are used and, therefore, provide additional information for
inclusion on the actual volume of transfer forms processed each month as part of the graph. The
number could be written in each column and would reveal if the error rate is a function of the volume
of forms processed.
Figure B-12
Percent Error Free Line Items
100
90
% Error free
80
70
60
50
Jan Feb Mar Apr May
Month
Appendix B - 26
R
TRADE
APPENDIX B

Finally, the team had come to its last performance measure. Performance Measure B was the total
hours spent on resolutions with a goal that 5% or less of the team’s time be spent on property list
discrepancies.
The team had collected the following data:
1. The total number of hours worked by the property specialist each month taken from
the time cards.
2. The total number of hours the property specialists charge to problem resolutions each
month.
Performance Measure B gauges the cost of the transfer form and reports errors in terms of time spent
resolving problems. Like previous measures, this one also results in a fraction that can be plotted on
a bar chart. Figure B-13 shows the results of five months of data collection. It shows if a goal is
being met, the process being made, and if any trends are apparent.
Figure B-13
Percent Time on Problem Resolution
With resolution time vs. total time
10
9
8
(54/736)
7
6
% Time
(30/608)
5
(24/640)
4
3
(10/672) (15/704)
2
1
0
Jan Feb Mar Apr May
Month
Appendix B - 27
R
TRADE
APPENDIX B
Step 8: Compare Actual Performance to Goal/Standard

The supervisor was satisfied with the team’s efforts and results. They had learned a lot about
performance measures and now understood their importance and why they needed to measure. They
realized that if you cannot measure your process, you cannot control it. If you cannot control it,
you cannot manage it. Without dependable measurements, intelligent decisions cannot be made.
They were almost finished with the process. The team needed to compare their actual performance
to their goals.
Once a comparison against their goals was completed, the team had several alternatives:
• Forget it. Variance is not significant - economically or statistically.

• Fix it.
• Challenge the goal (or standard).
• Review performance measures. Are they answering our questions?
The supervisor told her team that “if corrective actions are not necessary, the team would continue
the data collection cycle.”
Step 9: Corrective Action Necessary?

Is corrective action necessary? The supervisor instructed her team that if the answer to this question
was yes, they would need to take the necessary action to bring their performance back into line with
their goal(s)-the final step in closing the feedback loop.
She further stated that the key objectives of correction are:
1. Removal of defects, which are, in many cases, worker-controllable.

2. Removal of defect causes whether worker or management-controllable, dependent up on the
defect cause.
3. Attainment of a new state of process that will prevent defects from happening.
4. Maintenance or enhancement of the efficiency and effectiveness of the process, an essential
condition for continuing process optimization and ultimately increasing the competitiveness
and profitability of the business itself. The removal of defects and defect causes at the
expense of productivity or efficiency is inherently self-defeating.
Appendix B - 28
R
TRADE
APPENDIX B
After reviewing the Performance Measure Report, the team did not feel that Performance
Measurement A was answering their questions. Performance Measurement C more clearly achieved
their objective; therefore, they decided to delete Performance Measurement A from further data
collecting and consideration.
The team also decided to tackle the problem of late pick-ups. They had a goal of 95% on-time pick-
ups. They never met their goal. Why? After further discussions, the supervisor and team members
felt they would form a quality improvement team to look at this problem and identify the
solution(s). They can use a lot of the data they have already collected to assist them in finding the
root cause of the problem.
Additionally, they would continue to gather data for another five to six months, review it, and
determine if any further action would be necessary.
Conclusion
The Quality Improvement Team has been established and currently is working the problem trying to
determine the root cause. Therefore, the team would come back to address steps 10 and 11 after
they complete their investigation.
The team members told their supervisor they finally realized the value and importance of doing
performance measures on their processes.
The supervisor asked her team to summarize for her why they should measure. The team wrote
down the following:
Performance measures can be used for:
1. Control: to help reduce variation.

2. Self-Assessment: to assess how well our process is doing, including any improvements we
may have made.
3. Continuous Improvement: to identify defect sources, process trends, defect prevention,
and to determine process efficiency and effectiveness, and opportunities for improvement.
The supervisor was very pleased.
Appendix B - 29
R
TRADE
APPENDIX B
Appendix B - 30
R
TRADE
Appendix C
Sample Performance Measures

APPENDIX C
SAMPLE PERFORMANCE MEASURES
Sample Performance Measures

This, which is a collection from several sources, is included to stimulate your thinking about
appropriate measures. However, remember that the measures you select should fit the process,
products, and goals.
A. Accounting Performance Measurements
Percent of late reports Percent of errors in reports

Errors in input to Information Services Errors reported by outside auditors
Percent of input errors detected Number of complaints by users
Number of hours per week correcting or Number of complaints about inefficiencies or
changing documents excessive paper
Amount of time spent appraising/correcting Payroll processing time
input errors
Percent of errors in payroll Length of time to prepare and send a bill
Length of time billed and not received Number of final accounting jobs rerun
Number of equipment sales miscoded Amount of intra-Company accounting bill-back
activity
Time spent correcting erroneous inputs Number of open items
Percent of deviations from cash plan Percent discrepancy in MRB and line scrap
reports
Travel expense accounts processed in three days Percent of advances outstanding
Percent data entry errors in accounts payable Credit turnaround time
and general ledger
Machine billing turnaround time Percent of shipments requiring more than one
attempt to invoice
Number of untimely supplier invoices processed Average number of days from receipt to
processing
B. Clerical Performance Measurements
Misfiles per week Paper waste

Errors per type page Administration errors (not using the right
procedure)
Number of times messages are not delivered Percent of action items not done on schedule
Percent of inputs not received on schedule Percent of coding errors on time cards
Period reports not completed on schedule Percent of phone calls answered within two
rings
Percent of phone calls dialed correctly Pages processed error-free per hour
Clerical personnel/personnel support Percent of pages retyped
Percent of impressions reprinted
Appendix C - 1
R
TRADE
APPENDIX C
C. Product/Development Engineering Performance Measurements
Percent of drafting errors per print Percent of prints released on schedule

Percent of errors in cost estimates Number of times a print is changed
Number of off-specifications approved Simulation accuracy
Accuracy of advance materials list How well the product meets customer
expectations
Field performance of product Percent of error-free designs
Percent of errors found during design review Percent of repeat problems corrected
Time to correct a problem Time required to make an engineering change
Percent of reports with errors in them Data recording errors per month
Percent of evaluations that meet engineering Percent of special quotations that are successful
objectives.
Percent of test plans that are changed Number of meetings held per quarter where
(change/test plan) quality and defect prevention were the main
subject
Person-months per released print Percent of total problems found by diagnostics
as released
Number of problems that were also encountered Cycle time to correct customer problem
in previous products
Number of errors in publications reported from Number of products that pass independent
the plan and field evaluation error-free
Number of misused shipments of prototypes Number of unsuccessful pre-analyses
Number of off-specifications accepted Percent of requests for engineering action open
for more than two weeks
Number of days late to pre-analysis Number of restarts of evaluations and tests
Effectiveness of regression tests Number of days for the release cycle
Percent of corrective action schedules missed Percent of bills of material that are released in
error
Cost of input errors to the computer Cost of engineering changes per month
Spare parts cost after warranty Customer cost per life of output delivered
D. Finance Performance Measurements
Percent error in budget predictions Computer rerun time due to input errors
Percent of financial reports delivered on Number of record errors per employee
schedule
Percent of error-free vouchers Percent of bills paid so Company gets price
break
Percent of errors in checks Entry errors per week
Number of payroll errors per month Number of errors found by outside auditors
Number of errors in financial reports Percent of errors in travel advance records
Percent of errors in expense accounts detected Computer program change cost
auditors
Appendix C - 2
R
TRADE
APPENDIX C
E. Industrial/Plant Engineering Performance Measurements
Percent of facilities on schedule Percent of manufacturing time lost due to bad

layouts
Percent of error in time estimates Percent of error in purchase requests
Hours lost due to equipment downtime Scrap and rework due to calibration errors
Repeat call hours for the same problem Changes to layout
Percent deviation from budget Percent variation to cost estimates
Number of unscheduled maintenance calls Percent of equipment maintained on schedule
Number of hours used on scheduled maintenance Accuracy of assets report
Percent of equipment overdue for calibration Number of industrial design completions past
due
Percent of total floor space devoted to storage Number of errors found after construction had
been accepted by the company
Number of mechanical/functional errors in Maintenance cost/equipment cost
industrial design artwork
Percent of engineering action requests accepted
F. Forecasting Performance Measurements
Number of upward pricing revisions per year Number of project plans that meet schedule,
price, and quality
Percent error in sales forecasts Number of forecasting assumption errors
Number of changes in product schedules
G. Information Systems Performance Measurements
Keypunch errors per day Input correction on data entry

Reruns caused by operator error Percent of reports delivered on schedule
Errors per thousand lines of code Number of changes after the program is coded
Percent of time required to debug programs Number of cost estimates revised
Percent error in forecast Percent error in lines of code required
Number of coding errors found during formal Number of test case errors
testing
Number of test case runs before success Number of revisions to plan
Number of documentation errors Number of revisions to program objectives
Number of errors found after formal test Number of error-free programs delivered to
customer
Number of process step errors before a correct Number of revisions to checkpoint plan
package is ready
Number of changes to customer requirements Percent of programs not flow-diagrammed
Percent of customer problems not corrected per Percent of problems uncovered before design
schedule release
Percent change in customer satisfaction survey Percent of defect-free artwork
System availability Terminal response time
Mean time between system IPL’s Mean time between system repairs
Time before help calls are answered Rework costs resulting from computer program
Appendix C - 3
R
TRADE
APPENDIX C
H. Legal Performance Measurements
Response time on request for legal opinion Time to prepare patent claims
Percent of cases lost
I. Management Performance Measurements
Security violations per year Percent variation from budget

Percent of target dates missed Percent personnel turnover rate
Percent increase in output per employee Percent absenteeism
Percent error in planning estimates Percent of output delivered on schedule
Percent of employees promoted to better jobs Department morale index
Percent of meetings that start on schedule Percent of employee time spent on first-time
output
Number of job improvement ideas per employee Ratio of direct to indirect employees
Increased percent of market Return of investment
Percent of appraisals done on schedule Percent of changes to project equipment required
Normal appraisal distribution Percent of employee output that is measured
Number of grievances per month Number of open doors per month
Percent of professional employees active in Percent of managers active in community
professional societies activities
Number of security violations per month Percent of time program plans are met
Percent of documents that require two Percent of employees who can detect and repair
management their own errors
Percent of delinquent suggestions Improvement in opinion surveys
Number of decisions made by higher-level Percent of time cards that have errors on them
management than required by procedures signed by managers
Percent of employees taking higher education Number of damaged equipment and property
reports
Number of employees dropping out of classes Percent error in personnel records
Improvement in customer satisfaction survey Volume actual versus planned
Revenue actual versus plan Percent of procedures less than 10 pages
Number of procedures with fewer than three Number of formal reviews before plans are
acronyms and abbreviations approved
Percent of employees active in improvement Number of hours per year of career and skill
teams development training per employee
Number of user complaints per month Number of variances in capital spending
Percent revenue/expense ratio below plan Percent of executive interviews with employees
Percent of departments with disaster recovery Percent of appraisals with quality as a line item
plans that makes up more than 30 percent of the
evaluation
Percent of employees with development plans Direct/indirect ratio
Revenue generated over strategic period Number of iterations of strategic plan
Appendix C - 4
R
TRADE
APPENDIX C
I. Management Performance Measurements (continued)
Number of employees participating in cost Dollars saved per employee due to new ideas
effectiveness and/or methods
Result of peer reviews Number of tasks for which actual time exceeded
estimated time
Data integrity Warranty costs
Cost of poor quality
J. Manufacturing and Test Engineering Performance Measurements
Percent of process operations where sigma limit Percent of tools that fail certification
is within engineering specification
Percent of tools that are networked due to Number of process changes per operation due to
design errors error
Percent error in manufacturing costs Time required to solve a problem
Number of delays because process instructions Percent error in test equipment and tooling budget
are wrong or not available
Number of errors in operator training Percent of errors that escape the operator’s
documentation detection
Percent of testers that fail certification Percent error in yield projections
Percent error in output product quality Percent of designed experiments that need to be
revised
Percent of changes to process specifications Percent of equipment ready for production on
during process design review schedule
Percent of meetings starting on schedule Percent of drafting errors found by checkers
Percent error in yield projections Percent of manufacturing used to screen products
Number of problems that the test equipment Percent correlation between testers
cannot detect during manufacturing cycle
Number of waivers to manufacturing procedures Percent of tools and test equipment delivered on
schedule
Percent of tools and test equipment on change Percent functional test coverage of products
level control
Percent projected cost reductions missed Percent of action plan schedules missed
Equipment utilization In-process yields
Labor utilization index Asset utilization
Appendix C - 5
R
TRADE
APPENDIX C
K. Manufacturing/Shipping Performance Measurements
Complaints on shipping damage Percent of parts not packed to required

specifications
Percent of output that meets customers orders Suggestions per employee
and engineering specifications
Percent of jobs that meet cost Percent of jobs that meet schedule
Percent of product defect-free at measurement Percent of employees trained to do the job they
operations are working on
Accidents per month Performance against standards
Percent of utilities left improperly running at Percent unplanned overtime
end of shift
Number of security violations per month Percent of time log book filled out correctly
Time and/or claiming errors per week Time between errors at each operation
Labor utilization index Percent of operators certified to do their job
Percent of shipping errors Defects during warranty period
Replacement parts defect rates Percent of products defective at final test
Percent of control charts maintained correctly Percent of invalid test data
Percent of shipments below plan Percent of daily reports in by 7 a.m.
Percent of late shipments Percent of products error-free at final test
Scrap and rework cost
L. Personnel Performance Measurements
Percent of employees who leave during the first Number of days to answer suggestions
year
Number of suggestions resubmitted and Turnover rate due to poor performance
approved
Number of grievances per month Percent of employment requests filled on
schedule
Number of days to fill an employment request Time to process an applicant
Average time a visitor spends in lobby Time to get security clearance
Time to process insurance claims Percent of employees participating in
company-sponsored activities
Percent of complaints about salary Percent of personnel problems handled by
employees’ managers
Percent of employees participating in voluntary Percent of offers accepted
health screening
Percent of retirees contacted yearly by phone Percent of training classes evaluated excellent
Percent deviation to resource plan Wait time in medical department
Number of days to respond to applicant Percent of promotions and management
changes publicized
Percent of error-free newsletters Personnel cost per employee
Cost per new employee Management evaluation of management
education courses
Opinion survey ratings
Appendix C - 6
R
TRADE
APPENDIX C
M. Procurement/Purchasing Performance Measurements
Percent of discount orders by consolidating Errors per purchase order

Number of orders received with no purchase Routing and trace errors per shipment
order
Percent of supplies delivered on schedule Percent decrease in parts cost
Expediters per direct employees Number of items on the hot list
Percent of suppliers with 100 percent lot Labor hours per $10,000 purchases
acceptance for one year
Purchase order cycle time Number of times per year line is stopped due to
lack of supplier parts
Percent of parts with two or more suppliers Average time to fill emergency orders
Average time to replace rejected lots with good Percent of lots received on line late
parts
Time to answer customer complaints Percent of phone calls dialed correctly
Percent of purchase orders returned due to Percent of defect-free supplier model parts
errors or incomplete description
Percent projected cost reductions missed Time required to process equipment purchase
orders
Number of items billed but not received Stock costs
Supplier parts scrapped due to engineering Parts costs per total costs.
changes
Actual purchased materials cost per budgeted Cost of rush implants
cost
N. Production Control Performance Measurements
Percent of late deliveries Percent of errors in stocking

Number of items exceeding shelf life Percent of manufacturing jobs completed on
schedule
Time required to incorporate engineering Percent of errors in purchase requisitions
changes
Percent of products that meet customer orders Inventory turnover rate
Time that line is down due to assembly shortage Percent of time parts are not in stock when
ordered from common parts crib
Time of product in shipment Spare parts availability in crib
Percent of stock errors Percent of errors in work in process records
versus audit data
Number of bill of lading errors not caught in Cost of rush shipments
shipping
Cost of inventory spoilage
Appendix C - 7
R
TRADE
APPENDIX C
O. Quality Assurance Performance Measurements
Percent error in reliability projections Percent of product that meets customer

expectations
Time to answer customer complaints Number of customer complaints
Number of errors detected during design and Percent of employees active in professional
process reviews societies
Number of audits performed on schedule Percent of QA personnel to total personnel
Percent of quality inspectors to manufacturing Percent of QE’s to product and manufacturing
directs engineers
Number of engineering changes after design Number of process changes after process
review qualification
Errors in reports Time to correct a problem
Percent of suppliers at 100 percent lot Percent of lots going directly to stock
acceptance for one year
Percent of problems identified in the field Variations between inspectors doing the same
job
Percent of reports published on schedule Number of complaints from manufacturing
management
Percent of field returns correctly analyzed Time to identify and solve problems
Percent of lab services not completed on Percent of improvement in early detection of
schedule major design errors
Percent of errors in defect records Number of reject orders not dispositioned in
five days
Number of customer calls to report errors Number of committed supplier plans in place
Percent of correlated test results with suppliersReceiving inspection cycle time
Number of requests for corrective action being Time required to process a request for
processed corrective action
Number of off-specifications approved Percent of part numbers going directly to stock
Number of manufacturing interruptions caused Percent error in predicting customer
by supplier parts performance
Percent product cost related to appraisal scrap Percent skip lot inspection
and rework
Percent of qualified suppliers Number of problems identified in-process
Cost of scrap and rework that was not created at Level of customer surveys
the rejected operation
P. Security/Safety Performance Measurements
Percent of clearance errors Time to get clearance

Percent of security violations Percent of documents classified incorrectly
Security violations per audit Percent of audits conducted on schedule
Percent of safety equipment checked per Number of safety problems identified by
schedule management versus total safety problems
identified
Safety accidents per 1000,000 hours worked Safety violations by department
Number of safety suggestions Percent of sensitive parts located
Appendix C - 8
R
TRADE
Appendix D
Related References
APPENDIX D
RELATED REFERENCES
Related References
There have been many articles published and materials compiled across the Department of Energy
complex and industry that contain additional information in the area of measuring performance.
These references may provide additional information and guidance to assist you with your programs.
References
1. DP PI Executive Summary Report, 4th Qtr., CY 1993.
2. A Performance Improvement Measurement Methodology, Morgantown Energy Technology Center,
May 15, 1994.
3. Guidelines for Process Improvement, Morgantown Energy Technology Center, October 1994.
4. Performance Indicators for 2nd Quarter CY-1994, DOE/ORO.
5. Program Logic Models: A Step to Performance Measurement, SNL Contractor in Support of EE,
August 20, 1993.
6. The Use of Performance Indicators in the Public Sector, Peter Smith, University of York, UK.
7. Performance Based Management Lessons Learned, University of California, TRADE PI SIG Spring
Workshop Presentation, June 7, 1994.
8. A Baldridge-Based Environmental Management Self-Assessment Matrix and Implementation Guide,
Council of Great Lakes Industries, TRADE PI SIG Conference, June 6, 1994.
9. Performance Measurement Process Guidance Document, DOE/NV Family Quality Forum, June
1994.
10. DP Occurrence Analysis Report, 1st Quarter Calendar Year 1994, May 1994.
11. A Collection of Trending and Analysis Product Examples, TRADE OR SIG Workgroup, February
1993.
12. DOE/ES&H Occupational Safety and Health Observer newsletters, September 1994, fall 1994.
13. DOE/Nuclear Safety Operating Experience Weekly Summary newsletters, Summary 94-38,94-41
through 44.
14. Fort Calhoun Station Performance Indicators, Omaha Public Power District, April 1991, Docket
No. 50-285.
15. Fort Calhoun Station Performance Indicators, Omaha Public Power District, August 1994, Docket
No. 50-285.
16. Performance Indicators for Operating Commercial Nuclear Power Reactors, 1st Qtr. 1992, Nuclear
Regulatory Commission.
17. Historical Data Summary of the Systematic Assessment of Licensee Performance, NRC/Office of
Nuclear Reactor Regulation, February 1992, NUREG-1214, Rev 9.
18. Guidelines for Analyzing and Trending Incidents in Nuclear Power Plants, Nuclear Safety Analysis
Center/EPRI, June 1988, NSAC-119.
19. An Overview of Nuclear Power Plant Operations Experience Feedback Programs, NRC/Office for
Analysis and Evaluation of Operational Data, May 9, 1986, Report No. AEOD-5602.
Appendix D - 1
R
TRADE
APPENDIX D
RELATED REFERENCES
20. Development of Programmatic Performance Indicators, Pacific Northwest Laboratory, September

1988, NUREG/CR-5241.
21. RiverPulse, A Report on the Condition of the Tennessee River and Its Tributaries in 1993, TVA.
22. 1992 Performance Indicators for the U.S. Nuclear Utility Industry, INPO.
23. 1992 Annual Report Power Reactors, Office for Analysis and Evaluation of Operational Data, NRC,
NUREG-1272, Vol 7, No. 1, July 1993.
24. Handbook for Nuclear Power Plant Self Assessment Programs, Nuclear Safety Analysis Center,
EPRI, NSAC-170.
25. Performance Objectives and Criteri.a for Corporate Evaluations, INPO, August 1985, IPNO 85-029.
26. Measures for Two CCPS PSM Elements: Management of Change/Training & Performance, Prepared
for Center for Chemical Process Safety by Concord Associates and JBF Associates, June 1993,
CA/TR 93-016-01 (Proprietary.)
27. Industrial Hygiene Performance Indicators, presentation by David Weitzman (ESH) to DOE Annual
S & H Conference, October 27, 1994.
28. Occupational Injury and Property Damage Summary January-December 1993, EG&G Idaho Falls
for DOE Office for Safety and Quality Assurance, DOE/EH/1570-H5.
29. Industrial Hygiene Database Management System, DOE Office of Health Flyer at DOE Annual S &
H Conference, October 1994.
30. Actively Caring for a Total Safety Culture, INEL Presentation to DOE Safety & Health Annual
Conference, October 1994.
31. Misc. Information on TIS (Technical Information Services) obtained at October 1994 DOE Safety &
Health Annual Conference.
32. DP August 1994 Summary Report of the ORPS.
33. DOE Order 5000.3B Trending and Analysis Report as a Management Tool, presentation at Oct.
1994 TRADE Conference, Lynn Maestas, ALO Office.
34. DOE Order 5000.3B Trending and Analysis Report, 3rd Quarter CY94, DOE AL, November 1994.
35. Fall 1994 TIS FLASH, TIS newsletter.
36. Proposed New DOE Technical Standard, Project Number DOE-STD-TMP-23-94, Draft DOE Lessons
Learned Standard, November 1994.
37. Deliverable of the Product from the Policy Office Task Force for Performance Management,
Performance Management for the U.S. Department of Energy, August 31, 1994.
38. ES&H TIS Internet Resource Directory.
39. Monitoring and Analysis of Chemical Safety Occurrences, John Usher, BNL and Rama Sastry, DOE
Germantown, DOE Risk Management Quarterly Newsletter, October 1994.
40. Performance Measurement Made Easy, Energy Evaluation Seminar, Ronald Corbeil of Canada, Sept.
15, 1993.
41. Process Improvement Guide and Process Measurement Guide, U.S. Coast Guard, Feb. 1993.
42. E,S,&H Performance Analysis Upgrade, G. J. Briscoe, EG&G/INEL, May 1987.
43. Westinghouse Hanford Co. Operational Excellence Performance Indicators, 2nd Qtr. CY 1992.
Appendix D - 2
R
TRADE
APPENDIX D
RELATED REFERENCES
44. Fast Flux Test Facility Performance Monitoring Management Information, D. J. Newland,
Westinghouse Hanford Co., Oct. 1987.
45. Westinghouse Savannah River Co. Performance Indicators, May 1990, WSRC-RP-90-750-5.
46. Westinghouse Savannah River Co. Training Performance Indicators, June 1993, WSRC-RP-93-939-6.
47. Draft LLNL E,S,&H Performance Objectives, June 1994.
48. Kansas City Plant Trend Data, June 1990.
49. NRC Performance Indicator Program for Operating Nuclear Power Plants, Nuclear Legal &
Licensing Symposium, Edward L. Jordan, Director Office for Analysis and Evaluation of Operational
Data, June 22, 1987.
50. Performance Indicator Summary, Peach Bottom Atomic Power Station, 2nd Qtr. 1990.
51. Nuclear Power Performance Report, TVA, July 1992.
52. Site Performance Report, Browns Ferry Nuclear Plant, TVA, July 1992.
53. Business Plan Indicators, VA Power, August 1990.
54. VA Power Nuclear Performance, August 1990.
55. Angra I Risk Based Safety Indicators Program, presentation by Furnas Centrais Eletricas S.A. to
IAEA Technical Committee Meeting on Safety Indicators, November 1990.
56. Experiences from the Introduction of the WANO Performance Indicators in Sweden, presentation by
Stockholm Nuclear Training and Safety Center to IAEA Technical Committee Meeting on Safety
Indicators, November 1990.
57. NPP Safety Indicators Evaluated by the Plant Management Tool “Safety Analysis and Information
System,” presentation by TUV-Norddeutschland e.V. to IAEA Technical Committee Meeting on
Safety Indicators, November 1990.
58. Application of a New Set of Maintenance Related and System Based Performance Indicators to
Selected Past Operational Data of the Koeberg Nuclear Power Plant, Kirchsteiger and Woodcock,
ESKOM, Johannesburg, South Africa.
59. Performances des Centrales Nucleaires Rep 900 ET 1 300 MW, Electricite de France, Juillet 1990.
60. Operation Safety Performance Indicators of NPP USSR.
61. Environmental Management Worker Safety and Health Performance Indexing System, White Paper,
DOE Environmental Management Office of Safety and Health, January 1995.
62. Trends ‘93, A Compendium of Data on Global Change, Center for Global Environmental Studies,
ORNL.
63. ORNL Review, Vol. 26, Nos. 3&4, 1993.
64. Internal Correspondence on Performance Measures, A.E. Ekkebus, Jan 6, 1995.
65. WANO Performance Indicator Program, The Nuclear Professional, fall 1994.
66. 1994 Strategic Plan, Office of Environment, Safety, and Health, April 1994.
67. A Proposed Safety and Health Performance Measurement Process for the Department of Energy,
Martin Marietta Energy Systems, Inc., Gordon G. Fee letter to Joe LaGrone presenting results of
Sparks/Tupper study, Feb. 16, 1994.
68. DOE Order 5000.3B, Occurrence Reporting and Processing of Operations Information, Feb. 22,
1993.
Appendix D - 3
R
TRADE
APPENDIX D
RELATED REFERENCES
69. Strategic Plan, United States Department of Energy, April 1994.

70. Performance Trending Principles, James A. Christensen, Battelle Northwest Laboratories, 16th
Annual National Energy Division Conference.
71. Development of Qualitative Risk Assessment Criteria, Jeffery A. Mahan, Risk Management Quarterly,
January 1995.
72. Spent Fuel Working Group Report, Volume I, November 1993, U.S. DOE.
73. Safety Performance Indexing, Corporate Safety, Eastman Kodak Company, 1994.
74. Federal Express Quality Profile, TRADE PI-SIG Spring Workshop Handout, June 1994.
75. Motorola Presentation, TRADE PI-SIG Spring Workshop Handout, June 1994.
76. Managing and Measuring Performance in the Department of Energy, speaker overheads from
TRADE PI-SIG meetings, Oct. 1994 Annual TRADE Conference.
77. Using Performance Indicators to Demonstrate Compliance with DOE Orders, speaker overheads
from TRADE PI-SIG presentation, Nov. 1994 Annual TRADE Conference, Bob Warther, DNFSB.
78. Understanding Statistical Process Control, Donald J. Wheeler, SPC Press.
79. Advanced Topics in Statistical Process Control, Donald J. Wheeler, SPC Press.
80. Statistical Quality Control, Western Electric Co. Inc., Call AT&T at 1-800-432-6600.
81. Statistical Quality Control, Eugene L. Grant & Richard E. Leavenworth, McGraw-Hill Book Co.
82. Quality Control & Industrial Statistics, Acheson J. Duncan, Richard D. Irwin Inc.
Appendix D - 4
R
TRADE

Handbook All How To Measure Performance

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Handbook All How To Measure Performance

Uploaded by

Copyright:

Available Formats

How to Measure Performance

A Handbook of Techniques and Tools

Section 1: Development Processes

1.1 Performance Measurement Process .......................................................................... 1-3

1.2 Developing Performance Measures–A Systematic Approach................................ 1-35

1.3 Developing Performance Metrics–University of California Approach.................. 1-53

1.4 Performance Indexes................................................................................................ 1-59

Section 2: Data Tools

2.0 Overview................................................................................................................... 2-1

2.1 Data Collection Techniques ...................................................................................... 2-7

2.2 Charts, Graphs, and Diagrams ................................................................................ 2-13

2.3 Presentation Approaches........................................................................................ 2-31

Appendix A: Glossary of Terms........................................................................................... A-1

Appendix B: Performance Measurement Process Case Study...............................................B-1

Appendix C: Sample Performance Measures.........................................................................C-1

Appendix D: Related References........................................................................................... D-1

• Joel Anderson, Los Alamos National Laboratory

This handbook offers three such disciplined, systematic approaches.

1.1 Performance Measurement Process

What Are Performance Measures?

• how well we are doing

The following reflect the attributes of an ideal unit of measure:

• Reflects the customer’s needs as well as our own

Technological units of measure Numerous technological instruments to

Units of Measure Sensors

What Are the Benefits of Measurements?

Why Do We Need to Measure?

What Is the Foundation for a Performance Measurement System?

1. The Sensor evaluates actual performance.

Control Activity Sensor Goal/Standard

A brief description of each of the process steps follows:

3. Establish performance goal(s) or standards. All performance measures should be tied to a

6 Collect data Database 6

YES Make changes to

Step 1: Identify Process

A. What product or service do we produce?

OUTPUT: A LIST OF PROCESSES, KEY PROCESSES, AND

Step 2: Identify Critical Activity(ies) to be Measured

• What question do we need to answer?

• Where in the process can we get these data?

OUTPUT: A LIST OF THE CRITICAL ACTIVITY AREAS FOR

• Formulate precisely the question we are trying to answer.

Step 3: Establish Performance Goal(s) or Standard(s)

Good performance goals or standards are:

OUTPUT:A LIST OF GOALS FOR EACH CRITICAL

Step 4: Establish Performance Measurement(s)

• translates “what do I want to know” into a performance measure

Translate into Performance Measures

% on-time deliveries per month

number of on-time deliveries per month X 100%

Identify the Raw Data

Locate the Raw Data

Identify the Sensor

1. What am I trying to measure; what kind of data are they?

Determine How Often to Make Measurements

OUTPUT: THE PERFORMANCE MEASURE AND ITS

Step 5: Identify Responsible Party(ies)

• Collect the data

1. Know what the goals are

OUTPUT:A LIST OF PEOPLE AND THEIR AREAS OF

Step 6: Collect Data

◊ Cost of overnight mail

◊ Number of hours to process an engineering change request