07/02/2018 Design of Experiments (DOE) Tutorial

> Knowledge Center > Toolbox > Design of Experiments (DOE)

Design of Experiments (DOE)

Outline
1. Introduction
2. Preparation
3. Components of Experimental Design
4. Purpose of Experimentation
5. Design Guidelines
6. Design Process
7. One Factor Experiments
8. Multi-factor Experiments
9. Taguchi Methods

1. Introduction
The term experiment is defined as the systematic procedure carried out under controlled conditions
in order to discover an unknown e ect, to test or establish a hypothesis, or to illustrate a known
e ect. When analyzing a process, experiments are often used to evaluate which process inputs
have a significant impact on the process output, and what the target level of those inputs should
be to achieve a desired result (output). Experiments canAbout Us
be designed Online Training
in many di erentSoftware
ways to
collect this information. Design of Experiments (DOE) is also referred to as Designed Experiments
or Experimental Design - all of the terms have the same meaning.
Simulations Blended Learning Resources

Experimental design can be used at the point of greatest leverage to reduce design costs by
speeding up the design process, reducing late engineering Login
design changes, and reducing product
Contact
material and labor complexity. Designed Experiments are also powerful tools to achieve
manufacturing cost savings by minimizing process variation and reducing rework, scrap, and the
need for inspection.
This Toolbox module includes a general overview of Experimental Design and links and other
resources to assist you in conducting designed experiments. A glossary of terms is also available at
any time through the Help function, and we recommend that you read through it to familiarize
yourself with any unfamiliar terms.

2. Preparation
If you do not have a general knowledge of statistics, review the Histogram, Statistical Process
Control, and Regression and Correlation Analysis modules of the Toolbox prior to working with this
module.

https://www.moresteam.com/toolbox/design-of-experiments.cfm 1/11
07/02/2018 Design of Experiments (DOE) Tutorial

You can use the MoreSteam's data analysis software EngineRoom® for Excel to create and analyze
many commonly used but powerful experimental designs. Free trials of several other statistical
packages can also be downloaded through the MoreSteam.com Statistical Software module of the
Toolbox. In addition, the book DOE Simplified, by Anderson and Whitcomb, comes with a sample of
excellent DOE software that will work for 180 days after installation.

3. Components of Experimental Design

Consider the following diagram of a cake-baking process (Figure 1). There are three aspects of the
process that are analyzed by a designed experiment:
Factors, or inputs to the process. Factors can be classified as either controllable or uncontrollable
variables. In this case, the controllable factors are the ingredients for the cake and the oven that
the cake is baked in. The controllable variables will be referred to throughout the material as
factors. Note that the ingredients list was shortened for this example - there could be many
other ingredients that have a significant bearing on the end result (oil, water, flavoring, etc).
Likewise, there could be other types of factors, such as the mixing method or tools, the
sequence of mixing, or even the people involved. People are generally considered a Noise Factor
(see the glossary) - an uncontrollable factor that causes variability under normal operating
conditions, but we can control it during the experiment using blocking and randomization.
Potential factors can be categorized using the Fishbone Chart (Cause & E ect Diagram) available
from the Toolbox.
Levels, or settings of each factor in the study. Examples include the oven temperature setting
and the particular amounts of sugar, flour, and eggs chosen for evaluation.
Response, or output of the experiment. In the case of cake baking, the taste, consistency, and
appearance of the cake are measurable outcomes potentially influenced by the factors and their
respective levels. Experimenters often desire to avoid optimizing the process for one response at
the expense of another. For this reason, important outcomes are measured and analyzed to
determine the factors and their settings that will provide the best overall outcome for the
critical-to-quality characteristics - both measurable variables and assessable attributes.

Figure 1

https://www.moresteam.com/toolbox/design-of-experiments.cfm 2/11
07/02/2018 Design of Experiments (DOE) Tutorial

4. Purpose of Experimentation
Designed experiments have many potential uses in improving processes and products, including:
Comparing Alternatives. In the case of our cake-baking example, we might want to compare
the results from two di erent types of flour. If it turned out that the flour from di erent vendors
was not significant, we could select the lowest-cost vendor. If flour were significant, then we
would select the best flour. The experiment(s) should allow us to make an informed decision
that evaluates both quality and cost.
Identifying the Significant Inputs (Factors) A ecting an Output (Response) - separating the
vital few from the trivial many. We might ask a question: "What are the significant factors
beyond flour, eggs, sugar and baking?"
Achieving an Optimal Process Output (Response). "What are the necessary factors, and what
are the levels of those factors, to achieve the exact taste and consistency of Mom's chocolate
cake?
Reducing Variability. "Can the recipe be changed so it is more likely to always come out the
same?"
Minimizing, Maximizing, or Targeting an Output (Response). "How can the cake be made as
moist as possible without disintegrating?"
Improving process or product "Robustness" - fitness for use under varying conditions. "Can the
factors and their levels (recipe) be modified so the cake will come out nearly the same no matter
what type of oven is used?"
Balancing Tradeo s when there are multiple Critical to Quality Characteristics (CTQC's) that
require optimization. "How do you produce the best tasting cake with the simplest recipe (least
number of ingredients) and shortest baking time?"

5. Experiment Design Guidelines

The Design of an experiment addresses the questions outlined above by stipulating the following:
The factors to be tested.
The levels of those factors.
https://www.moresteam.com/toolbox/design-of-experiments.cfm 3/11
07/02/2018 Design of Experiments (DOE) Tutorial

The structure and layout of experimental runs, or conditions.

A well-designed experiment is as simple as possible - obtaining the required information in a cost
e ective and reproducible manner.

MoreSteam.com Reminder: Like Statistical Process Control,

reliable experiment results are predicated upon two conditions: a
capable measurement system, and a stable process. If the
measurement system contributes excessive error, the experiment
results will be muddied. You can use the Measurement Systems
Analysis module from the Toolbox to evaluate the measurement
system before you conduct your experiment.
Likewise, you can use the Statistical Process Control module to
help you evaluate the statistical stability of the process being
evaluated. Variation impacting the response must be limited to
common cause random error - not special cause variation from
specific events.

When designing an experiment, pay particular heed to four potential traps that can create
experimental di culties:
1. In addition to measurement error (explained above), other sources of error, or unexplained
variation, can obscure the results. Note that the term "error" is not a synonym with "mistakes".
Error refers to all unexplained variation that is either within an experiment run or between
experiment runs and associated with level settings changing. Properly designed experiments can
identify and quantify the sources of error.
2. Uncontrollable factors that induce variation under normal operating conditions are referred to as
"Noise Factors". These factors, such as multiple machines, multiple shifts, raw materials, humidity,
etc., can be built into the experiment so that their variation doesn't get lumped into the
unexplained, or experiment error. A key strength of Designed Experiments is the ability to
determine factors and settings that minimize the e ects of the uncontrollable factors.
3. Correlation can often be confused with causation. Two factors that vary together may be highly
correlated without one causing the other - they may both be caused by a third factor. Consider
the example of a porcelain enameling operation that makes bathtubs. The manager notices that
there are intermittent problems with "orange peel" - an unacceptable roughness in the enamel
surface. The manager also notices that the orange peel is worse on days with a low production
rate. A plot of orange peel vs. production volume below (Figure 2) illustrates the correlation:
Figure 2

https://www.moresteam.com/toolbox/design-of-experiments.cfm 4/11
07/02/2018 Design of Experiments (DOE) Tutorial

If the data are analyzed without knowledge of the operation, a false conclusion could be reached
that low production rates cause orange peel. In fact, both low production rates and orange peel
are caused by excessive absenteeism - when regular spray booth operators are replaced by
employees with less skill. This example highlights the importance of factoring in operational
knowledge when designing an experiment. Brainstorming exercises and Fishbone Cause & E ect
Diagrams are both excellent techniques available through the Toolbox to capture this
operational knowledge during the design phase of the experiment. The key is to involve the
people who live with the process on a daily basis.
4. The combined e ects or interactions between factors demand careful thought prior to
conducting the experiment. For example, consider an experiment to grow plants with two inputs:
water and fertilizer. Increased amounts of water are found to increase growth, but there is a point
where additional water leads to root-rot and has a detrimental impact. Likewise, additional
fertilizer has a beneficial impact up to the point that too much fertilizer burns the roots.
Compounding this complexity of the main e ects, there are also interactive e ects - too much
water can negate the benefits of fertilizer by washing it away. Factors may generate non-linear
e ects that are not additive, but these can only be studied with more complex experiments that
involve more than 2 level settings. Two levels is defined as linear (two points define a line), three
levels are defined as quadratic (three points define a curve), four levels are defined as cubic, and so
on.

6. Experiment Design Process

The flow chart below (Figure 3) illustrates the experiment design process:
Figure 3

https://www.moresteam.com/toolbox/design-of-experiments.cfm 5/11
07/02/2018 Design of Experiments (DOE) Tutorial

7. Test of Means - One Factor Experiment

One of the most common types of experiments is the comparison of two process methods, or two
methods of treatment. There are several ways to analyze such an experiment depending upon the
information available from the population as well as the sample. One of the most straight-forward
methods to evaluate a new process method is to plot the results on an SPC chart that also includes
historical data from the baseline process, with established control limits.
Then apply the standard rules to evaluate out-of-control conditions to see if the process has been
shifted. You may need to collect several sub-groups worth of data in order to make a determination,
although a single sub-group could fall outside of the existing control limits. You can link to the
Statistical Process Control charts module of the Toolbox for help.
An alternative to the control chart approach is to use the F-test (F-ratio) to compare the means of
alternate treatments. This is done automatically by the ANOVA (Analysis of Variance) function of
statistical software, but we will illustrate the calculation using the following example: A commuter
wanted to find a quicker route home from work. There were two alternatives to bypass tra c
bottlenecks. The commuter timed the trip home over a month and a half, recording ten data points
for each alternative.

MoreSteam Reminder: Take care to make sure your experimental

runs are randomized - i.e., run in random order. Randomization is
necessary to avoid the impact of lurking variables. Consider the
example of measuring the time to drive home: if a major highway
project is started at the end of the sample period increases
commute time, then the highway project could bias the results if a
given treatment (route) is sampled during that time period.
Scheduling the experimental runs is necessary to ensure
independence of observations. You can randomize your runs using
pennies - write the reference number for each run on a penny with
a pencil, then draw the pennies from a container and record the
order.

https://www.moresteam.com/toolbox/design-of-experiments.cfm 6/11
07/02/2018 Design of Experiments (DOE) Tutorial

The data are shown below along with the mean for each route (treatment), and the variance for
each route:

As shown on the table above, both new routes home (B&C) appear to be quicker than the existing
route A. To determine whether the di erence in treatment means is due to random chance or a
statistically significant di erent process, an ANOVA F-test is performed.

The F-test analysis is the basis for model evaluation of both single factor and multi-factor
experiments. This analysis is commonly output as an ANOVA table by statistical analysis software, as
illustrated by the table below:

The most important output of the table is the F-ratio (3.61). The F-ratio is equivalent to the Mean
Square (variation) between the groups (treatments, or routes home in our example) of 19.9 divided
by the Mean Square error within the groups (variation within the given route samples) of 5.51.
The Model F-ratio of 3.61 implies the model is significant.The p-value ('Probability of exceeding the
observed F-ratio assuming no significant di erences among the means') of 0.0408 indicates that
there is only a 4.08% probability that a Model F-ratio this large could occur due to noise (random
chance). In other words, the three routes di er significantly in terms of the time taken to reach home
from work.

https://www.moresteam.com/toolbox/design-of-experiments.cfm 7/11
07/02/2018 Design of Experiments (DOE) Tutorial

The following graph (Figure 4) shows 'Simultaneous Pairwise Di erence' Confidence Intervals for
each pair of di erences among the treatment means. If an interval includes the value of zero
(meaning 'zero di erence'), the corresponding pair of means do NOT di er significantly. You can use
these intervals to identify which of the three routes is di erent and by how much. The intervals
contain the likely values of di erences of treatment means (1-2), (1-3) and (2-3) respectively, each of
which is likely to contain the true (population) mean di erence in 95 out of 100 samples. Notice the
second interval (1-3) does not include the value of zero; the means of routes 1 (A) and 3 (C) di er
significantly. In fact, all values included in the (1, 3) interval are positive, so we can say that route 1 (A)
has a longer commute time associated with it compared to route 3 (C).

Figure 4

Other statistical approaches to the comparison of two or more treatments are available through the
online statistics handbook - Chapter 7:

Statistics Handbook

8. Multi-Factor Experiments
Multi-factor experiments are designed to evaluate multiple factors set at multiple levels. One
approach is called a Full Factorial experiment, in which each factor is tested at each level in every
possible combination with the other factors and their levels. Full factorial experiments that study all
paired interactions can be economic and practical if there are few factors and only 2 or 3 levels per
factor. The advantage is that all paired interactions can be studied. However, the number of runs
goes up exponentially as additional factors are added. Experiments with many factors can quickly
become unwieldy and costly to execute, as shown by the chart below:

https://www.moresteam.com/toolbox/design-of-experiments.cfm 8/11
07/02/2018 Design of Experiments (DOE) Tutorial

To study higher numbers of factors and interactions, Fractional Factorial designs can be used to
reduce the number of runs by evaluating only a subset of all possible combinations of the factors.
These designs are very cost e ective, but the study of interactions between factors is limited, so the
experimental layout must be decided before the experiment can be run (during the experiment
design phase).

MoreSteam Reminder: When selecting the factor levels for an experiment, it

is critical to capture the natural variation of the process. Levels that are close
to the process mean may hide the significance of factor over its likely range
of values. For factors that are measured on a variable scale, try to select levels
at plus/minus three standard deviations from the mean value.

You can also download a 30-day free trial EngineRoom for Excel, MoreSteams statistical data analysis
software (an Excel add-in), to design and analyze several popular designed experiments. The
software includes tutorials on planning and executing full, fractional and general factorial designs.

9. Advanced Topic - Taguchi Methods

Dr. Genichi Taguchi is a Japanese statistician and Deming prize winner who pioneered techniques to
improve quality through Robust Design of products and production processes. Dr. Taguchi
developed fractional factorial experimental designs that use a very limited number of experimental
runs. The specifics of Taguchi experimental design are beyond the scope of this tutorial, however, it
is useful to understand Taguchi's Loss Function, which is the foundation of his quality improvement
philosophy.
Traditional thinking is that any part or product within specification is equally fit for use. In that case,
loss (cost) from poor quality occurs only outside the specification (Figure 5). However, Taguchi
makes the point that a part marginally within the specification is really little better than a part
marginally outside the specification.
As such, Taguchi describes a continuous Loss Function that increases as a part deviates from the
target, or nominal value (Figure 6). The Loss Function stipulates that society's loss due to poorly
performing products is proportional to the square of the deviation of the performance
characteristic from its target value.

https://www.moresteam.com/toolbox/design-of-experiments.cfm 9/11
07/02/2018 Design of Experiments (DOE) Tutorial

Taguchi adds this cost to society (consumers) of poor quality to the production cost of the product
to arrive at the total loss (cost). Taguchi uses designed experiments to produce product and process
designs that are more robust - less sensitive to part/process variation.
References
Webster's Ninth New Collegiate Dictionary

Additional Online Resources

An excellent online Statistics Handbook is available that covers Design of Experiments and many
other topics. See Section 5 - "Improve" for a complete tutorial on Design of Experiments.
Check the White Paper Section for related online articles.

Books
Mark J. Anderson and Patrick J. Whitcomb, DOE Simplified (Productivity, Inc. 2000). ISBN 1-56327-
225-3. Recommended - This book is easy to understand and comes with copy of excellent D.O.E.
software good for 180 days.
George E. P. Box, William G. Hunter and J. Stuart Hunter, Statistics for Experimenters - An
Introduction to Design, Data Analysis, and Model Building (John Wiley and Sons, Inc. 1978). ISBN
0-471-09315-7
Douglas C. Montgomery, Design and Analysis of Experiments (John Wiley & Sons, Inc., 1984) ISBN
0-471-86812-4.
Genichi Taguchi, Introduction to Quality Engineering - Designing Quality Into Products and
Processes (Asian Productivity Organization, 1986). ISBN 92-833-1084-5

Summary
Designed experiments are an advanced and powerful analysis tool during projects. An e ective
experimenter can filter out noise and discover significant process factors. The factors can then be
used to control response properties in a process and teams can then engineer a process to the exact
specification their product or service requires.
A well built experiment can save not only project time but also solve critical problems which have
remained unseen in processes. Specifically, interactions of factors can be observed and evaluated.
Ultimately, teams will learn what factors matter and what factors do not.

https://www.moresteam.com/toolbox/design-of-experiments.cfm 10/11
07/02/2018 Design of Experiments (DOE) Tutorial

Additional Resources
Recorded Webcast: "Experimental Design in the Transactional Arena"

Need a tool to setup

and run a DOE?
Download a 30-day trial
version of EngineRoom
for Excel, MoreSteam's Microsoft® Excel®-based statistical tool, and
start today.

Want to learn more?

MoreSteam.com o ers a wide range of Lean Six Sigma online courses,
including Black Belt, Green Belt, and DFSS training. Flexible training at an
a ordable price.
View our current course catalog

Get MoreNews! Visit Toolbox Knowledge Center

Our monthly email Home Visit our resources
delivers the latest See what else is section more free
ideas and resources. available in our free materials.
Toolbox.
Subscribe Today! VIEW MORE
BACK TO THE TOOLBOX RESOURCES

© MoreSteam e-Learning & Software & Technology Resources Company

9976 Brewster Lane Certification Data Analysis Software All Free Resources Support Center
Powell, OH 43065 Course Catalog Training Simulations Blended Learning
hello@moresteam.com Training for Enterprise Technology Playbook
614.602.8190 Deployments Newsletter
Privacy Policy Training for Individuals Blog
Terms of Use Certification Home

https://www.moresteam.com/toolbox/design-of-experiments.cfm 11/11