ME-5102

Research Methodology &

Design of Experiments
Lect. # 3
DESIGN OF EXPERIMENTS

Dr. Nazeer Ahmad Anjum

Mechanical Engineering Program
Engineering University Taxila
 Key concepts: 3

Intro to Research Design

Requirements of Research Design
Definition of Research Design
Types of Research Design
Factors Affecting Research Design
Hypothesis Formulation
Hypothesis Testing

07-Mar-17

Experimental Research Steps 4

State the research problem
Determine if experimental methods Feasible/apply
Specify the independent variable(s)
Specify the dependent variable(s)
State the tentative hypothesis
Determine measures to be used
Pause to consider potential success
Identify intervening (extraneous) variables
Formal statement of research hypothesis
Design the experiment
Final estimate of potential success
Conduct the study as planned
Analyze the collected data
Prepare a research Design report

07-Mar-17
 Introduction 5
Why is this trip necessary? Goals of the
DOE
An abbreviated history of DOX
Some basic principles and terminology
The strategy of experimentation
Guidelines for planning, conducting and
analyzing experiments

07-Mar-17

What is Research Design? 6

A plan for selecting the sources and types
of information used to answer research
questions
A framework for specifying the
relationships among the study variables
A scheme that outlines each procedure
from the hypothesis to the analysis

07-Mar-17
Requirements of Research Design 7
Identify the kind of Research one Intend to Do
Be Realistic
Be Precise and
Be Flexible

07-Mar-17

DOE History 8
The agricultural origins, 1918 1940s
R. A. Fisher & his co-workers
Profound impact on agricultural science
Factorial designs, ANOVA
The first industrial era, 1951 late 1970s
Box & Wilson, response surfaces
Applications in chemical & process industries
The second industrial era, late 1970s 1990
Quality improvement initiatives in many companies
Taguchi and robust parameter design, process
robustness
The modern era, beginning 1990
07-Mar-17
 Introduction to DOX 9
An experiment is a test or a series of tests
Experiments are used widely in the engineering
world
Process characterization & optimization
Evaluation of material properties
Product design & development
Component & system tolerance determination
Plays an important role in new product design
To develop a ROBUST design a process affected
minimally by external sources of variability.
All experiments are designed experiments, some
are poorly designed, some are well-designed
07-Mar-17

Meaning of Research Design 10

IntheformallanguageResearchdesignsuggests
A plan of action to be carried out in connection
withaproposedresearchwork
It provides a guideline for the checking and
counter checking of the research work which
enables the researcher to keep track of work
actionandlethimtoknowthatheismovinginthe
rightdirection
Thedesignisaspecificpresentationofthevarious
stepsintheprocessofresearch
Designingprovidesaclearcutpictureofthework
beforestartingthework,and
It helps in identifying the difficulties that may
emergeintheprocessofresearch 07-Mar-17
Need and Importance of Research11
Design
Research without a proper design is said to
be business without advertisement.
It is said a business without advertisement
is winking at an famous personality in dark
where you know what you are observing
but no body else does
Similarly research without a proper design is
working in the dark without knowing the
proper way.

07-Mar-17

Role of DOE in Process 12

Improvement
DOE is a formal mathematical method for
systematically planning and conducting
scientific studies that change experimental
variables together in order to determine
their effect of a given response.
DOE makes controlled changes to input
variables in order to gain maximum
amounts of information on cause and effect
relationships with a minimum sample size.

07-Mar-17
 Role of DOE in Process 13
Improvement
DOE is more efficient that a standard
approach of changing one variable at a
time in order to observe the variables
impact on a given response.

DOE generates information due to the

effect of various factors have on a response
variable and in some cases may be able to
determine optimal settings for those factors.

07-Mar-17

Role of DOE in Process 14

Improvement
DOE encourages brainstorming activities
associated with discussing key factors that
may affect a given response and allows the
experimenter to identify the key factors for
future studies.
DOE is readily supported by numerous
statistical software packages available on
the market.
Assignment # 2: Learning Design Expert or
any other Software and analyzed
Experimental data 07-Mar-17
Experimental Research Design 15
Experimental research design is also called as
experimentation or causal research design.
It is defined as the process where events
occur in a setting at the discretion of the
experimenter, and controls are used to
identify the sources of variation in subjects
response.
This definition indicates that the experimenter
systematically manipulates some factor(s)
to get a response from the subject.

07-Mar-17

Predominant Areas of Experimental 16

Research
Product design: which product a consumer
would buy most.
Package design: which attractive package a
producer should design based on the
available literature.
Pricing Policies: the price elasticity of
demand can be better tested with the help of
experiments. Experiments are often used for
measuring this aspect pricing.
Promotion Policies: the promotional
policies have been very widely explored
through experimentation as compared with
the other areas of marketing.

07-Mar-17
 Methods of Experimental Design 17
There are two types of research design:
Formal research design
Completely randomized
Randomized block
Latin square, and
Factorial design.
Informal research design
After only design
After only with control design
Before and after without control design
Before and after with control design, and
Ex-post facto design
07-Mar-17

Formal Research Design 18

Randomized Block Design:
In this type of design, the population is
divided into groups which are relatively
homogeneous in respect of some selected
variables.
The variable selected for grouping the
population is one that is believed to be
related to the measures to be obtained in
respect of dependent variables.

07-Mar-17
 Formal Research Design 19
Latin Square Design:
This is the most frequently used design in case
of agricultural research process.
An experiment has to be made on the effects
of five different varieties of fertilizers on the
yield of a certain crop.
Suppose that there are 2 factors, fertility and
seeds that found to be important.

07-Mar-17

Informal Research Design 20

1. After-only Design:
This design consists of applying the experimental
variables to an experimental group and measuring
the dependent variable.
For example: A detergent powder manufacturer with brand
nameSURFEXCELLwanttoknowthemarketresponseofhis
product. He advertised in TV for a month. Now he wants to
measuretheimpactofTVadvertisementonhissale.Henow
comparesthesaleoftheproductafteradvertisement

2. After-Only controls Design :

In this case two equivalent groups are selected: one
is control group and other one is experiment group.
The treatment is provided to the experimental
group. 07-Mar-17
 Informal Research Design 21
3. Before-after without Control Design:
In this design the researcher measures the test
unit (independent variables) before and after it
has been subjected to the treatment.
4. Before-After with control Design :
In this case an experimental unit is selected
and studied during the experimental period.

07-Mar-17

Informal Research Design 22

5. Ex-post Facto Design:
(After the Fact) In this case, rather than
selecting the experimental design before the
experiment, none of the groups is selected
before the experiment.
It is a quasi-experimental study examining how
an independent variable, present prior to the
study in the participants, affects a dependent
variable.

07-Mar-17
 CONCLUSION: 23
Thus from the mentioned methods it is clear
that the process of research design play an
important role in the analysis of research.
This helps the researcher, particularly the
beginners, a way to get the desired result.

07-Mar-17

Engineering Experiments 24
Experimental Example: A Metallurgical Engineer is
interested in studying the effect of two different
hardening processes, oil quenching and saltwater
quenching, on an aluminum alloy.
The objective of the experimenter is to determine which
quenching solution produces the maximum hardness for
this particular alloy.
The engineer decides to subject a number of alloy
specimens or test coupons to each quenching medium
and measure the hardness of the specimens after
quenching.
The average hardness of the specimens treated in each
quenching solution will be used to determine which
solution is best.
07-Mar-17
 Results and Conclusions of 25
Experiment
The results and conclusions depend to a large extent on the
manner in which the data were collected.
Lets consider previous example, the metallurgical engineer
used specimens from one heat in the oil quench and
specimens from a second heat in the saltwater quench.
Now, when the mean hardness is compared, the engineer
is unable to say how much of the observed difference is
the result of the quenching media and how much is the
result of inherent differences between the heats.
Thus, the method of data collection has adversely
affected the conclusions that can be drawn from the
experiment.

07-Mar-17

Engineering Experiments 26
Possible Questions
1. Are these two solutions the only quenching media of
potential interest?
2. Are there any other factors that might affect hardness
that should be investigated or controlled in this experiment?
3. How many coupons of alloy should be tested in each
quenching solution?
4. How should the test coupons be assigned to the
quenching solutions, and in what order should the data
be collected?
5. What method of data analysis should be used?
6. What difference in average observed hardness between
the two quenching media will be considered important?

07-Mar-17
 Results and Conclusions of 27
Experiment
Experiments are used to study the performance of
processes and systems.
The process or system can be
represented by the model shown in
figure 1.1.
We usually visualize the process as a
combination of Manpower,
Materials, Machines, Methods,
Money, and other resources that
transforms some input (often a
material) into an output that has one
or more observable responses.
Some of the process variables x1, x2, . . . , xp are controllable,
whereas other variables z1, z2, , Zq are uncontrollable (although
they may be controllable for purposes of a test).
07-Mar-17

Engineering Experiments 28
Reduce time to design/develop new products
& processes
Improve performance of existing processes
Improve reliability and performance of
products
Achieve product & process robustness
(Taguchi) [reducing variation in a product
without eliminating the causes of the variation]
Evaluation of materials, design alternatives,
setting component & system tolerances, etc.

07-Mar-17
 Objectives of the Experiment 29
1. Determining which variable(s) are most
influential on the response y
2. Determining where to set the influential x's so
that y is almost always near the desired
nominal value
3. Determining where to set the influential x's so
that variability in y is small.
4. Determining where to set the influential x's so
that the effects of the uncontrollable variables
z1, z2, . zq, are minimized.

07-Mar-17

BASIC STEPS IN DOE 30

Four elements associated with DOE:

1. The design of the experiment,
2. The collection of the data,
3. The statistical analysis of the data, and
4. The conclusions and recommendations
made as a result of the experiment.

07-Mar-17
 The Basic Principles of DOX 31
Replication
Repetition of a basic experiment without
changing any factor settings
Allows the experimenter to estimate the
experimental error (noise) in the system used
to determine whether observed differences in
the data are real or just noise.
Replication versus repeat measurements?
(see page 13)
Allows the experimenter to obtain more
statistical power (ability to identify small
effects)
07-Mar-17

The Basic Principles of DOX 32

Blocking
A technique used to increase the precision of
an experiment by breaking the experiment into
homogeneous segments (blocks) in order to
control any potential block to block variability.
Examples: (multiple lots of raw material,
several shifts, several machines, several
inspectors).
Any effects on the experimental results as a
result of the blocking factor will be identified
and minimized.
07-Mar-17
 The Basic Principles of DOX 33
Randomization
Running the trials in an experiment in random
order
A statistical tool used to minimize potential
uncontrollable biases in the experiment by
randomly assigning material, people, order that
experimental trials are conducted or any other factor
not under the control of the experimenter.
Results in averaging out the effects of the
extraneous factors that may be present in order to
minimize the risk of these factors affecting the
experimental results.
07-Mar-17

DOE Terminology 34
Confounding
AExample:
confoundingThevariable, also known
relationship betweenas athethird variable
force applied orto
a
mediator
a ball andvariable, can adversely
the distance affect the relation between
the ball travels.
the
Theindependent variable
natural prediction andbe
would dependent
that the ballvariable.
given the most
This
forcemay
wouldcause
travelthe researcher
furthest. to analyze
However, the results
if the confounding
incorrectly.
variable is aThe results may
downward slantedshow a false
piece correlation
of wood to help
between theball,
propel the dependent andwould
the results independent variables,
be dramatically leading
different.
to an incorrect
The slanted rejection
wood is ofthe the null hypothesisvariable that
confounding
changes the outcome of the experiment.

07-Mar-17
 DOE Terminology 35
Confounding
A concept that basically means that multiple effects
are tied together into one parent effect and cannot be
separated. For example,
Two people flipping two different coins would result in
the effect of the person and the effect of the coin to be
confounded
As experiments get large, higher order interactions
are confounded with lower order interactions or main
effect.
For example, if you are researching whether lack of
exercise leads to weight gain, lack of exercise is your
independent variable and weight gain is your
dependent variable.
07-Mar-17

DOE Terminology 36

Factors
Experimental factors or independent variables
(continuous or discrete) an investigator
manipulates to capture any changes in the
output of the process.
Other factors of concern are those that are
uncontrollable and those which are
controllable but held constant during the
experimental runs.

07-Mar-17
 DOE Terminology 37

Responses
Dependent variable measured to describe
the output of the process.

Treatment Combinations (run)

Experimental trial where all factors are set
at a specified level.

07-Mar-17

DOE Terminology 38
Fixed Effects Model
If the treatment levels are specifically chosen
by the experimenter, then conclusions reached will
only apply to those levels.
Use fixed-effects (FE) whenever you are only interested
in analyzing the impact of variables that vary over
time.
The fixed-effects model controls for all time-invariant
differences between the individuals, so the estimated
coefficients of the fixed-effects models cannot be biased
because of omitted time-invariant characteristics[like culture,
religion, gender, race, etc]

07-Mar-17
 DOE Terminology 39
Fixed Effects Model
FE explore the relationship between predictor and
outcome variables within an entity (country,
person, company, etc.). Each entity has its own
individual characteristics that may or may not
influence the predictor variables. FE models are
designed to study the causes of changes within a
material, process, person [or entity]
Examples:, being a male or female could influence
the opinion toward certain issue, or
The political system of a particular country could have
some effect on trade or GDP; or
The business practices of a company may influence its
stock price
07-Mar-17

DOE Terminology 40

Random Effects Model

If the treatment levels are randomly chosen

from a population of many possible
treatment levels, then conclusions reached
can be extended to all treatment levels in the
population.

07-Mar-17
 Planning a DOE 41

Everyone involved in the experiment should

have a clear idea in advance of exactly what is
to be studied, the objectives of the
experiment, the questions one hopes to
answer and the results anticipated

07-Mar-17

Planning a DOE 42

Select a response/dependent variable(s) that

will provide information about the problem
under study and the proposed measurement
method for this response variable, including an
understanding of the measurement system
variability

07-Mar-17
 Planning a DOE 43
Choose an appropriate experimental design
(relatively simple design and analysis methods are
almost always best).
It will allow your experimental questions to be
answered once the data is collected and analyzed.
But keeping in mind tradeoffs between statistical
power and economic efficiency.
At this point in time it is generally useful to simulate
the study by generating and analyzing artificial data
to insure that experimental questions can be
answered as a result of conducting your experiment.

07-Mar-17

Planning a DOE 44

Select the independent variables/factors

(quantitative or qualitative) to be investigated
in the experiment, the number of levels for
each factor, and the levels of each factor
chosen either specifically (fixed effects model)
or randomly (random effects model).

07-Mar-17
 Planning a DOE 45

Perform the experiment (collect data) paying

particular attention such things as
randomization and measurement system
accuracy, while maintaining as uniform an
experimental environment as possible. How
the data are to be collected is a critical stage
in DOE

07-Mar-17

Planning a DOE 46

Analyze the data using the appropriate

statistical model insuring that attention is paid
to checking the model accuracy by validating
underlying assumptions associated with the
model.
Be liberal in the utilization of all tools, including
graphical techniques, available in the
statistical software package to insure that a
maximum amount of information is
generated

07-Mar-17
 Planning a DOE 47

Based on the results of the analysis, draw

conclusions/inferences about the results.
Interpret the physical meaning of these results.
Determine the practical significance of the
findings, and
Make recommendations for a course of
action including further experiments

07-Mar-17

Strategy of Experimentation 48
Best-guess experiments
Used a lot by engineers and scientists; based on technical or
theoretical knowledge
More successful than you might suspect, but there are
disadvantagesMay continue for a long time or may stop
One-factor-at-a-time (OFAT) experiments
Sometimes associated with the scientific or
engineering method
Devastated by interaction, also very inefficient
Statistically designed experiments
Based on Fishers factorial concept

07-Mar-17
 Factorial Designs 49
The general approach to plan and conducting the experiment is
called the strategy of experimentation. There are several strategies
that an experimenter could have
In a factorial experiment, all possible
combinations of factor levels are
tested
The golf experiment:
Type of driver (over or regular)
Type of ball (balata or 3-piece)
Walking vs. riding a cart
Type of beverage (Water vs Bear)
Time of round (am or pm)
Weather
Type of golf spike
Etc, etc, etc
07-Mar-17

Factorial Designs 50
Driver Effect

Driver Effect = 92 + 94 + 93 + 91 - 88 + 91 + 88 + 90
4 4
= 3.25
07-Mar-17
 Factorial Designs 51
Ball Effect

Ball Effect = 88 + 91 + 92 + 94 - 88 + 90 + 93 + 91
4 4
= 0.75 07-Mar-17

Factorial Designs 52
Ball-Driver Effect

Ball-Driver Interaction Effect = 92+94+88+90 _ 88+91+93+91

4 4
= 0.25
07-Mar-17
 Factorial Designs with Several 53
Factors
Three Factor Factorial
Driver, Ball Type, Beverage Type

07-Mar-17

Factorial Designs with Several 54

Factors
A Fractional Factorial
Four Factor Factorial
Driver, Ball Type, Beverage Type, Mode of Travel

07-Mar-17
Application of Experimental Design 55
Application of experimental design in process
development

1. Improved process yields

2. Reduced variability and closer conformance
to nominal or target requirements
3. Reduced development time and
4. Reduced overall costs

07-Mar-17

Application of Experimental Design 56

Application of experimental design in Engineering
design
1. Evaluation and comparison of basic design
configurations
2. Evaluation of material alternatives
3. Selection of design parameters so that the
product will work well under a wide variety
of field conditions, that is, so that the
product is robust
4. Determination of key product design
parameters that impact product
performance
07-Mar-17
Planning, Conducting & Analyzing an 57
Experiment
1. Recognition of & statement of problem
2. Choice of factors, levels, and ranges
3. Selection of the response variable(s)
4. Choice of design
5. Conducting the experiment
6. Statistical analysis
7. Drawing conclusions, recommendations

07-Mar-17

Planning, Conducting & Analyzing an 58

Experiment
At initial stage get statistical thinking/knowledge.
Your non-statistical knowledge is crucial to
success
Pre-experimental planning (steps 1-3) vital
Think and experiment sequentially (use the KISS
principle) "Keep it simple & straightforward"

See Coleman & Montgomery (1993)

Technometrics paper + supplemental text material

07-Mar-17
 Types of Designs 59
The basic structure of a research study
Threats to particularly relevant to
experimental research
Types of designs (Campbell & Stanley, 1963)
Pre-experimental
True experimental and
Quasi-experimental

07-Mar-17

True Experimental Designs 60

Best type of research design because of
their ability to control threats to internal
validity
Utilizes random selection of participants
and random assignment to groups
Example: Pretest/posttest control group
design

07-Mar-17
 Pre-experimental Designs 61
Weak experimental designs in terms of
control
No random sampling
Threats to internal and external validity
are significant problems
Many definite weaknesses
Example: One-group pretest/posttest design

07-Mar-17

Quasi Experimental Designs 62

These designs lack either random
selection of participants or random
assignment to groups
They lack some of the control of true
experimental designs, but are generally
considered to be fine
Example: Nonequivalent group design

07-Mar-17
 Methods of Control 63

Physical manipulation
Selective manipulation
Matched pairs and block designs
Counterbalanced designs
Statistical techniques

07-Mar-17

Selective Manipulation 64
Intent is to increase likelihood that treatment
groups are similar at the beginning of study
Matched pairs design
Participants are matched according to some
key variable and then randomly assigned to
treatment group
Block design extension of matched pairs to
3 or more groups
Counterbalanced design
All participants receive all treatments, but in
different orders 07-Mar-17
 Physical Manipulation 65
Best way to control extraneous variables
Researcher attempts to control all
aspects of the research, except the
experimental treatment
Difficult to control all variables
Some variables cannot be physically
controlled

07-Mar-17

Statistical Techniques 66
Applied when physical manipulation or
selective manipulation is not possible
Differences among treatment groups are
known to exist at beginning of study
Groups may differ on initial ability
Analysis of covariance (ANCOVA)
Adjusts scores at the end of the study based
upon initial differences

07-Mar-17
 Common Sources of Error 67
Many possible sources of error can cause the results of
a research study to be incorrectly interpreted. The
following sources of error are more specific threats
to the validity of a study than those described
previously, Selected examples:

Hawthorne Effect
Placebo Effect
John Henry Effect
Rating Effect
Experimenter Bias Effect

07-Mar-17

Hawthorne Effect 68
The alteration of behavior by the subjects of a study
due to their awareness of being observed.
If a change in lighting would affect productivity.
A specific type of reactive effect in which merely being
a research participant in an investigation may affect
behavior.

The Hawthorne Effect refers to the fact that people will

modify their behavior simply because they are being
observed.

Suggests that, as much as possible, participants

should be unaware they are in an experiment and
unaware of the hypothesized outcome
07-Mar-17
 John Henry Effect 69
A threat to internal validity wherein research participants in
the control group try harder just because they are in the
control group
This effect was coined to explain the unexpected outcome
of an experiment caused by the control group's
knowledge of its role within the experiment.
The control group's perceived role as a baseline or a
comparison group to the experimental condition,
specifically one testing an innovative technology, can
cause the control group to behave in an unnatural way to
outperform the new technology.
The group's knowledge of its position within the experiment
as a baseline comparison causes the group to perform
differently and, often more specifically, better than usual,
eliminating the effect of the experimental manipulation.07-Mar-17

Placebo Effect (I shall please) 70

Participants may believe that the experimental
treatment is supposed to change them, so they
respond to the treatment with a change in
performance.

A phenomenon, in which the recipient

perceives an improvement in condition due to
personal expectations, rather than the
treatment itself, is known as the placebo effect
or placebo response. Placebos are an
important methodological tool in medical
research.
07-Mar-17
 Rating Effect 71
Variety of errors associated with ratings of
a participant or group
Halo effect
Overrater error
Underrater error
Central tendency error

07-Mar-17