SEM711

TIM BODISCO

Product Development
Technologies
Lecture 5: Basics of
Deakin University CRICOS Provider Code: 00113B

CONTEN
TS
Why do experiments
Ethics
Experiments
Experiment Design
Bias
Questions?

Deakin University CRICOS Provider Code: 00113B

WHY EXPERIMENT
Researchers/Engineers/Scientists/etc use experiments
to answer questions
Comparative experiments can help us make better
decisions
We can design experiments specifically to test single
parameters or the interaction between parameters,
while minimising any bias
Or at least minimising the error

Because we are in control with an experiment we are

better able to make inferences about the nature of
differences we observe during the experiment
Deakin University CRICOS Provider Code: 00113B

OBSERVATION
We will focus on experiments, but a lot of this will have
parallels to observational studies
An observational study is fundamentally different to
an experiment because the researcher is not in control
Observational studies have a risk that misleading
conclusions can easily be drawn
Observed differences between treatment groups could be due to some
other hidden mechanism
Correlation does not imply causation

This does not diminish their utility. Some problems can

only be investigated from an observation perspective
Deakin University CRICOS Provider Code: 00113B

CONSTRAINTS
Not all experiments are possible
Perhaps they could cause harm:
Humans
Animals
Environment

Is there a relationship between smoking and lung

cancer?
We wont cover moral/ethical issues with
experimental design
Ill leave it up to you to consider this for yourself if the situation
arises in your future careers

But this does lead us to our next topic

Deakin University CRICOS Provider Code: 00113B

ETHICS
Full disclosure
Dont hide anything
You dont want any skeletons in the
closet coming back to scare you later in
your career!

Dont invent the truth

Be objective and unbiased

Give credit for information/ideas

Dont claim something was your
discovery when it wasnt
Deakin University CRICOS Provider Code: 00113B

ETHICS
Dont cook the data
Retaining only those points that fit
the theory/prove your point while
you discard those that refute it

Dont trim the data

Smooth out irregularities to make
the data more accurate/precise

Dont forge the data

Dont invent your data!
Be careful to not do this
unintentionally
Your experimental design also
needs to be unbiased
Deakin University CRICOS Provider Code: 00113B

All data is real

Explain outlying data
It may be more important
than you first think
If you do choose to
exclude a data point in
your analysis, explain why
and justify this
Dont hide it

ETHICS
Engineering is a profession which relies on ethical
behaviour
Without ethics:
The profession loses public trust
There is risk of loss of life, property or other resources

Once you have demonstrated unethical behaviour your

reputation as a professional engineer will likely be
unrecoverable

You could still have a career in politics

Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Your goal is to assess
your hypothesis
In this case that would be that
your device meets the design
criteria and can be tested to
show that the theoretical
dynamics are the same as
reality

Therefore, you design

an experiment to
achieve your objective of
finding data to support
or refute your hypothesis
Deakin University CRICOS Provider Code: 00113B

Scientific experimentation involves

the testing of falsifiable hypotheses

Karl Poppers Critical

Rationalism
Verification cannot be
assumed from past
experience
Falsification only can
be demonstrably
shown

EXPERIMENTS
Scientific experimentation involves the testing of
falsifiable hypotheses
Forces restriction of experiments to ideas that can be falsified
(establishes what is testable)
Instills a sense of skepticism, which may be helpful in critically
analysing data
Teaches the concept of framework
Collected data might be limited in its applicability

Therefore, you design an experiment to achieve

your objective of finding data to support or refute
your hypothesis

Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Be careful that your choice of hypotheses is not biasing
the conclusion by:
Demanding only one type of measurement
Grouping all opposing data as negative
Allowing for validation even when a result may not represent the norm

A hypothesis by definition is a statement without an

inductive basis
Therefore caution needs to be taken when using it to establish validation
Consider a question such as: Does smoking cause cancer

Experiments are usually aimed at confirmation,

therefore
there is an inherent tension between the experiment
and the paradigm framing the experiment
Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
The issues of the hypothesis framework can be
overcome with a problem/question approach
One that does not make such a distinction between
positive and negative data
Here a model is validated or it is not

Framed within an open-ended question, which should

allow an answer
More flexible as it allows questions such as how, what
and why
Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS

Problem/Question Framework
Question

Answer

Question

Answer
Model

Query Model

Verification

Perturbed
Model
Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Return to the question, does smoking cause cancer
Here a hypothesis would be: smoking causes cancer
This could be validated by observing smokers with cancer
Whether the reality here is true or not, this mode will likely establish
credibility for that claim

Would be better to pose in a problem/question framework

What are the effects of smoking?
This way it is not limiting and it is not ignoring other influencing
parameters
Particularly parameters which influence others
It also forces us to consider how we will measure this
Design of many inductive experiments to examine effects of different
aspects of smoking

Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Rather than testing the hypothesis: the device
meets the design criteria and can be tested to show
that the theoretical dynamics are the same as reality
It would be better to ask: what are the effects of
moving the central rotating component of my device
and how does the subsequent movement compare
to the theoretical dynamics

Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Once you know what you
want to achieve, you will
need to put together an indepth path to get there
You must think through the
whole process end-to-end
Identify the milestones
Be clear about the
definitions
Deakin University CRICOS Provider Code: 00113B

The data
collection
/experiment
execution
What will I do
Why will I do it
Where will I do it
Who will help me
How will I do it
How well do I have
to do it
When will I do it

EXPERIMENTS
Common mistakes
The variation due to
experimental error is ignored
Important parameters are
not controlled
Effects of different
parameters are not isolated
Overly simple one-factor-ata-time experimental designs
are used
Interactions between
parameters are ignored
Too many experiments are
conducted

Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Consider what is practical/ethical
Should I test my product in the real world, or under simulated
conditions in a laboratory
Where can I do my experiments

How will I measure/isolate my variables?

This is often where a lot of the thinking occurs
Not always easy
Need to be careful, if you isolate all your variables too well you may
miss the important interactions between them
Can you measure the variables and prove that the induced effect
happened?
Can you establish that the phenomena was caused solely by the
induced
effect?
Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
One factor at a time experiments

Not statistically efficient

Misleading or wrong conclusions if the factors, k, have interactions
Therefore, usually not the recommended approach

Full factorial design (all combinations)

Can find the effect of all factors

Too much time, money and resource
If there are a lot of factors, this likely is not a good choice either
Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
Partial factorial design (all of the most important
combinations)
Has the potential to miss something important
Careful planning can mitigate this risk
If you only miss the unimportant/negligible interactions is it really a big
deal?
Especially if the alternative is not having the resource to do the work!
Saves time and expense
In most cases is the optimal approach
Takes planning and thought
Really consider what you are testing, what you expect to find and how
you think everything will interact
Often starting where you think it is most likely to fail can save you time
Pessimistic outlook, but sensible
Deakin University CRICOS Provider Code: 00113B

EXPERIMENTS
The goal of proper experimental design is to get the
maximum information with the minimum number of
experiments!
Above all, keep in mind that correlation does not always
mean causation. And regardless of how strong you think
your evidence is, it does not constitute proof
Sometimes the data is misleading
What if I gave you Cs-137 and asked you what type of
radiation it emits?
Youd likely setup an experiment where you measured the
radiation from it and conclude that it is a gamma emitter
Youd be wrong. It is a beta emitter, it decays into
Ba-137m, which is a gamma emitter.
Deakin University CRICOS Provider Code: 00113B

EXPERIMENT
DESIGN

Not overly dissimilar to the product design process

you learned in Week 2!
Phase 0:
Planning

Phase 1:
Concept
Development

Phase 2:
SystemLevel
Design

Phase 3 :
Detail
Design

Phase 4:
Testing &
Refinement

Phase 5:
Production
Ramp-Up

Phase 0:
Objective of
the
Experiment

Phase 1:
Selection of
factors/level
s and
expected
interactions

Phase 2:
Selection of
experimental
design

Phase 3:
Experiment

Phase 4:
Statistical
data analysis

Phase 5:
Conclusions
/
Disseminatio
n

Deakin University CRICOS Provider Code: 00113B

PICK A NUMBER

1 2 3 4
It is very easy to introduce
unintentional bias in an experiment.
Everything must be well thought out
and well controlled
What does this result tell you about
bias?
Deakin University CRICOS Provider Code: 00113B

INTRODUCTION OF
BIAS
Method used to perform the
experiment
Selection methods
Transfer effects

Outside influence

Introducing someone elses

bias

Measuring equipment

Changes as a result of having already been

tested

Asymmetric transfer
Does A influence B more than B influences A?

Fatigue
Analyst preconception
Inference methods
Deakin University CRICOS Provider Code: 00113B

LETS LEAVE IT
THERE
If you are interested in
this topic, consider
reading up on the
following:
DOE
Taguchi
Factorial Analysis
Experimental Methods

Deakin University CRICOS Provider Code: 00113B

Questions or
Concerns?

Deakin University CRICOS Provider Code: 00113B