Statistical vs.

Practical
Significance
 Statistical Significance
Significant differences (i.e., reject the null
hypothesis) means that differences in group
means are not likely due to sampling error.
The problem is that statistically significant
differences can be found even with very
small differences if the sample size is large
enough.
 Statistical Significance
In fact, differences between any sample
means will be significant if the sample is
large enough.
For example men and women have
different average IQs
 Practical Significance
Practical (or clinical) significance asks the
larger question about differences
Are the differences between samples big
enough to have real meaning.
Although men and women undoubtedly
have different IQs, is that difference large
enough to have some practical implication
 Practical Significance
The fifth edition of the APA (2001) Publication Manual
states:
that it is almost always necessary to include
some index of effect size or strength of
relationship in your Results section. The
general principle to be followed is to provide
the reader not only with information about
statistical significance but also with enough
information to assess the magnitude of the
observed effect or relationship. (pp. 2526)
 Practical Significance
Generally assessed with some measure of
effect size
Effect size can be grouped into two
categories:
Difference measures
Variance accounted for measures
 Difference effect sizes
Simple mean difference
Suppose you design at control group
experiment to evaluate the effects of CBT
on depression.
Experimental group post test score = 18
Control group post test score = 16
Difference = 18 16 = 2
 Difference effect sizes
Problem with simple mean difference
Dependent on the scale of measurement
Ignores normal variation in scores
For example, if the following example was based on a
scale with a SD of 15 points, a 2 point difference would
be small treatment would only effect depression by .
13 SDs.
If the example was based on a scale with a SD of 1
point, a 2 point difference would be very large
treatment had a 2 SD effect
 Difference effect sizes
We can overcome this problem by standardizing
the mean differences
One measure of this was done by Gene Glass
= (meantx meancontrol)/ Sdcontrol

Other SDs may be used such as a pooled

(combined) SD from the Tx and Control groups
If variances are equal

X1 X 2
d
( S 1 S 2) 2
2 2
 If variances are unequal

X G1 X G 2 s12 n1 1 s22 n2 1
d s pooled
s pooled n1 n2 2
 Difference effect sizes:
Interpreting
Cohen proposed a general method for interpreting
these type of effect sizes
d = .2 small effect
d = .5 medium effect
d = .8 large effect

This is a guideline for interpretation.

You need to interpret effect sizes in the context of
the research
Variance accounted for measures
When comparing variables, variance accounted for
measures tell us how well one variable predicts
another or the magnitude of the relation.
R2 is one such measure from correlational or
regression analysis.
Eta squared () is often used in ANOVA as a
measure of shared variance.
Omega squared (2) is also used with ANOVA
Variance accounted for measures:
Interpreting
Correlations can be judged as:
R = .1 small
R = .3 moderate
R = .5 large

For measures of variance based on a squared

value take the square root to get a
correlation
 Confidence Intervals
Statistics are used to estimate the true
population value.
When providing statistics (estimates of
population values) it is useful to provide a
range of values that are likely to include the
true population value.
Calculated with the standard error of the
statistic
 Confidence Intervals for means
Confidence intervals = mean z(SEM)
Z = 1.96 for a 95% confidence interval
(you can estimate with Z=2 for a 95% confidence interval)
If the mean of a sample = 100 and the SEM = 2
Then a 95% confidence interval would be:
100 1.96(2) = 100 3.92
Or 100 2(2) = 100 4 is close enough for govt.
work
 Confidence Intervals
Use confidence intervals when you want to
show where some true value is likely to be
Reporting test results