Open Dialogue:

The widespread abuse of statistics by

researchers: What is the problem and
what is the ethical way forward?
Professor Stephen Gorard

The paper presents and illustrates two areas of widespread abuse of statistics in social science research. The
first is the use of techniques based on random sampling but with cases that are not random and often not
even samples. The second is that even where the use of such techniques meets the assumptions for use,
researchers are almost universally reporting the results incorrectly. Significance tests and confidence
intervals cannot answer the kinds of analytical questions most researchers want to answer. Once their
reporting is corrected, the use of these techniques will almost certainly cease completely. There is nothing to
replace them with but there is no pressing need to replace them anyway. As this paper illustrates, removing
the erroneous elements in the analysis is usually sufficient improvement (to enable readers to judge claims
more fairly). Without them it is hoped that analysts will focus rather more on the meaning and limitations
of their numeric results.

Which kind of statistics is being abused?

T
He TerM ‘statistics’ is an ambiguous
one. It emerged from the collation and
use of figures concerning the nation
state from the 17th century onwards in the
UK, and subsequently in the Us and else-
where (porter, 1986). such figures involved
relatively simple analyses, and ‘political arith-
metic’ was largely used to lay bare inefficien-
cies, inequalities and injustice (Gorard,
2012). However, more recently and for many
commentators the term has come to mean a
set of techniques derived from sampling
theory, and/or the products of those tech-
niques. It is the abuse of such techniques
that is the subject of this new paper. These
techniques include the use of standard
errors, confidence intervals and significance
tests (both explicitly and disguised within
more complex statistical modelling). They
are supposedly used to help analysts to
decide whether something that is found to
be true of the sample achieved in a piece of
research is also likely to be true of the known
population from which that sample was
drawn. all of these statistical techniques,
including confidence intervals, are based on
a modified form of an argument modus
tollendo tollens. In formal logic, the argument
of denying the consequent is as follows:
If a is true then B is true
B is not true
Therefore, a is not true also
This is a perfectly valid argument, and the
conclusion must be true, as long as the
premising statements are all definitive. If B is
not true then it is certain that a is not true.
However, as soon as tentativeness or proba-
bility enters the argument fails:
If a is true then B is probably true also
B may not be true
Therefore, a may not be true also
This is not really a valid argument, and the
truth of the conclusions is contingent
on many factors beyond pure logic.
Characteristic a may or may not be true. If it
is true, characteristic B could be true as well,
or not. The observation that B may (or may
not) be true says almost precisely nothing
about the truth of a. The first premise may
be likened to the null hypothesis in statistical

The Psychology of Education Review, Vol. 38, No. 1, Spring 2014 3

© The British Psychological Society – ISSN 0262-4087
Professor Stephen Gorard
analysis, and the second to the evidence
from the achieved research sample. The
probabilistic argument is now contingent
upon the frequency with which a and B are
true together in reality, and on the accuracy
of the research finding about the likelihood
of B being true. Knowing both of these facts,
it would be possible to draw a probabilistic
conclusion about the likelihood of a being
true (the desired research conclusion). But
in reality, neither of these facts would be
known. In fact, the main supposed objective
of the analysis would be to help decide on
the accuracy of the research finding that B
may not be true. The analysis assumes from
the outset something about that which it is
supposed to be assessing. The misunder-
standing caused by this assumption is wide-
spread.
Using sampling theory techniques in
inappropriate contexts
However, the most obvious abuse of
sampling theory techniques is their use in
situations for which they were not designed
and for which they ought not to be used.
data for a population cannot have a stan-
dard error, by definition. The standard error
is defined as the standard deviation of a
random sampling distribution, of samples
drawn repeatedly from a population. It is
used (but incorrectly, see below) to try and
estimate the proximity of the sample mean
to the population mean. When working with
population data the population mean is
known, therefore, such an estimation is
neither needed nor valid. Of course, the
population data may be incomplete due to
missing cases or missing values, but this is a
cause of bias not a consequence of random
sampling variation. Bias ought to be
addressed in any analysis (although it rarely
is addressed by those who use ‘statistics’
instead) but it cannot be addressed through
significance tests and the like. None of the
techniques of sampling theory statistics can
or should be used with population data.
When commentators like Goldstein (2008,
p.396) advocate the use of confidence inter-
4 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
vals with population-based data, they are
betraying ignorance of the meaning of confi-
dence intervals (see below), misleading
policy-makers and other researchers, and
harming those who will be affected by
supposedly evidence-informed decisions.
exactly the same applies to samples other
than the random samples on which sampling
theory techniques such as significance tests
are based (Fielding & Gilbert 2000). Oppor-
tunity, convenience, snowball samples and
the like also do not have a standard error, by
definition. Findings derived from such
samples have no probabilistic uncertainty;
they will just have bias. In the same way that
findings from population data can be tenta-
tively generalised to other cases not in the
population, so findings from non-random
samples can be generalised to other cases.
But in both situations the generalisation can
only be based on judgement, and how well
the sampled cases match the non-sampled
ones in terms of what it already known. In
reality, the judgement is not a generalisation
from the sample (or population) but a deci-
sion about what is already known about
non-sampled cases and how well they match
the sampled ones. None of this concerns
random sampling variation. When
researchers like Carr and Marzouq (2012),
to take just one of many available examples,
cite significance tests and p-values derived
from two complete classes of children in one
primary school, they are making a key analyt-
ical error. Their probabilities cannot mean
anything in the context where only a conven-
ience sample of a year group from one
school is involved. even if their results had
been based on a random sample, the statis-
tical population which such results could be
generalised to does not exist outside the
sample. such abuse of statistical techniques
simply has to cease. as with Goldstein’s use
of confidence intervals for population data,
such abuse of non-random samples leads to
errors, wasted opportunities, vanishing
breakthroughs, and unwarranted conclu-
sions.

The final situation for this first kind of
abuse is when random samples are planned
but not achieved. strictly speaking an incom-
plete random sample is not a random
sample at all. rolling 1000 unbiased dice to
estimate the probability of gaining each
outcome would be a (pseudo-)random
process. rolling the dice and then re-rolling
any that showed a six would not lead to a
good estimate of the probability of gaining
each outcome. This is obvious. In the same
way, selecting 1000 cases by chance from a
known population is very different from
selecting 1000 cases and then replacing 100
of these because they refused to participate.
This means that in almost all real-life
research situations, sampling theory statis-
tical techniques are not relevant, do not
mean anything and must not be used. In a
sense, the paper could end at this point,
because it would be rare for an analyst to be
dealing with a complete random sample.
Misunderstanding and misrepresenting
the outputs of significance tests
However, there is a second kind of wide-
spread abuse of statistics that is even worse
but somewhat harder to explain. This is
because there is such a common misunder-
standing of this form of analysis. put simply,
statistical analysis even when conducted
appropriately and with all underlying
assumptions met does not do what most
analysts want and what many methods
instructors portray that it does. The nature
of the conditional probabilities involved is
commonly and mistakenly reversed, whether
through incompetence or intention to
deceive.
This confusion between the probabilities
for a sample and a population is clear in the
logic of significance testing and the quota-
tion of p-values. as with the modus tollens
argument above, a significance test assumes
from the outset that what is being ‘tested’ is
true for the population, and so calculates the
probability of obtaining a specific value from
the random sample achieved (siegel 1956).
analysts then generally mistake this
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
The widespread abuse of statistics by researchers
(pdata|Hyp) as being the probability of what
is being ‘tested’ also being true for the popu-
lation, given the value obtained from the
random sample achieved (pHyp|data).
These two probabilities are clearly very
different, and neither can be safely inferred
from the other. One may be small and the
other large, or vice versa, or any combina-
tion in between (Gorard, 2010). The p-value
calculation depends on the initial assump-
tion of a null hypothesis about what is true
for the population. as soon as it is allowed
that the null hypothesis may not be true, the
calculation goes wrong. The actual computa-
tion for a significance test involves no real
information about the population, and this
means that the same sample from two very
different populations would yield the same
p-values. a sample mean of 50 would, quite
absurdly, produce the same p-value if the
population mean were 40, 50, 60 or 70, etc.
This is because the population value is not
known (else there would be no point on
conducting the significance test), and the
entire calculation is based only on the
achieved sample value.
To illustrate the common misunder-
standing of this, consider a simplified situa-
tion. There is a bag, containing 100
well-shuffled balls of identical size, and the
balls are known to be of only two colours.
a sample of 10 balls is selected at random
from the bag. This sample contains seven
red balls and three blue balls. The analytical
question to be addressed is: how likely is it
that this observed difference in the balance
of the colours between the two samples is
also true of the original 100 balls in each
bag? The situation is clearly analogous to
many analyses reported in social science
research. The bag of balls is the population,
from which a sample is selected randomly.
a moment’s thought shows that it is not
possible to say anything very much about the
other 90 balls in the bag. The remaining 90
might all be red or all blue, or any share of
red and blue in between. Yet the purpose of
such a significance test analysis is to find out
via sampling something about the balance of
5
Professor Stephen Gorard
colours in the bag. Without knowing what is
in the bag there is no way of assessing how
improbable it is that the sample has ended
up with seven red balls. Once this impossi-
bility is realised, the pointlessness of signifi-
cance testing becomes clear.
What a significance test does instead is to
make an artificial assumption about what is
in the bag. Here the null hypothesis might
be that the bag contains 50 balls of each
colour at the outset. Knowing this it becomes
relatively easy to calculate the chances of
picking seven reds and three blue in a
random sample of 10. If this probability is
small (traditionally less than one-in-20, or
0.05) it is customary to claim that this is
evidence that the bag must have contained
an unbalanced set of balls at the outset. This
claim is obviously nonsense. The assumption
of the null hypothesis tells us nothing about
what is actually in the bag. For example,
imagine that the bag started with 80 red balls
and 20 blues. The sample is drawn as above,
and contains seven reds. The significance
test approach assumes that there are 50 reds
in the bag and calculates a probability of
getting seven in a sample of 10. This proba-
bility will be clearly incorrect because the
balls are less balanced in fact than the null
assumption requires. Now imagine that the
sample is still the same but that the bag had
80 blue balls and only 20 red originally. The
significance test approach again assumes
that there are 50 reds in each bag and calcu-
lates the same probability of getting seven
red from one and five from the other. This
probability will also be clearly incorrect
because the balls are less balanced than the
null assumption requires. More absurdly,
this second probability must be the same as
the first one since they are both calculated in
the same way on the same assumption. so
the significance test would give exactly the
same probability of having drawn seven reds
in a random sample from a bag of 80 per
cent reds as from a bag of 20 per cent reds.
This absurdity happens because the test
takes no account of the actual proportion of
each colour in the population. It cannot,
6 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
since finding out that balance is supposed to
be the purpose of the analysis.
Of course, the probability of getting seven
reds from a bag containing 80 reds is different,
a priori, to the probability of getting seven reds
from a bag containing 20 reds. But the signifi-
cance test is conducted post hoc. There is no
way of telling what the remaining population
is from the sample alone. To imagine other-
wise, would be equivalent to deciding that
rolling a three followed by a four with a die
showed that the die was biased (since the
probability of that result is only 1/36, which is
much less than five per cent, of course).
For anyone who has spotted this misun-
derstanding, there is little doubt that their
use of significance testing would cease (Falk
& Greenbaum, 1995). No one wants to know
the probabilistic answer the tests actually
provide (about the probability of the
observed data given the assumption), and
the test cannot provide the answer analysts
really want (the probability of the assump-
tion being true given the data observed).
This conclusion is not new (Harlow et al.,
1997). It has been known for a long time,
perhaps since their earliest adoption, that
significance tests do not work as hoped for,
and may well be harmful because their
results are so widely misinterpreted (Carver,
1978). Yet unwary methods resources and
purported experts continue to peddle the
fiction that p-values are, or are closely related
to, the probability of the sample result being
‘true’, real or relevant. relatively recent
examples among many include the following
in a textbook on social science methods:
[statistical significance is] ‘the likelihood
that a real difference or relationship
between two sets of data has been found’
(somekh & lewin, 2005, p.224).
and perhaps even more worrying is the
‘explanation’ (in relation to statistical
modelling) given during the training of
heavily selected UK national experts in
rigorous evaluation:
significance of b4 indicates whether there
is evidence of an interaction effect
(Connolly, 2013, slide 5).

Both of these explanations are the wrong
way around. The ‘significance’ value is really
the likelihood of finding a fake ‘difference’
or ‘effect’ if none actually exists. This is a
very different value to the likelihood of there
actually being a difference or effect. It is like
saying the probability of being a professional
footballer if a person is over six feet tall is the
same as the probability of a person being
over six feet tall if they are a footballer. The
first of these values will be much, much
smaller than the second. To confuse the two
as the supposed experts above do is to make
a very serious mistake. It is possible to
convert one figure to the other using Bayes’
theorem, as long as the unconditional prob-
abilities are already known (such as what
proportion of people are footballers and
what proportion are over six feet tall). But
there would be no point in conducting a
significance test in this situation since both
conditional probabilities would be calcu-
lated precisely.
Misunderstanding and misrepresenting
confidence intervals
Faced with increasing criticism of signifi-
cance testing and its abuse, in 1999 the
american psychological association (apa)
set up a Task Force on statistical Inference.
This considered a ban on the reporting of
such tests in all apa journals. Unfortunately,
their final recommendation fell short of
such a radical but useful step, and apa
instead focused on moving beyond signifi-
cance to a consideration of the ‘precision’ of
any research findings. Its influential publica-
tion manual now states that:
[Null hypothesis significance testing] is
but a starting point and that additional
reporting elements such as effect sizes,
confidence intervals, and extensive
description are needed (apa, 2010,
p.33).
This is a shame because confidence intervals
use the same underlying logic as significance
tests, share the same fatal flaws, and are at
least as widely misunderstood. For example,
talking about confidence intervals, Goldstein
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
The widespread abuse of statistics by researchers
(2008, p.399) says of their use in value-added
calculations:
a confidence interval provides a range of
values that, with a given probability –
typically 0.95 – is estimated to contain the
true value of the school score.
Connolly (2007, p. 149) says that a 95 per
cent confidence interval shows that:
There is a 95 per cent chance that the
true population mean is within just
under two standard errors of the same
mean.
Both of these statements are wrong. With
population data, or where the true popula-
tion value (such as its mean) was already
known, there would be no need for confi-
dence intervals (CIs). a CI is calculated only
from the sample value, and no reference at
all is made to the true population value (how
could it be?). Instead of the above, a CI for a
sample value means precisely this:
If we assume that the value from a
complete random sample is identical to
the true population value, then the CIs of
many repeated complete random
samples of the same size would contain
the population value for 95 per cent (or
selected interval) of these samples.
This is why any reported CI for a specific
sample is centred around the sample value.
Of course, based on this correct definition
(of how a CI is actually calculated) the tech-
nique is completely useless. It cannot be
used to assess how close the sample value is
to the unknown population value, because it
is based on the assumption that the two are
identical from the outset. as soon as it is
allowed that the two might differ at all, then
the calculation of the CI fails. If the sample
mean is not at the precise centre of the
normally distributed population (or
sampling distribution) then it is not true that
95 per cent of the population will lie within
1.96 standard deviations from the sample
mean. The absurdity of this kind of artificial
calculation is perhaps even clearer when
considering what happens in an example.
Imagine that a sample mean was 50, and that
this was drawn from a population with mean
7
Professor Stephen Gorard

60. The CI would have a particular range
centred around 50. Now imagine that all else
remains the same but that the population
mean was actually 70. The CI would remain
the same because the CI is unrelated to the
actual population mean. This suggests that a
CI based on an estimate of 50 for a real value
of 60 would imply the same level of accuracy
as for a real value of 70. In practice, and even
when used as intended, CIs are pointless.
Worse than this, because even purported
authorities are explaining their interpreta-
tion incorrectly, they are being used to draw
invalid inferences. again, money and
research effort are being wasted and those
intended to benefit from research may be
being harmed. simply stating the number of
cases underlying any sample value is suffi-
cient and valid.
What should happen instead?
There is a tendency to want to cling to tradi-
tional statistics, not understanding them or
even knowing that they do not makes sense,
due to not being sure what to do instead. In
general, the answer is that nothing should be
done instead. removal of the error is
improvement enough. In the paper used as
an example above (among countless others),
Carr and Marzouq (2012) presented Table 1
(p.7) as below and textual discussion of
these findings (p.6):
as seen in Table 1 children endorsed all
four of the achievement goals to similar
degrees. However, the range of responses
for both the mastery-approach and
mastery avoidance scales were narrower
than the performance scales and were
focused at the top end of the scale.
Correlations between goals (Table 1) are
consistent with the 2 x 2 framework
where goals sharing a dimension
(mastery/performance or approach/
avoidance) are positively correlated while
those not sharing a dimension are
unrelated (elliot & McGregor, 2001;
elliot & Murayama, 2008). although this
pattern of correlation is evident in this
sample the association between mastery
approach and performance approach
goals is smaller than expected, just
approaching significance.

Table 1: Descriptives and intercorrelations for achievement goal responses.

Variable M SD Min Max Range 1 2 3

1. Performance Approach 3.39 1.07 1.00 5.00 4.00 –
2. Performance Avoidance 3.58 .88 1.67 5.00 3.33 .69** –
3. Mastery Approach 4.51 .53 3.00 5.00 2.00 .20+ .10 –
4. Mastery Avoidance 3.92 .89 3.00 5.00 3.00 .13 .41** .42**
+p<0.06; *p<0.05; **p<0.001 (one-tailed)

8 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014


Clearly, much of this reporting is incorrect.
With a convenience sample of 58 children
from one school, Carr and Marzouq (2012)
should not be discussing statistical ‘signifi-
cance’ or quoting p-values. Therefore, parts
of the report such as the gobbledegook at
the foot of the table can be simply removed.
In addition, the use of decimal places should
be curtailed. It is unlikely that the reported
means are really accurate to five one thou-
sandths of a unit in a study measuring things
as vague as ‘performance approach’ with
only 58 cases. The result could look like this.
as seen in Table 1 children endorsed all
four of the achievement goals to similar
degrees. However, the range of responses
for both the mastery-approach and
mastery avoidance scales was narrower
than the performance scales and focused
at the top end of the scale. Correlations
between goals (Table 1) are consistent
with the 2 x 2 framework where
goals sharing a dimension (mastery/
performance or approach/avoidance)
are positively correlated while those not
sharing a dimension are unrelated (elliot
& McGregor, 2001; elliot & Murayama,
2008). although this pattern of
correlation is evident in this sample the
association between mastery approach
and performance approach goals is
smaller than expected.
Table 1: Descriptives and intercorrelations for achievement goal responses.
Variable M SD
1. Performance Approach 3.4 1.1
2. Performance Avoidance 3.6 0.9
3. Mastery Approach 4.5 .0.5
4. Mastery Avoidance 3.9 0.9
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
The widespread abuse of statistics by researchers
Nothing much has changed with the invalid
p-values removed. If the findings of the
paper were important (or not) before, they
remain so now that they are reported
without abusing statistics. It is entirely
possible that making the results simpler, and
not misleading readers or even the
researchers themselves with false probabili-
ties, would encourage a greater emphasis on
the analytical issues that really matter and on
the substantive (or not) nature of the results.
Key issues in this example appear to be
whether the measures are measuring
anything at all, whether they can measure it
accurately, how they could be calibrated,
what the bias might be in the sample, the
nature of any non-response, and how any of
these initial errors might propagate through
ensuing calculations. The answers to these
questions and others like them will help
readers and researchers decide whether the
results warrant the claim in the paper – that
the researchers have tested ‘the 2 x 2 achieve-
ment goal model’ (p.6). Moving away from
the convenient but invalid push-button
approach to analysis might yield benefits
beyond mere cessation of the abuse. It might
introduce more transparency and judge-
ment in reporting (Gorard, 2006).
Min Max Range 1 2 3
1.0 5.0 4.0
1.7 5.0 3.3 +7
3.0 5.0 2.0 +0.2 +0.1
3.0 5.0 3.0 +0.1 +0.4 +0.4
9
Professor Stephen Gorard
Conclusion
When an analyst is trying to decide on the
substantive importance of an apparent
research finding, they are faced with a
number of alternative explanations. If they
have used a random sample then one of
these explanations is that the result is a fluke
introduced by sampling variation. This is the
explanation that significance testing, confi-
dence intervals and associated statistical
techniques are meant to address (but which
they do not). However, this is only one expla-
nation. Other methods-based explanations
include design errors, bias in the sample,
errors in measuring or recording data,
researcher effects and so on. These other
explanations ought to be considered and
discussed whether the sample is a random
one or not, or even if a population is
involved. But the current abuse of signifi-
cance testing seems to have replaced all
other considerations. What should happen
instead of the false logic of statistics is a
greater focus on the meaning and authority
of the evidence that analysts uncover, using
transparent judgement to decide whether a
difference is worth pursuing or whether a
coefficient is worth retaining in a model.
There are a number of simple techniques
than can assist in making and portraying
these judgements, including graphical
displays, and a range of effect sizes from
odds ratios to r2.
Of course, none of the above is any kind
of argument against measurement or the
crucial role of numbers in social science
research. This paper is rather an argument
that researchers should take numbers more
seriously, and think rather more carefully
than at present about their meaning. simi-
larly, this is not an argument against the
random selection of cases in a sample, or the
random allocation of what is in effect popu-
lation data to treatment groups in a trial
10 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
design. randomisation is the best protection
against imbalance or bias in the sample or
the groupings, both in terms of known char-
acteristics that could be matched and in
terms of the unknown characteristics that
any attempted matching procedure is forced
to neglect (Gorard, 2013). But random
sampling is used to minimise bias, not so that
significance tests can be run. With a high
quality random sample the best estimate of
any equivalent value for the linked popula-
tion will be the sample value. No amount of
dredging with the sample data alone (as
happens with standard errors, significance
tests and CIs) can improve this estimate.
real people, their lives, well-being,
health and education are affected by
research evidence-informed decisions in
policy and practice. at present, these deci-
sions are (unknown to policy-makers and
practitioners) overly influenced by a super-
stitious ritual that few seem to understand
but many seem happy to follow and pass on
to new researchers. This ritual was described
by rozeboom (1997, p.335) as:
surely the most bone-headedly mis-
guided procedure ever institu-tionalised
in the rote training of science students.
as illustrated above, the techniques of
sampling theory statistics do not work as
intended, and can give misleading results
leading to vanishing breakthroughs and
even harmful interventions. It is time for this
wasteful and dangerous nonsense to cease.
Correspondence
Professor Stephen Gorard
professor of education and Well-being,
school of education,
durham University,
leazes road,
durham dH1 1Ta.
email: s.a.c.gorard@durham.ac.uk
 The widespread abuse of statistics by researchers

References
american psychological association (apa) (2010).
Publication manual of the APA (6th ed.). Wash-
ington, dC: apa.
Carr, a. & Marzouq, s. (2012). The 2 x 2 achievement
goal framework in primary school: do young
children pursue mastery-avoidance goals?.
The Psychology of Education Review, 36(2), 3–8.
Carver, r. (1978). The case against statistical signifi-
cance testing. Harvard Educational Review, 48,
378–399.
Connolly, p. (2013). Analysis of Randomised Controlled
Trials (RCTs). presentation to Conference of eeF
evaluators: Building evidence in education,
london.
Connolly, p. (2007). Quantitative data analysis in educa-
tion. New York: sage.
Falk, r. & Greenbaum, C. (1995). significance tests
die hard: The amazing persistence of a proba-
bilistic misconception. Theory and Psychology, 5,
75–98.
Fielding, J. & Gilbert, N. (2000). Understanding social
statistics. london: sage.
Goldstein, H. (2008). evidence and education policy
– some reflections and allegations. Cambridge
Journal of Education, 38(3), 393–400.
Gorard, s. (2006). Towards a judgement-based statis-
tical analysis. British Journal of Sociology of Educa-
tion, 27(1), 67–80.
Gorard, s. (2010). all evidence is equal: The flaw in
statistical reasoning. Oxford Review of Education,
36(1), 63–77.
Gorard, s. (2012). The increasing availability of
official datasets: Methods, opportunities, and
limitations for studies of education. British
Journal of Educational Studies, 60(1), 77–92.
Gorard, s. (2013). Research design: Robust approaches for
the social sciences. london: sage.
Harlow, l., Mulaik, s. & steiger, J. (1997). What if there
were no significance tests? Marwah, NJ: lawrence
erlbaum.
porter, T. (1986). The rise of statistical thinking.
princeton: princeton University press.
rozeboom, W. (1997). Good science is abductive not
hypothetico-deductive. In l. Harlow, s. Mulaik &
J. steiger (eds.), What if there were no significance
tests? Marwah, NJ: lawrence erlbaum.
siegel, s. (1956). Non-parametric statistics for the
behavioural sciences. Tokyo: McGraw Hill.
somekh, B. & lewin, C. (2005). Research methods in the
social sciences. london: sage.

Research Digest
Blogging on brain and behaviour
subscribe by rss or email
www.researchdigext.org.uk/blog
Become a fan
www.facebook.com/researchdigest
Follow the digest editor at
www.twitter.com/researchdigest

The Psychology of Education Review, Vol. 38, No. 1, Spring 2014 11

Open Dialogue peer review:
A response to Gorard
Professor Gene V. Glass
Random selection, random assignment
and Sir Ronald Fisher
N eVerY GeNeraTION, the widespread
I misunderstandings of the application of
inferential statistical techniques in psycho-
logical research must be dispelled. In the
1960s, roseboom (1960), Grant (1962), and
a few others performed the service. In 1978,
the late paul e. Meehl shined light on what
was going horribly wrong with empirical
research in the ‘soft’ social sciences, among
which he included social, clinical and educa-
tional psychology. The title of Meehl’s article
hints at many of the issues currently on the
table: ‘Theoretical risks and tabular asterisks:
sir Karl, sir ronald, and the slow progress of
soft psychology.’ as with ‘The widespread
abuse of statistics by researchers: What is the
problem and what is the ethical way
forward?,’ Meehl even opens his treatment
of the then prevalent methodological mess
with an exposition of popper’s famous modus
tollens.
stephen Gorard (2014) has performed
the function of setting straight the misappli-
cation of statistical significance testing for
this generation of educational psychologists.
I hasten to record my agreement with nearly
everything he has written on the topic. let
me briefly make one small point of elabora-
tion and two smaller points of clarification.
The fiction that probability statements
are meaningful in the absence of random
acts underlying them is preposterous. For
example: a researcher grabs the 50 closest
undergraduates available to her, asks how
many are left-handed – answer: eight – and
proceeds to place a 95 per cent confidence
interval around the proportion; conclusion:
‘The interval .08 to .29 was generated by a
process that has a 95 per cent probability of
12 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
capturing the proportion of persons in the
population who are left-handed.’ Now there
is nothing wrong with the way this finding is
reported; but the question remains, ‘What
population?’ The fact is, there is no popula-
tion to which an inference is being made.
Our researcher has made a probabilistic
statement about a non-probabilistic event.
I remember questioning my superiors –
whom I shall not name, but who were icons
in the history of educational psychology –
about this situation many decades ago.
at the time, their answer was accepted
ex cathedra: ‘The population is that popula-
tion from which this sample could have been
randomly drawn.’ Of course, now, looking
back, this answer is absurd, and only was
given to justify applying the statistical tech-
niques that seemingly aggrandised some
rather pedestrian research. If I was forced to
describe the population ‘from which this
sample could have been randomly drawn’,
I could only imagine a population that looks
like the sample itself, only much larger in
number. If non-random samples are just
replicas of the populations from which they
are drawn, then there is no need for any
kind of inference. On this, Gorard and I are
in perfect accord.
There exists one important application
of statistical tests that does not depend on
random sampling from populations. sir
ronald a. Fisher invented it and illustrated it
in his famous text The Design of Experiments
(1935). When subjects are randomly
assigned to conditions in a comparative
experiment, the eventual observed results
can be compared with what would be
expected if the random assignment had
been the sole cause of any observed differ-
ence. and this comparison can legitimately
 A response to Gorard

be stated in the form of a probability state- References

ment, for example, ‘The difference observed
would occur fewer than one time in 1000 by
mere random assignment alone.’ The
resulting test is known as a permutation test
(see ‘permutation test’ in http://
en.wikipedia.org/wiki/permutation_test).
One final point. I studied some of my
statistics under G.e.p. Box, the famous statis-
tician at the University of Wisconsin, and the
son-in-law of sir ronald a. Fisher. It was
common in those days for professors to
punish students who gave interpretations of
confidence intervals like this: ‘The proba-
bility is 95 per cent that the population
proportion of left-handed persons lies
between .08 and .29.’ Box, however, said,
relax; it makes perfect sense when under-
stood in the Bayesian sense.
Fisher, r.a. (1935). The design of experiments.
edinburgh: Oliver & Boyd.
Gorard, s. (2014). The widespread abuse of statistics
by researchers: What is the problem and what is
the ethical way forward? The Psychology of Educa-
tion Review, 38(1), 3–11.
Grant, d.a. (1962). Testing the null hypothesis and
the strategy and tactics of investigating theo-
retical models. Psychological Review, 69(1), 54–61.
Meehl, p.e. (1978). Theoretical risks and tabular
asterisks: sir Karl, sir ronald, and the slow
progress of soft psychology. Journal of Consulting
and Clinical Psychology, 46(4), 806–834.
rozeboom, W.r. (1960). The fallacy of the null-
hypothesis significance test. Psychological Bulletin,
57(4), 416–428.

Correspondence
Professor Gene V. Glass
arizona state University and
University of Colorado, Boulder.
email: glass@asu.edu

The Psychology of Education Review, Vol. 38, No. 1, Spring 2014 13

Open Dialogue peer review:
A response to Gorard
Professor Christine Howe
N WHaT, I aM sUre, is an intentionally
I iconoclastic analysis of the use of statistics
in social science research, stephen Gorard
makes five major points. First, statistical
inference is often couched in terms that
resemble modus tollens, which creates an
inappropriate aura of logic. second,
sampling theory techniques are frequently
used when working with populations, when
they are only appropriate with samples that
are drawn from populations. Third,
sampling theory techniques presuppose
random samples, and randomisation is not
always achieved. Fourth, significance tests
presume some ‘truth’ at the population level
and test the probability of obtaining
observed sample values given that truth.
However, social scientists often assume that
they can infer truths at the population level
given sample values, that is, make precisely
the reverse inference. Fifth, in many cases, it
would be preferable to publish descriptive
data only, accompanied with measures of
effect size. In the following, I shall discuss
the points, not from the perspective of social
science in general but of psychology in
particular. psychology is the discipline in
which I have worked for most of my career.
The gist of my argument is that while
stephen Gorard’s points do have relevance
for psychology and, therefore, I welcome
them, the relevance is of a lower magnitude
than his paper envisages and for somewhat
different reasons.
I have relatively little to say about the first
couple of points. statistical arguments do
indeed sometimes follow the linguistic struc-
ture of modus tollens. Moreover, there is no
doubt that, as stephen Gorard points out,
the validity of statistical arguments depends
upon contingent probabilities and not logic.
14 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
However, I am unconvinced that the distinc-
tion is seldom understood, and, therefore,
that ‘the misunderstanding caused by this
assumption is widespread’. surely the
distinction between logical and probabilistic
reasoning is generally appreciated within the
academic community. as regards the second
point, it is certainly inappropriate to use
techniques drawn from sampling theory
when working with populations, but
research based on population data is actually
quite rare within psychology. The examples
I can think of are concerned with atypical
groups where the numbers are relatively
small, for example, in forensic and clinical
contexts or relating to developmental disor-
ders. However, the vast majority of studies in
psychology are concerned with samples.
Ideally the samples used in research
should be random, and moving to stephen
Gorard’s third point it can be a cause of
concern when randomisation is not
achieved. Nevertheless, I do not agree that
sampling theory techniques should be
deemed inappropriate just because the
sample was based on opportunity, conven-
ience or snowballing. In my opinion, the
onus lies upon researchers who use such
methods of selection to demonstrate repre-
sentativeness, and to make a case for treating
their sample as if it were random, that is, to
defend the judgement that stephen Gorard
acknowledges is inevitable in such circum-
stances. In other words, my views are closer
to those of the apa task force that addressed
the issue in the 1990s: ‘Using a convenience
sample does not automatically disqualify a
study from publication, but it harms your
objectivity to try to conceal this by implying
that you used a random sample. sometimes
the case for the representativeness of a

convenience sample can be strengthened by
explicit comparison of sample characteristics
with those of a defined population’
(Wilkinson and the Task Force on statistical
Inference, 1999, p.595). If the case for repre-
sentativeness is strong, sampling theory tech-
niques may prove as useful with convenience
samples as with a truly random sample, and
like the many thousands of psychological
studies conducted in these circumstances
(for any of an enormous number of practical
reasons) the results will turn out to gener-
alise. If the case is weak, the weakness will
normally be exposed at some point along the
line, whether or not sampling theory tech-
niques were used and abused.
From a psychological perspective, the key
point in stephen Gorard’s paper is probably
the fourth one relating to significance
testing, for significance tests are undoubt-
edly pre-eminent amongst the discipline’s
nuts and bolts. Fortunately, they are not typi-
cally used in an inappropriate fashion, for
the starting point in much psychological
research is indeed the presumption of popu-
lations that do not differ, that is, the situation
amongst those depicted in the paper where
the bag of balls contains 50 red balls and 50
blue balls. When the research is concerned
with putatively universal human characteris-
tics like working memory, the populations
are the human race as it currently exists and
the human race as it might be if the charac-
teristics were different. In other words, the
populations are not necessarily real, let
alone in possession of qualities that research
aspires to document. When the focus is upon
variables like quality of parenting, teaching
or peer relations, the populations are those
who are privy to high quality experiences
and those whose experiences are not so
good. Here the bag of balls analogy may
need extending to a continuum, but the
general point continues to apply. This is that
in all cases, the null hypothesis is no differ-
ence at the population level, and that
samples are assessed to estimate the proba-
bility of their parameters given this hypoth-
esis. For this reason, generations of
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
A response to Gorard
psychology undergraduates are trained to
write laboratory reports in terms of testing
and sometimes rejecting the null hypothesis.
In other words, it is part of the discipli-
nary wisdom that, as stephen Gorard writes
‘the significance value is really the likelihood
of finding a fake difference or effect if none
actually exists’. Moreover far from ‘no-one
[wanting] to know the probabilistic answer
the tests actually provide’, such answers lie at
the heart of what psychology is about. The
reason for this apparent perversity is that as
well as testing null hypotheses, psychologists
reject these when the probability of sample
parameters is relatively small (conventionally
.05 or lower) and infer that alternative
hypotheses of population difference are
more plausible. Once more, psychology
undergraduates are taught to regard such
inferences as conjectures, not as necessary
consequences of significance testing, and to
avoid at all costs extrapolating probabilities
from those computed for the null hypoth-
esis. However, space in academic journals is
tight, and what is spelled out in student
reports typically becomes implicit in profes-
sional publications. In particular, a tradition
has developed of expressing alternative
hypotheses (e.g. as the claims to be exam-
ined), and leaving unstated the null
hypotheses that are actually tested with
sampling theory techniques. I should like to
think that most psychologists know that this
practice is only a tradition, although some
may, I guess, forget this with the passage of
time. I certainly do not believe that it results
from what stephen Gorard refers to as
‘incompetence or intention to deceive’.
Where I detect dangers is in dissemination of
psychological research to the wider public,
because in that context uncritical use of
disciplinary discourse and presumption of
shared understanding could mislead.
stephen Gorard’s point about real people
being affected by research evidence is well
taken.
In general then, I do not think there is
widespread misuse of significance testing
within psychology, but does such testing
15
Professor Christine Howe

actually add something? I am unfamiliar with Correspondence

the hapless study of achievement goals that Professor Christine Howe
stephen Gorard cites. (Why this particular professor of education,
study, I kept wondering?). Its convenience Faculty of education,
sample may or may not be legitimately University of Cambridge,
treated as if it were random. Its measures 184 Hills road,
may or may not be appropriately regarded as Cambridge CB2 8pQ.
interval scales. However, assuming the pre- email: cjh82@cam.ac.uk
requisites for correlational analysis are
fulfilled, the asterisks in the first table and Reference
the p-values at its foot strike me as more than Wilkinson, l. and the Task Force on statistical Infer-
ence (1999). statistical methods in psychology
‘gobbledegook’. rather they reflect tests that
journals: Guidelines and explanations. American
take account of sample parameters (e.g. N) Psychologist, 54, 594–604.
in a more formal and standardised fashion
than could be achieved using the second
table even if this table was accompanied with
graphical displays like scatter-plots. Of
course, like any acceptance of alternative
hypotheses, the reasoning process is inferen-
tial and provisional, and for that reason
I could not agree more with stephen
Gorard’s call for focus on ‘the meaning and
authority of the evidence analysts uncover’.
The proof of the pudding, though, is ulti-
mately in the eating. Just as I argued earlier
for sampling so I would argue for signifi-
cance testing: to the extent that null hypoth-
esis testing and conjectured alternative
hypotheses result in a body of knowledge
that advances theoretical understanding and
practical action, it can, despite undoubted
abuses, be justified. dud psychology is not
unusual and sometimes the problems lie
with misused statistics. Nevertheless, on
balance, I think the evidence is positive.

16 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014

Open Dialogue peer review:
A response to Gorard
Professor Dave Putwain
Purism over pragmatism?
TepHeN GOrard outlines in his
S paper several ways in which social scien-
tists may not be using statistical tech-
niques in the correct way(s) or way(s) in
which they were intended. By and large this
position and the sentiments behind it are to
be applauded. Would one wish to be using
analytic techniques inappropriately or
wrongly in their research? Would you wish to
inadvertently pass on the use of inappro-
priate techniques to others as part of your
teaching? even if this were the case, I am not
sure that it would strictly represent an abuse
of statistics, which, according to my
dictionary, indicates that there must be a
moral or ethical dimension. stephen’s paper
does not set out the case for why using statis-
tical techniques incorrectly is ethically or
morally dubious, although he does hint
where this could lie where, for example,
findings from wrongly-used statistical
analyses are used to inform or justify policy
decisions and poor outcomes. It would seem
more appropriate to describe the arguments
and practices set out in his paper as the
misuse, rather than abuse, of statistics.
I presume that the term abuse was used as a
rhetorical device intended to set out a
position and provoke a response, and there
is nothing wrong with that, per se. so, based
on my understanding of the thrust of the
arguments presented in the paper, I will
respond to the misuse of statistics and focus
on the use of inferential statistics with
samples that are not randomly sampled.
I have chosen to focus on this aspect in
particular, partly because some of the other
issues in the paper (e.g. misuse of confi-
dence intervals) are corollaries of this. In
addition, this is the issue (of those raised in
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
the paper) that, from my reading of the
educational psychology literature, is made
most frequently. One could pick up any
educational psychology journal and find
examples of researchers performing inferen-
tial statistical analysis on samples that were
anything other than random. To illustrate
this, a recent edition of the Journal of Educa-
tional Psychology (august, 2013), arguably one
of the most prestigious journals in the field
of educational psychology, and one which
should represent the most learned and pre-
eminent scholars, contained 24 quantitative
articles. all of them used inferential statis-
tics, yet none reported a random sample.
stephen and I would agree that it is common
practice. I think it is only fair to note that
stephen’s argument is technically correct.
The various analytic approaches used in this
edition of the Journal of Educational Psychology
included almost every analysis that one
could try: bivariate and intraclass correlation
coefficients, t-tests, analysis of variance,
single and multi-level regression models and
structural equation model. all were accom-
panied by tests of significance, which esti-
mated the likelihood of sampling error on
the basis that the sample was a random one.
With with some caveats regarding the use of
multilevel models in which data were clus-
tered, none of these studies had random
samples. although not reported in the
majority of cases, one would infer that these
were purposeful or convenience samples
and that, strictly speaking, these techniques
should, therefore, not have been used.
In the majority of cases, attaining
random samples from populations of
interest is not a realistic goal for psycho-
logical or educational researchers. either
persons or groups are unavailable or
17
Professor Dave Putwain
unwilling to participate for perfectly good
reasons, or it is simply not possible to obtain
the kinds of information about the popula-
tion required to select persons or groups at
random. Indeed, it would only require one
person from a random sample to withdraw
their participation or data and that sample is
no longer random. Hence, most researchers
use the next best available option, which is
typically a convenience sample. The
approach advocated in stephen’s paper is to
subsequently drop inferential analyses where
sampling is anything other than random.
I would like to consider the principal reason
why I believe researchers in the field of
educational psychology, and perhaps more
widely in fields of education, social science
and psychology, continue to use such
methods of analysis. Given the standard of
statistical expertise taught in undergraduate
UK psychology courses (to meet thresholds
for Bps accreditation) and on ersC-recog-
nised Master’s level courses in pre-doctoral
training, along with the explanations and
advice offered in popular textbooks which
focus on statistical analysis, I would argue
that the majority of analysts know that these
are techniques that require the use of
random samples and that their purpose is to
establish sampling error. I propose that the
continued use of inferential statistics with
non-random samples does not, necessarily,
arise from ignorance about their intended
purpose.
rather it is a pragmatic decision,
although rarely articulated, that if a non-
random sample shows broadly the same
characteristics as a random sample would,
then it is permissible to use inferential statis-
tics with the aim of attempting to establish
sampling error. For instance, if an analyst is
dealing with continuous data and expects a
random sample to be normally distributed,
but only a convenience sample is available
(as is typically the case), and if the conven-
ience sample was normally distributed
(within acceptable limits), it is typically
treated as if it were random. researchers still
wish to treat their data is if they represent
18 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
underlying population and, accordingly, to
estimate the degree of error present in their
sample parameters (means, correlation coef-
ficients, path coefficients and so on) as they
pertain to an underlying population. The
critical issue is the extent to which the
convenience sample would have resembled
the random sample, had such a sample been
available. If the random and convenience
samples are considered to be well matched,
then conclusions about the degree of
sampling error are likely to be very similar,
the substantive conclusions regarding
research findings will remain unchanged,
and one can move forward to make recom-
mendations for practice or policy without
undue concern that such recommendations
are erroneous or misplaced.
I would argue that it is not so heinous a
decision to continue using inferential statis-
tics, but that analysts should pay more
careful consideration to the sample charac-
teristics, something that stephen highlights.
The reporting of distribution characteristics
for continuous data (such as skewness and
kurtosis) appears to be becoming more
common in educational psychology journals,
but it should be essential if one is using infer-
ential statistics when, in the strictest sense,
they shouldn’t. Interestingly, in the example
cited in stephen’s paper (Carr & Marzouq,
2012), there was absolutely no difference to
the interpretation of the findings, regardless
of whether the inferential statistics accompa-
nied the interpretation of means and corre-
lations. stephen uses this to support his
argument that it is possible to interpret find-
ings without the use of inferential statistics
and I would not disagree. However, in
another in another sense it also undermines
that argument. When inferential statistics
were used, it did not result in misinterpreta-
tion and it is difficult to imagine what the
ethical and moral consequences of this
might be. so the issue, to me, boils down to
one simple point: purism or a pragmatism?
If one is a purist, these practices are not
acceptable; if a pragmatist, then they are
permissible.
 A response to Gorard

If stephen is determined to persuade the Correspondence

pragmatists that they are wrong, I don’t think Professor Dave Putwain
they will be convinced solely by a technical Centre for literacy and Numeracy research,
argument that they may already be aware of. Faculty of education,
However, they may be swayed by the possible edge Hill University,
moral or ethical implications of using infer- st. Helen’s road,
ential statistics with non-random samples. so, Ormskirk,
in addition to the technical points made in lancashire l39 4Qp.
stephen’s paper, I would like to see some email: putwaind@edgehill.ac.uk
clear examples of research or analysis where
the misplaced use of inferential statistics have
led to erroneous conclusions that have subse-
quently lead to poor changes in practice or
policy. This would considerably strengthen
the claim that there are ethical and moral
reasons for not using inferential statistics in
this way and may help convince the pragma-
tists that there are good reasons for engaging
more deeply with the technical argument.
While I applaud stephen’s aim to generate
discussion, I fear that without this additional
aspect, his argument will not have the impact
he is hoping for.

The Psychology of Education Review, Vol. 38, No. 1, Spring 2014 19

Open Dialogue peer review:
A response to Gorard
Dr Ben Styles
T Is rIGHT to question the application of
I frequentist statistics to any sample in any
field of research. When making a point
estimate of a population parameter, this
question is fundamental. If the sample were
random but not simple random then we
have to take into account the sample design
when estimating our parameter and its stan-
dard error. If the sample were one of conven-
ience or the response rate of a random
sample were low then we are not strictly justi-
fied in assigning confidence intervals to
point estimates regardless of any sophisti-
cated weighting we might carry out. When
asking whether an internally valid result
from an rCT is generalisable to a wider
population the question is the same: if an
intervention is shown to have a certain effect
size and 95 per cent confidence interval, are
we justified in claiming that, had we run the
trial many times within the population in
question, 95 per cent of the confidence
intervals produced would contain the true
effect size? Unless we had carried out
random sampling of subjects, the answer has
to be no. It is very often the case in educa-
tion research evaluations including rCTs
that the sampling was not random. even if it
purports to be a random sample of schools,
response rate is unlikely to be sufficient to
avoid bias. Gorard is, therefore, right to
question the use of frequentist confidence
intervals. However, he goes too far in advo-
cating a world where we do not attempt to
measure uncertainty and this short response
attempts to show how the use of these tradi-
tional techniques can be beneficial even for
the worst samples. The response does not
attempt to unpick the notion that frequen-
tist confidence intervals are useless even for
a completely random sample. This latter
20 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
argument may well benefit from a Bayesian
statistician’s response. Questioning how
uncertainty is reported is sensible; arguing
for its removal from reporting is against a
fundamental tenet of research itself.
I see two arguments that justify the reten-
tion of frequentist confidence intervals in
certain scenarios when sampling was not
random. The first concerns the generalisa-
tion of effect size and is more of an intuitive
argument than a statistical one. The second
concerns the concept of a virtual population
and is more justifiable.
Generalisation of effect size
I believe it is wise, up to a point, to operate
less stringent sampling protocols for the esti-
mate of differences in outcomes between
groups than is necessary for point estimates.
such flexibility in education research is
needed due to the consistent difficulty of
recruiting schools. The groups might be
randomised arms of a trial or test-takers
whose results on two or more tests are being
equated. What matters in these scenarios is
that the relative performance of the groups
can be compared. providing the allocation
to trial arms (or tests) was at random and
attrition was minimal, we have no worries
about bias; what is at stake here is how we
report uncertainty. Can we be more justified
in generalising a group difference from a
non-random sample than a population
parameter point estimate? strictly speaking,
we cannot generalise in either scenario.
However, in practice, what we are often
surmising in education research when we
create confidence intervals around an effect
size is that the difference in performance
between groups of students in a trial whose
schools have not been randomly sampled

might be representative of students else-
where in the country in similar schools;
should they receive the same intervention.
Intuitively this appears less of a leap of faith
than claiming that the actual performance,
an example of a population parameter, lies
between its confidence intervals (the respon-
dent is aware of the simplification inherent
in ‘lies between’). The main reason why such
generalisation may be reasonable is that it is
for the same intervention on the same
outcome rather than the estimation of a
measure that might be influenced by a
myriad of other factors.
Virtual population
The previous argument is definitely ques-
tionable since the effect itself may also be
vulnerable to this myriad of other factors.
However, it might be strong enough to justify
another trial or even intervention roll-out if
the sample is seen to be representative
‘enough’. From a frequentist point of view,
when analysing the results of any trial, we
need to establish how easily the results we
see could have occurred by chance alone;
the basis of a frequentist statistical test. simi-
larly, it is useful to estimate a confidence
interval; thus encapsulating the chance
element of the effect size we see. If we reject
any generalisation to a wider population,
Gorard’s conundrum comes into sharp
focus: if this were not a random sample,
where are the other subjects from which we
could have sampled that might give rise to
the other results upon which our confidence
interval is based? They certainly do not exist
physically. However, it is often helpful to
regard them as existing virtually. The
concept of a virtual population is often used
without acknowledgement, for example,
when assigning a confidence interval to a
school’s value-added results even when all
students are measured (Goldstein, 2008).
rather than conceptualising the 95 per cent
confidence interval as one of many from a
large series of trials run on members of a
physical population, 95 per cent of which
would contain the true effect; we imagine
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
A response to Gorard
the trial being run many times on students in
the same schools at the same time in a virtual
population from which we did sample
randomly. This allows us to quantify chance
and gives meaning to the p-values and confi-
dence intervals used. Whilst the concept is
abstract, ignoring uncertainty is far worse
and may result in concluding that things
work when they do not and vice versa; even if
this is just for the sample in question.
By introducing the concept of a virtual
population, we are acknowledging that the
students in the trial can be regarded as a
physical ‘population’. They were not
randomly sampled so no further physical
population exists to which confidence inter-
vals can apply strictly. Gorard states that ‘data
for a [physical] population cannot have a
standard error, by definition.’ Indeed, if we
are estimating a population parameter and
have measured everyone in the population,
we need no standard error. However, things
are rarely so straightforward since the popu-
lation itself may be limited in terms of the
research question and may need to be seen
as a sample within a larger virtual population
as illustrated in the previous paragraph.
Merely reporting effect sizes and
numbers of participants would engender a
culture of conjecture around the uncertainty
of any result. Bayesian statistics has a lot to
offer the concerns raised by Gorard and,
rather than not attempting to measure
uncertainty, all users of statistics should
embrace the frequentist versus Bayesian
debate more seriously.
Correspondence
Dr Ben Styles
research director, National Foundation
for educational research,
The Mere Upton park, slough.
Berkshire sl1 2dQ.
email: b.styles@nfer.ac.uk
Reference
Goldstein, H. (2008). evidence and education policy
– some reflections and allegations. Cambridge
Journal of Education, 38(3), 393–400.
21
Open Dialogue peer review:
A response to Gorard
Professor Victor H.P. van Daal & Dr Herman J. Adèr
Gorard makes two claims: (1) there is a widespread abuse of statistics in social sciences; and (2) researchers
almost universally report results incorrectly. According to Gorard the way forward is to correct the reporting,
so that the abuse will disappear.
e HaVe a NUMBer OF concerns
W with this paper, which are listed
below.
1. Gorard does not provide any hard
data – and here we don’t need any inferen-
tial statistics – on how widespread the abuse
of statistics in the social sciences is, nor does
he inform us of how ‘universal’ the poor
reporting is. so, we don’t get any idea at all
about how widespread the abuse of statistics
and how universal the poor reporting of
results are. Furthermore, even if statistical
methods would be abused and reporting
would be poor it remains unclear in what
way and to what extent this invalidates the
interpretation of published research results.
Finally, is there any evidence that statistics
abuse and/or faulty reporting necessarily
leads to bad decisions by policy makers or
other stakeholders? In our opinion, wrong
decisions are caused by not being able to
assess the methodological quality of any
research and by not being able to assess
which pieces of research are relevant for the
decision to be taken.
2. The take on probabilistic reasoning
presented in the beginning of the paper is
not convincing. The consequence would be
that, for example, weather forecasts should
not be trusted, because they are based on
probabilistic reasoning. For an in-depth
discussion of probabilistic reasoning, see
pearl (2000). Morgan and Winship (2007)
provide a more accessible treatment, while
shadish, Cook and Campbell (2002) focus
on causal inferential reasoning in social
science research.
22 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
3. Unlike in physics where truly random
samples are used, convenience samples
dominate in the social sciences. The main
problem with convenience samples concerns
generalisation. What is found in a conven-
ience sample cannot be generalised to the
population from which samples are drawn.
In the field of medical research often a
so-called rCT design (randomised clinical
trial) is used, in which different treatments
(usually two) are randomly assigned to
different patients, so that alternative expla-
nations for any treatment effect found can
be ruled out. However, random assignment
can still go wrong, though statistical tech-
niques are available to fix such problems
(Van renswoude, 2013). statistical tech-
niques can and should not be abandoned,
because hypothesis testing and effect size
estimation are essential for the interpreta-
tion of any rCT. Generalisation in medical
research is achieved by conducting meta-
analyses, in which studies relevant for a
specific topic are systematically combined.
4. Gorard seems to join others, who time
and again discuss the inadequacy of statis-
tical reasoning. However, this is not very
helpful for researchers. Instead, researchers
should systematically be taught to scrutinise
their conclusions in view of the limitations of
the statistical techniques used, so-called
content robustness (adèr, 2008), and, equally
important, in view of the influence of poten-
tial confounders in their designs. In general:
a researcher should take a methodological
point of view rather than a statistical one.
 A response to Gorard

5. another perspective of quantitative References

data analysis is completely ignored in
Gorard’s paper: the difference between
exploratory and confirmatory data analysis:
his discussion is confined to the second form
of analysis, whereas most data in the social
sciences are of explorative nature. every
researcher in the field is aware of this differ-
ence and of the different statistical tech-
niques that should be applied.
6. Finally, even in an explorative data
analysis, statistical inference can be applied
by using cross-validation. a relative large
data set (though this requirement can be
weakened) is randomly split in two parts.
One part is used to describe the sample and
the data, and to generate hypotheses. Corre-
sponding statistical hypotheses are then
tested on the other part of the data set.
adèr, H.J. (2008). Methodological quality. In H.J.
adèr & G.J. Mellenbergh (eds.), Advising on
research methods: A consultant’s companion
(pp.49–70). Huizen, The Netherlands: Johannes
van Kessel publishing.
Morgan, s.l. & Winship, C. (2007). Counterfactuals
and causal inference: Methods and principles for social
research. New York: Cambridge University press.
pearl, J. (2000). Causality: Models, reasoning, and
inference. Cambridge, Ma: Cambridge University
press.
shadish, W.r., Cook, d.T. & Campbell, d.T. (2002).
Experimental and quasi-experimental designs for
generalised causal inference. New York: Houghton
Mifflin.
Van renswoude, d.r. (2013). random or non-
random assignment: What difference does it
make? In H.J. adèr & G.J. Mellenbergh (eds.),
Advising on research methods: Selected topics 2013.
Huizen, The Netherlands: Johannes van Kessel
publishing.

The Authors
Professor Victor H.P. van Daal
edge Hill University.
Dr Herman J. Adèr
Johannes van Kessel advising.

Correspondence
Professor Victor H.P. van Daal
director,
The Centre for literacy and
Numeracy research,
Faculty of education,
edge Hill University,
st. Helen’s road, Ormskirk,
lancashire, l39 4Qp.
email: vandaalv@edgehill.ac.uk

The Psychology of Education Review, Vol. 38, No. 1, Spring 2014 23

Open Dialogue peer review:
A response to Gorard
Dr Patrick White
Orard’s paper is a welcome
G reminder of the problems that under-
mine the value of some of the most
commonly-used statistical analyses. His
paper concentrates on two important
problems inherent in the use of inferential
statistics in social science that, while purport-
edly widely recognised, have not led to any
dramatic decline in the use of these tech-
niques. as Gorard acknowledges, the points
he makes are by no means new; they have
been regularly raised by concerned
commentators and date back to the very
early use of this type of statistical analysis.
However, these views have not always been
expressed as clearly or forcefully, and Gorard
illustrates his points using the type of illus-
trative examples seldom seen in other
commentaries. While I do not expect his
paper to be warmly welcomed by many of the
respondents in this issue, the points he raises
are important ones that cannot be ignored
by a responsible research community.
Two objections to the use of inferential
statistics in social science
The two key points that Gorard raises are as
follows:
1. Most social science data do not meet the
assumptions required for the use of
inferential statistics. despite this, these
techniques are widely used regardless of
whether these assumptions are met.
2. p(data|hypothesis) ≠ p(hypothesis|data).
This problem undermines the use not
only of tests of statistical significance that
produce p-values but also has implications
for the correct interpretation of
confidence intervals, which have been
promoted as a more useful alternative.
24 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
Neither of these problems can be easily over-
come by any amount of statistical or philo-
sophical manoeuvring. The history of debate
in this area suggests, however, that when
these issues are raised there is no shortage of
advocates ready to defend the continued use
of inferential statistics, even in cases where
the underlying assumptions are not met (for
examples see some of the contributions to
Morrison and Henkel (1970) and Harlow et
al. (1997)). The arguments presented in
favour of their continued use are often
convoluted and sometimes display an almost
religious attachment to these techniques
that has led some commentators to suggest
that users of inferential statistical are akin to
some kind of ‘cult’ (Ziliak & McCloskey
2008). In this response I will expand on the
points made by Gorard with reference to the
wider literature on the topic and my experi-
ence of how this debate impacts on the
process of peer review.
Meeting the assumptions for the use of
inferential statistical techniques
as Gorard correctly points out, in social
science research data rarely (if ever) meet
the assumptions underlying the use of infer-
ential statistical techniques. The difficulty of
achieving true random samples and full
response (both at the case and variable
levels) mean that these assumptions are
unrealistic in the context of most social
science research. This does not prevent
purported ‘experts’ ignoring these assump-
tions, however, and defending the use of
inferential statistics with inappropriate data.
a high profile example of this is Hans
rosling, hailed by some as a key player in
stimulating public interest in statistics and
increasing statistical literacy. Within the first

few minutes of his much-lauded The Joy of
Stats documentary he calculates confidence
intervals for data that have not been sampled
randomly (rosling 2013). With such high
profile examples of misuse, it is perhaps not
surprising that flouting the assumptions
required for these statistics is routine.
In social science research, using inferen-
tial statistics regardless of having non-
random, incomplete or even population
data is commonplace. One only has to
glance at recent publications in highly-
ranked social science journals to see that
there is little in the way of quality control
exercised by reviewers or editors in relation
to the incorrect use of these techniques.
This situation would be sufficiently
worrying if it stopped there. However, the
effects of the widespread acceptance of the
inappropriate use of inferential statistics are
not limited to the publication of articles by
authors who, for whatever reason, are erro-
neously using these techniques. My experi-
ence – and that of close colleagues – suggests
that the common and accepted abuse of
these techniques leads to a further problem.
authors who have quite properly eschewed
the use of inferential statistics because their
data do not meet the required assumptions
are often asked by reviewers to include the
results of these tests in their research reports
in order to have them accepted for publica-
tion. I have come across this situation in
papers I have authored or co-authored and
also in reports from colleagues who have
received these kinds of recommendations
from reviewers. Mostly commonly, however,
I have seen it in the comments from other
reviewers on many occasions when I have
acted as a referee. a related situation also
frequently occurs when, as a referee,
I recommend that the inappropriate use of
inferential statistics be removed from a
paper before publication.
What is notable in the situations
described above is the fervour displayed by
advocates of the inappropriate use of infer-
ential statistics, both when defending their
own practice and insisting that others follow
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
A response to Gorard
suit. also interesting are the arguments used
to justify their position. In my experience the
most common defence involves an appeal to
the idea of a ‘superpopulation’.
The myth of the ‘superpopulation’
In educational research it is not uncommon
for research to use population data. The
populations used can range from the quite
modest, such as data on all students in a
single school, to those such as the National
pupil database (Npd) that include every
student enrolled in a state school in
england. What these data sets have in
common is that they have not been gener-
ated through processes of random sampling
but cover all the cases in a particular institu-
tion, geographical area, and so on. It is quite
common for analyses of these population
data sets to include the use of inferential
statistics. as Gorard points out, the results of
such analyses are meaningless. With popula-
tion data there is no need for any inference
as any analyses are conducted at the popula-
tion level. any errors in the results will not
be due to random sampling and so, in any
case, inferential statistics should not be used.
as Berk (2004, p.42) concludes:
If the data are a population, there is no
sampling, no uncertainty because of
sampling, and no need for statistical
inference. Indeed, statistical inference
makes no sense. The only game is
describing patterns in the data on hand.
Unlike in some situations in the physical
sciences, in the social sciences other defi-
ciencies in the data (such as drop-out, non-
response or measurement error) cannot be
assumed to be random. The problem of bias
caused by non-response has been acknowl-
edged for many years (see Hansen &
Hurwitz 1946) and repeated studies have
demonstrated the non-random nature of
non-response (e.g. sheikh & Mattingly
1981).
as mentioned above, a common defence
of the use of inferential statistics with popu-
lation data appeals to the idea of a ‘super-
population’, ‘hyperpopulation’, ‘infinite
25
Dr Patrick White
population’ or ‘hypothetical universe’. This
argument is also used in cases where conven-
ience sampling has been used, when non-
response rates are very high, or in other
cases where a sample is clearly of a non-
random nature.
relating such a concept to statistical
inference dates back to at least 1941 with the
publication of an article by Hagood (1970)
and the idea has more recently been cham-
pioned by Goldstein (2003) among others.
Hagood (1970, p.66) defines this phenom-
enon as ‘the universe of all possible samples
(which may be limited universes) which
could have been produced under similar
conditions of time, place, culture, and other
relevant factors’ (p.66). advocates of using
inferential statistics with population data or
non-random samples argue that we are ulti-
mately interested in ‘universals’, rather than
the actual situation at hand, and we can use
inferential statistics to generalise from popu-
lation data to these wider truths. Goldstein
(2003, p.164, original emphasis) recom-
mends considering ‘the actual population
as if it was a realisation of a conceptually infi-
nite population extending through time,
and also possibly through space’. This, he
claims, enables the researcher to ‘make
generalisations and predictions beyond the
units that comprise the real population’.
We would probably all agree that discov-
ering ‘universal’ truths that transcend time,
place (or space) and culture is desirable.
agreeing with this aim, however, is very
different from agreeing that it is possible.
and it is different again from agreeing that it
is possible to do this using population data
(or other data not selected randomly) and
inferential statistics.
as Berk (2004) notes, the first problem
with the superpopulation approach is that
these superpopulations are imaginary and
do not actually exist. The second issue is that
even if we could agree that such ‘superpopu-
lations’ exist, data used in any existing
research project cannot have been randomly
selected from any of these ‘superpopula-
tions’. as Morrison and Henkel (1970,
26 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
p.306) point out, there is no ‘empirically
demonstrable random generation proce-
dure’ and so no statistical inference is
possible. In any case, it is difficult to know
what any probabilities or other inferential
outputs would actually refer to as they ‘do
not directly answer any empirical question’
(Berk, 2004, p.51). These difficulties,
however, have not stopped many researchers
appealing to this idea. But as Freedman
(2004, p.989) argues, ‘the frequency with
which the assumption has been made in the
past does not provide any justification for
making it again, and neither does the
grandiloquent name… the problem of
induction is unlikely to be solved by fiat’.
Berk (2004, p.52) is similarly dismissive of
these arguments and the lack of critical eval-
uation of the assumptions that underlie
them: ‘It is as if by mouthing the term super-
population, a spell is cast making all statistical
inference legitimate’.
appeals to ‘superpopulations’ are philo-
sophical in nature rather than statistical.
My experience of explaining this concept to
colleagues with little expertise in statistics
usually results in gasps of disbelief that such
a concept could be taken seriously at all.
That the idea is fundamentally flawed has
not prevented attempts to use it to defend
the inappropriate use of inferential statistical
techniques among educational and social
researchers.
Does any of the above matter?
The ‘appropriate’ use of inferential statistics
is the first point addressed by Gorard in his
paper. Given that social science data are
unlikely to meet the required assumptions,
Gorard suggests that the matter could end
there, with the conclusion that inferential
statistics are not suited to social science. His
second point, however, suggests that the
issue of the appropriate use of inferential
statistical techniques is actually a red
herring. rather than the debate beginning
and ending with this conclusion, it is actually
the case that we do not need to scrutinise the
extent to which social science data meet

these assumptions. even if it can be demon-
strated that data could or do meet the neces-
sary requirements, the outputs of inferential
statistics do not tell us anything we want to
know. put simply, they are not useful even in
the (hypothetical?) cases where the under-
lying assumptions for their use are met.
P-values are not the only problem.
One of the elements that sets Gorard’s paper
apart from most critiques of the use of infer-
ential statistics is that his concern is not
limited to the use of null-hypothesis statis-
tical testing (NHsT) and the associated
reliance on p-values. Criticisms of the use of
p-values are relatively common but often
advocate a change of focus not only to effect
sizes (which are not an inferential tech-
nique) but also to the use of standard errors
(ses) and confidence intervals (CIs) (e.g.
Cumming, 2012; Hubbary & lindsay, 2008;
lambdin, 2012). even rozeboom (1960)
and Cohen (1994), both who have written
about the logical problems with NHsT and
are ardent critics of the use of p-values, view
CIs as an acceptable alternative. This view is
also shared by Meehl (1997), who has
written extensively on the problems of
NHsTs in psychology.
While discussion of other issues can be
found in the literature, the problem central
to the use of p-values is that the probability
that they refer to is not the probability we
want to know. P-values can only provide the
probability of the data given a hypothesis.
What we actually want to know is the proba-
bility of a hypothesis given the data. as
Gorard shows in his paper, the former prob-
ability – p(d|H) – is of no use to researchers
and it is not possible to convert this informa-
tion to the more useful latter probability –
p(H|d) – using only sample data. This
problem has led to the commentators
mentioned above advocating the use of CIs
as an alternative to p-values.
However, as Gorard shows, the advan-
tages of using ses and CIs in place of, or in
addition to, p-values, are illusionary. Once
the correct definition of a confidence
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
A response to Gorard
interval is adopted, it becomes unclear how
such information would be of use to a
researcher. as with NHsT and p-values, the
starting point is still based on assumptions
about the population that are unverified and
cannot be tested using sample data. The
logical problem that renders NHsT and
p-values useless also prevents the informa-
tion provided by CIs being useful. Those
who correctly criticise the use of p-values on
these grounds often seem to miss the point
that all inferential statistical techniques
suffer from a similar flaw.
apart from Gorard, there are very few
commentators who are willing to abandon
the ‘project’ of inferential statistics alto-
gether. Concerns are expressed that to
abandon inferential statistics as a ‘bad job’
would be akin to ‘throwing the baby out with
the bath water’. It is now relatively uncontro-
versial for researchers to express concern
about the use of NHsT and p-values, and
these concerns are even beginning to be
raised in texts aimed at undergraduate
students (e.g. Field 2009). recommending
abandoning the use of all inferential statis-
tical techniques, however, is much less
common and likely to generate considerably
more criticism.
However, what is missing from the more
cautious interrogations of the use of inferen-
tial statistics is a convincing account of why
we should continue to use any of them at all.
Those who advocate a move to CIs do not
adequately explain how the information
provided by these measures can be useful
when they are interpreted correctly. My current
view is that no such account will be forth-
coming simply because it is not possible to
construct one.
Against inferential statistics
as I stated at the beginning of this response,
I do not expect Gorard’s views to be popular
– especially those that extend the traditional
critique of NHsT and p-values to other infer-
ential outputs such as CIs. I expect my
support of his views to receive similar reac-
tions. However, I believe these arguments
27
Dr Patrick White
need to be repeated if any progress is to be
made in changing the way we conduct and
teach statistical analysis. We need to
abandon these practices that are at best
wasteful and at worst harmful. data in social
science does not meet the assumptions
required for inferential statistics and, in any
case, the outputs produced by these tech-
niques are not useful. It’s time to move on.
References
Berk, r. (2004). Regression Analysis: A constructive
critique. Thousand Oaks, Ca: sage.
Cohen, J. (1994). ‘The earth Is round (p<.05)’.
American Psychologist, 49(12), 997–1003.
Cumming, G. (2012). Understanding the new statistics:
Effect sizes, confidence intervals and meta-analysis.
New York: routledge.
Field, a. (2009). Discovering statistics using IBM SPSS
Statistics. london: sage.
Freedman, d.a. (2004). sampling. In M.s. lewis-
Beck, a. Bryman & T.F. liao (eds.), Sage
encyclopaedia of social science research methods
(pp.987–991). Thousand Oaks, Ca: sage.
Goldstein, H. (2003). Multilevel statistical models
(3rd ed.) london: arnold.
Hagood, M.J. (1970). The notion of a hypothetical
universe. In d.e. Morrison & r.e. Henkel (eds.),
The significance test controversy. Chicago: aldine.
Hansen, M.H. & Hurwitz, W.N. (1946). The problem
of non-response in sample surveys. Journal of the
American Statistical Association, 41(236), 517–529.
Harlow, l.l., Mulaik, s.a. & steiger, J.H. (eds.)
(1997). What if there were no significance tests?
Mahwah, NJ: lawrence erlbaum.
Hubbard, r. & lindsay, r.M. (2008). Why p-values
are not a useful measure of evidence in statistical
significance testing. Theory & Psychology, 18(1),
69–88.
28 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
Correspondence
Dr Patrick White
senior lecturer,
department of sociology,
University of leicester,
University road,
leicester le1 7rH.
email: patrick.white@le.ac.uk
lambdin, C. (2012). significance tests as sorcery:
science is emprical – significance tests are not.
Theory Psychology, 22(1), 67–90.
Meehl, p.e. (1997). The problem is epistemology, not
statistics: replace significance tests by confidence
intervals and quantify accuracy of risk numerical
predications. In l.l. Harlow, s.a. Mulaik & J.H.
steiger (eds.), What if there were no significance
tests? Mahwah, NJ: lawrence erlbaum.
Morrison, d.e. & Henkel, r.e. (eds.) (1970).
The significance test controversy. Chicago: aldine.
rosling, H. (2013). The Joy of Stats. BBC 4.
16 October. accessed 25 January 2014, from:
www.bbc.co.uk/programmes/b00wgq0l
rozeboom, W.W. (1960). The fallacy of the null
hypothesis significance test. Psychological Bulletin,
57, 416–428.
sheikh, K. & Mattingly, s. (1981). Investigating non-
response bias in mail surveys. Journal of Epidemi-
ology and Community Health, 35, 293–296.
Ziliak, s.T. & McCloskey, d.N. (2008). The cult of statis-
tical significance: How the standard error cost us jobs,
justice and lives. ann arbor, MI: University of
Michigan press.
Open Dialogue:
Authors’ response to peer commentary
Professor Stephen Gorard
Perpetuating the ‘preposterous’
Y BrIeF paper on the abuse of
M statistics (‘abuse’ being a synonym
for ‘misuse’ in the Oxford English
Dictionary, and explained as meaning ‘to use
mistakenly or for a bad purpose’) argued
three main points.
1. Once random sampling variation is
eliminated as a possible explanation for
any apparent finding, analysts need to
focus on alternative explanations based
on design, measurement, bias, and
attrition rather more than they do now.
2. The widespread use of probability
calculations such as significance tests for
eliminating random sampling variation
as a possible explanation is based on a
misunderstanding.
3. The kinds of probability calculations
involved in significance tests and
confidence intervals cannot be used with
non-random cases, such as convenience
samples and population data, anyway.
It is heartening to read support for these
ideas in the responses from such a high-
profile influence in the area (Gene Glass,
from whom I have borrowed the term
‘preposterous’) and an emerging influence
on methods in social science (patrick White,
the only respondent to engage fully with the
second point above). Their response papers
take these points forward and provide
elegant examples with both clarity and some
subtle humour. Yet, in a sense I cannot
understand why everyone does not compre-
hend and support these three relatively
simple points. They were not made by me in
a spirit of ‘purism’ (dave putwain), or of
‘iconoclasm’ (Christine Howe). These three
points are simply the truth of the matter.
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
© The British Psychological Society – ISSN 0262-4087
They have long been recognised as the truth
by some before us, and papers and books
have even been written to try and explain the
political, financial and career reasons why
others continue to ignore their obvious
truth.
A few important misunderstandings
It is intriguing that the four respondents
who wish to retain the use of significance,
etc., all focus on the third point above,
because this is surely the hardest to defend
against. The computation of significance,
standard errors and confidence intervals
(and the associated algorithms, now hidden
by software) are clearly predicated on true
random sampling. The fact that the software
still operates and provides an ‘answer’ when
the data does not come from randomisation
and contains no probabilistic uncertainty is
merely an illustration of the well-known
garbage-in garbage-out principle. I guess
that for these respondents to accept the
truth (and in clearer terms than putwain’s
‘technically correct’, and the double-nega-
tive ‘would not disagree’, or Ben styles’ ‘not
strictly justified’, for example) would mean
the end of statistics as it is practiced, the end
of purported expertise in these methods,
and the casting of doubt over prior work.
Howe states resolutely ‘I do not agree
that sampling theory techniques should be
deemed inappropriate just because the
sample was based on opportunity, conven-
ience or snowballing’. To some extent this is
merely a description of current practice and
so many researchers must agree with Howe.
But to see it so blatantly in print like that is
truly shocking. It is the exact opposite of
29
Professor Stephen Gorard
everything that appears in reputable
methods texts, and it is the kind of error that
leads Glass to say ‘The fiction that proba-
bility statements are meaningful in the
absence of random acts underlying them is
preposterous’.
Common responses when I write or
lecture about the abuse of statistics are
‘everyone does it’, ‘it has happened for a
long time’ and eventually ‘we already know
all this but what should we do instead?’.
I hope readers can see that none of these is
a valid counter-argument, and that they
remain invalid when deployed by four
respondents here who largely re-state their
own existing practice. styles claims that in
rejecting the use of significance and CIs I am
rejecting any attempt to consider uncer-
tainty in research findings. This is not true,
and the original paper urges researchers to
consider a wider and more important range
of factors that lead to uncertainty but which
are ignored by the significance approach
(such as design bias or respondent attrition).
I feel I am the more concerned because I do
not just want to pretend I am assessing
uncertainty via an invalid technique.
Howe claims that I argued that we should
not use convenience samples and quotes
apa guidance suggesting that convenience
samples are perfectly proper. They are, and
I never suggested otherwise. In fact, I clearly
stated that we often have no practical alter-
native. What apa does not say is that we
should use significance tests with conven-
ience samples. as ever, it is presumably easier
to mis-portray what I said and argue with
that. Van daal and adèr do something
similar. I showed that denying the conse-
quent is only valid in logic when the prem-
ises are certain, and that the modus tollens
argument fails once any premise is uncertain
or probabilistic. They portray this as me
saying that probabilistic argument in general
is invalid, including weather forecasting as
an illustration. But weather forecasting does
not employ this ‘denying the consequent’
argument structure at all. These three exam-
ples show the lengths that commentators
30 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
have to go to in order to try and defend the
indefensible.
There are some less common variants in
the responses that try to maintain the edifice
of significance testing. If we have a non-
random sample we could randomly sample
from within that and then use significance
with the sub-sample (Van daal & adèr). This
seems truly desperate. purportedly, there are
techniques to ‘fix’ a non-random sample and
make it back into a random one (Van daal &
adèr). No there are not, because if we knew
the key values for the missing cases then we
would have a complete sample. If not, we can
only use the values we do have to make up
for what is missing, so enhancing the bias
caused by the missing cases in the first place
(Gorard, 2013). The same practical problem
eliminates putwain’s suggestion that if the
achieved sample looks similar to what the
random sample would have been if available,
then using significance is justified. To
imagine a random sample based on a
convenience sample, and then try to
compute real probabilities accurately based
on that imagination is surely incorrect.
even stranger is the notion that the
super-, hyper- or virtual-population invented
to help differentiate between theoretically
finite and infinite populations can then be
used to justify treating actual population
data as a random sample (styles). Just
envisage what styles means when he writes
‘we imagine the trial being run many times
on students in the same schools at the same
time in a virtual population from which we
did sample randomly’. and note that this
entirely ignores the logical problems raised
by point two at the outset. I have written
about this absurdity many times before (e.g.
Gorard, 2008), and White handles this
briefly but well in his response.
The search for an alternative or what to
do ‘instead’ of significance, especially with
non-random samples/allocation, is an odd
one. since the existing approaches do not
work we must abandon them. a Bayesian
approach would certainly be more logical
but is no panacea and no substitute for

judgement. as my original paper outlined,
we do not need alternatives as such since we
should be considering all competing
substantive, design and methods explana-
tion for any apparent finding anyway (even if
we want to eliminate chance first). But signif-
icance testing has somehow come to replace
such real analysis, perhaps because the latter
is not push-button. Nowhere is this more
apparent than in the Carr and Marzouq
(2012) paper cited in my original paper.
I would have thought that my views on that
paper were plain, and that once the
goobledegook is removed nothing of scien-
tific value is left. But I do not intend to
pursue this further. That paper was merely
chosen as a recent example to represent
hundreds similar in this journal and the
many thousands in journals worldwide.
The abuse of significance is a
big problem
I did not, in my brief article, explain just how
widespread the abuse of sampling theory
with non-random samples is, nor how often
the results of statistical analyses are poorly
reported. I assumed that respondents like
Van daal and adèr would know (and realise
the damage that ensues). They do tell
readers that ‘convenience samples dominate
in the social sciences’ – which surely means
that significance tests should hardly ever be
encountered, as their computation depends
entirely on prior randomisation. However,
these tests are still widely encountered, as
Van daal and adèr should and probably do
know, and as putwain helpfully illustrates via
consideration of articles in the Journal of
Educational Psychology. None of the 24 articles
involving numbers was based on random
samples, and all of them used inferential
statistics (incorrectly). This does not surprise
me and I have done similar analyses of
education journals and found the same
thing (e.g. Gorard, 2008). The problem is so
widespread that it is almost universal, and
perhaps in some strange way that makes it
hard for Van daal and adèr to notice. simi-
larly, although the meaning of significance
The Psychology of Education Review, Vol. 38, No. 1, Spring 2014
Authors’ response to peer commentary
tests is often correctly described in methods
texts, these tests are then generally misused
in the examples and in research practice.
The error is now usually only implicit since
analysts simply take the probability of the
data observed given the truth of the null
hypothesis as being the same as or closely
related to the probability of the null hypoth-
esis being true given the data observed. That
is, they ‘reject’ the null hypothesis where the
probability of the data observed under that
hypothesis is low, and they do so without
explanation or justified argument.
This key error of confusing pHyp|data
with pdata|Hyp is again almost universal.
That is why I proposed asking all researchers
to explain the steps in the ‘logic’ they are
using explicitly, since once they have written
the argument down in clear (if they can) it
should be obvious to them and their readers
that the argument is invalid (see the worked
example in Chapter 5 of Gorard, 2013).
significance tests just do not work as
intended by their users, even when applied
to random sampling/allocation. The situa-
tion is even worse for confidence intervals
because I have yet to encounter an analyst
who can correctly explain what CIs mean.
CIs are as bad as p-values since they ‘suffer
from similar flaws to p-values, exaggerating
both the size of implausible effects and their
significance’ (Matthews, 1998, p.5), yet they
are even harder to describe. and why should
the fact that 95 per cent of hypothetical
repeated sample figures lie within 1.96 popu-
lation standard deviations of the population
average then imply that the population
average will lie within 1.96 one-sample stan-
dard deviations of a specific one-sample
average 95 per cent of the time? It will not
and it does not, as anyone can see if they
think about it clearly. please let us not worry
about whether it is a ‘standard deviation’ or
a ‘standard error’, a ‘population’ or a
‘sampling distribution’. let the reader insert
their terms of choice, and the argument of
CIs remains just as clearly nonsense.
These problems are not just common-
place, they are also dangerous. at best, they
31
Professor Stephen Gorard

make research reports harder to read, References

perhaps confusing readers, and certainly Gorard, s. (2002). ethics and equity: pursuing the
perspective of non-participants. Social Research
wasting people’s time with producing,
Update, 39, 1–4.
publishing and consuming fake results. In a Gorard, s. (2008). Quantitative research in education:
worse case, they waste public funding, Volumes 1 to 3. london: sage.
causing needless opportunity costs, and they Gorard, s. (2010). all evidence is equal: The flaw in
waste people’s energy pursuing what turn statistical reasoning. Oxford Review of Education,
36(1), 63–77.
out to be all too easily predictable vanishing
Gorard, s. (2013). Research design: Robust approaches for
breakthroughs. at worst, these errors the social sciences. london: sage.
damage lives and kill people (see examples Matthews, r. (1998). Bayesian critique of statistics in
from diet, cancer research, and heart treat- health: The great health hoax.
ments in Matthews 1998). http://www2.isye.gatech.edu/~brani/
isyebayes/bank/pvalue.pdf
‘seventy years ago ronald Fisher gave
scientists a mathematical machine for
turning baloney into breakthroughs, and
flukes into funding’ (Matthews, 1998,
p.1).
This is especially shocking because research
in the UK is largely funded by the public
(taxpayers and charity-givers). Where social
science has impact in practice, the effect is
largely on the public in areas of policy like
education, crime, housing, transport and
health. Yet ethical committees and guide-
lines still largely ignore the interests of the
wider public in their focus on possible harm
to the researchers and the researched
(Gorard, 2002). I urge putwain to pursue the
implications of this second principle of
ethics, now creeping into ethical guidelines
such as those of the sra. public money is
being wasted, and public lives are being
made worse (or at least not improved as
much as they could be), by this invalid prac-
tice of significance testing. To call it merely a
‘cult’ is to downplay its importance. It should
cease. Now.

32 The Psychology of Education Review, Vol. 38, No. 1, Spring 2014

Copyright of Psychology of Education Review is the property of British Psychological
Society and its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.