13 Chen RT, Weierbach R, Bisoffi Z, Cutts F, Rhodes P, Ramaroson S, et al.
21 Ashy P, Bukh J, Hoff G, Leehay J, Lisse IM, Mordhorst CH, Pedersen IR.
Medical Statistics
Laboratory, Inperial
Cancer Research Fund,
London WC2A 3PX
Douglas G Altman, head
Department of Public
Health Sciences,
St George's Hospital
Medical School,
London SW17 ORE
J Martn Bland, reader in
medical statistics
Correspondence to:
Mr Altman.
BMJ 1995;311:485
BMJ VOLUME 311 19AuGusT1995
A "post-honeymoon period" measles outbreak in Muyinga Sector, Burundi.
IntrEpidemiol 1994;23:185-93.
14 Holt EA, Boulos R, Halsey NA, Boulos IM, Boulos C. Childhood survival in
Haiti: Protective effect of measles vaccination. Pediarics 1990;85:188-94.
15 Koenig MA, Khan MA, Wojtyniak B, Clemens JD, Chakraborty J, Fauveau
V, et al. The impact of measles vaccination upon childhood mortality in
Matiab, Bangladesh. Bufl World Healh Organ 1990;68:441-7.
16 Velema JP, Alihonou EM, Gandaho T, Hounye FH. Childhood mortality
among users and non-users of primary health care in a rural West African
community. IntJEpidemiol 1991;20:474-9.
17 Fleiss JL. The statistical basis of meta-analysis. Sta Mehods Med Res
1993;2:121-45.
18 Garenne M, Aaby P. Pattern of exposure and measles mortality in Senegal.
YInfDis 1990;161:1088-94.
19 Desgrees du Lou A, Pison G, Aaby P. The role of immunizations in the recent
decline in childhood mortality and the changes in the female/male mortality
ratio in rural Senegal. Am YEpidemiol (in press).
20 Harris MF. The safety of measles vaccine in severe illness. S Aft Med Y
1979:38.
Statistics Notes
Absence ofevidence is not evidence ofabsence
Douglas G Altman, J Martin Bland
The non-equivalence of statistical significance and
clinical importance has long been recognised, but this
error of interpretation remains common. Although a
significant result in a large study may sometimes not be
clinically important, a far greater problem arises from
misinterpretation of non-significant findings. By
convention a P value greater than 5% (P>0 05) is called
"not significant." Randomised controlled clinical
trials that do not show a significant difference between
the treatments being compared are often called
"negative." This term wrongly implies that the study
has shown that there is no difference, whereas usually
all that has been shown is an absence of evidence of a
difference. These are quite different statements.
The sample size of controlled trials is generally
inadequate, with a consequent lack of power to
detect real, and clinically worthwhile, differences in
treatment. Freiman et al found that only 30% of a
sample of 71 trials published in the New England
Journal of Medicine in 1978-9 with P>0-1 were large
enough to have a 90% chance of detecting even a 50%
difference in the effectiveness of the treatments being
compared, and they found no improvement in a similar
sample of trials published in 1988. To interpret all
these "negative" trials as providing evidence of the
ineffectiveness of new treatments is clearly wrong and
foolhardy. The term "negative" should not be used in
this context.2
A recent example is given by a trial comparing
octreotide and sclerotherapy in patients with variceal
bleeding.' The study was carried out on a sample of
only 100 despite a reported calculation that suggested
that 1800 patients were needed. This trial had only a
5% chance of getting a statistically significant result if
the stated clinically worthwhile treatment difference
truly existed. One consequence of such low statistical
power was a wide confidence interval for the treatment
difference. The authors concluded that the two
treatments were equally effective despite a 95%
confidence interval that included differences between
the cure rates of the two treatments of up to 20 per-
centage points.
Similar evidence of the dangers of misinterpretation
of non-significant results is found in numerous meta-
analyses (overviews) of published trials, when few or
none of the individual trials were statistically large
enough. A dramatic example is provided by the
overview of clinical trials evaluating fibrinolytic'
treatment (mostly streptokinase) for preventing
High measles mortality in infancy related to intensity of exposure. y Pediatr
1986;109:40-4.
22 Koenig MA. Mortality reductions from measles and tetanus immunization: a
review of the evidence. In: Hill K, ed. Child suwival pnornes in the 1990s.
Baltimore: Johns Hopkins Institute for Intemational Programs, 1992:43-72.
23 Hartfield J, Morley D. Efficacy of measles vaccine. Joumnal of Hygiene
(Cambridge) 1963;61:143-7.
24 Hull HF, Williams PJ, Oldfield F. Measles mortality and vaccine efficacy in
rural West Africa. Lancet 1983;i:972-5.
25 Griffin DE, Ward BJ, Esolen LM. Pathogenesis of measles virus infection: An
hypothesis for altered responses. IlInfect Dis 1994;170:S24-31.
26 Petralli JK, Merigan TC, Wilbur JR. Action of endogenous interferon against
vaccinia infection in children. Lancet 1965ii:401-5.
27 Rooth I, Sinani HM, Smedman 1, Bjorkman A. A study of malaria infection
during the acute stage of measles infection.J Trop Med Hyg 1991;94:195-8.
28 Gellin BG, Katz SL. Measles: state of the art and future directions. J Infect Dis
1994;170:S3-14.
(Accepted 15June 1995)
reinfarction after acute myocardial infarction. The
overview of randomised controlled trials found
a modest but clinically worthwhile (and highly sig-
nificant) reduction in mortality of 22%,4 but only five of
the 24 trials had shown a statistically significant effect
with P<005. The lack of statistical significance of
most of the individual trials led to a long delay before
the true value of streptokinase was appreciated.
While it is usually reasonable not to accept a new
treatment unless there is positive evidence in its
favour, when issues of public health are concerned
we must question whether the absence of evidence
is a valid enough justification for inaction. A recent
publicised example is the suggested link between some
sudden infant deaths and antimony in cot mattresses.
Statements about the absence of evidence are common
-for example, in relation to the possible link between
violent behaviour and exposure to violence on television
and video, the possible harmful effects of pesticide
residues in drinking water, the possible link between
electromagnetic fields and leukaemia, and the possible
transmission of bovine spongiform encephalopathy
from cows. Can we be comfortable that the absence of
clear evidence in such cases means that there is no risk
or only a negligible one?
When we are told that "there is no evidence that A
causes B" we should first ask whether absence of
evidence means simply that there is no information at
all. Ifthere are data we should look for quantification of
the association rather than just a P value. Where risks
are small P values may well mislead: confidence
intervals are likely to be wide, indicating considerable
uncertainty. While we can never prove the absence of a
relation, when necessary we should seek evidence
against the link between A and B-for example, from
case-control studies. The importance of carrying out
such studies will relate to the seriousness of the
postulated effect and how widespread is the exposure
in the population.
1 Freiman JA, Chalmers TC, Smith H, Kuebler HR. The importance of beta, the
type BI error, and sample size in the design and interpretation of the
randomized controlled trial: survey of two sets of "negative" trials. In: Bailar
JC, Mosteller F, eds. Medical ues of stanstics. 2nd ed. Boston, MA: NEJM
Books, 1992:357-73.
2 Chalmers I. Proposal to outlaw the term "negative trial." BM71985;29:1002.
3 Sung JJY, Chung SCS, Lai C-W, Chan FKL, Leung JWC, Yung M-L,
Kassianides C, ea aL Octreotide infusion or emergency sclerotherapy for
variceal haemorrhage. Lancet 1993;342:637-41.
4 Yusuf S, Collins R, Peto R, Furberg C, Stampfer MJ, Goldhaber SZ, et aL
Intravenous and intracoronary fibrinolytic therapy in acute myocardial
infarction: overview of results on mortality, reinfarction and side-effects from
33 randomized controlled trals. EurHeart_ 1985;6:556-85.
485
This site uses cookies. More info Close By continuing to browse the site you are agreeing to our use of
cookies. Find out more here Close

Education And Debate

Statistics Notes: Measurement error

BMJ 1996; 313 doi: https://doi.org/10.1136/bmj.313.7059.744 (Published 21 September 1996) Cite this as: BMJ
1996;313:744

J Martin Bland, professor of medical statisticsa, Douglas G Altman, headb

a
Department of Public Health Sciences, St George's Hospital Medical School, London SW17 0RE,
b
IRCF Medical Statistics Group, Centre for Statistics in Medicine, Institute of Health Sciences, PO Box 777, Oxford OX3 7LF

Correspondence to: Professor Bland.

Several measurements of the same quantity on the same subject will not in general be the same. This may be
because of natural variation in the subject, variation in the measurement process, or both. For example, table 1
shows four measurements of lung function in each of 20 schoolchildren (taken from a larger study1). The first
child shows typical variation, having peak expiratory flow rates of 190, 220, 200, and 200 1/min.

Child No PEFR (l/min) Mean SD

1 190 220 200 200 202.50 12.58

2 220 200 240 230 222.50 17.08

3 260 260 240 280 260.00 16.33

4 210 300 280 265 263.75 38.60

5 270 265 280 270 271.25 6.29

6 280 280 270 275 276.25 4.79

7 260 280 280 300 280.00 16.33

8 275 275 275 305 282.50 15.00

9 280 290 300 290 290.00 8.16

10 320 290 300 290 300.00 14.14

 Child No PEFR (l/min) Mean SD

11 300 300 310 300 302.50 5.00

12 270 250 330 370 305.00 55.08

13 320 330 330 330 327.50 5.00

14 335 320 335 375 341.25 23.58

15 350 320 340 365 343.75 18.87

16 360 320 350 345 343.75 17.02

17 330 340 380 390 360.00 29.44

18 335 385 360 370 362.50 21.02

19 400 420 425 420 416.25 11.09

20 430 460 480 470 460.00 21.60

Table 1
Repeated peak expiratory flow rate (PEFR) measurements for 20 schoolchildren

Let us suppose that the child has a true average value over all possible measurements, which is what we really
want to know when we make a measurement. Repeated measurements on the same subject will vary around the
true value because of measurement error. The standard deviation of repeated measurements on the same
subject enables us to measure the size of the measurement error. We shall assume that this standard deviation
is the same for all subjects, as otherwise there would be no point in estimating it. The main exception is when the
measurement error depends on the size of the measurement, usually with measurements becoming more
variable as the magnitude of the measurement increases. We deal with this case in a subsequent statistics note.
The common standard deviation of repeated measurements is known as the within-subject standard deviation,
which we shall denote by sw.

To estimate the within-subject standard deviation, we need several subjects with at least two measurements for
each. In addition to the data, table 1 also shows the mean and standard deviation of the four readings for each
child. To get the common within-subject standard deviation we actually average the variances, the squares of the
standard deviations. The mean within-subject variance is 460.52, so the estimated within-subject standard
deviation is sw = (square root)460.52 = 21.5 1/min. The calculation is easier using a program that performs one
way analysis of variance2 (table 2). The value called the residual mean square is the within-subject variance.
The analysis of variance method is the better approach in practice, as it deals automatically with the case of
subjects having different numbers of observations. We should check the assumption that the standard deviation
is unrelated to the magnitude of the measurement. This can be done graphically, by plotting the individual
subject's standard deviations against their means (see fig 1). Any important relation should be fairly obvious, but
we can check analytically by calculating a rank correlation coefficient. For the figure there does not appear to be
a relation (Kendall's (tau) = 0.16, P = 0.3).
 Source of Degrees of Sum of Mean Variance Probability
variation freedom squares square ratio (F) (P)

Children 19 285318.44 15016.78 32.6 <0.0001

Residual 60 27631.25 460.52

Total 79 312949.69

Table 2
One way analysis of variance for the data of table 1

Fig 1
Fig 1
Individual subjects' standard deviations plotted against their means

A common design is to take only two measurements per subject. In this case the method can be simplified
because the variance of two observations is half the square of their difference. So, if the difference between the
2 2
two observations for subject i is di the within-subject standard deviation sw is given by s w = 1/2n(summation)d i,
where n is the number of subjects. We can check for a relation between standard deviation and mean by plotting
for each subject the absolute value of the differencethat is, ignoring any signagainst the mean.

The measurement error can be quoted as sw. The difference between a subject's measurement and the true
value would be expected to be less than 1.96 sw for 95% of observations. Another useful way of presenting
measurement error is sometimes called the repeatability, which is (square root)2 x 1.96 sw or 2.77 sw. The
difference between two measurements for the same subject is expected to be less than 2.77 sw for 95% of pairs
of observations. For the data in table 1 the repeatability is 2.77 x 21.5 = 60 1/min. The large variability in peak
expiratory flow rate is well known, so individual readings of peak expiratory flow are seldom used. The variable
used for analysis in the study from which table 1 was taken was the mean of the last three readings.1

Other ways of describing the repeatability of measurements will be considered in subsequent statistics notes.

References
1.Bland JM, Holland WW, Elliott A. The development of respiratory symptoms in a cohort of Kent schoolchildren.Bull
Physio-Path Resp1974;10:699716.

2.Altman DG, Bland JM. Comparing several groups using analysis of variance.BMJ1996;312: 1472.
This site uses cookies. More info Close By continuing to browse the site you are agreeing to our use of
cookies. Find out more here Close

Education And Debate

Statistics Notes: Measurement error and correlation

coefficients
BMJ 1996; 313 doi: https://doi.org/10.1136/bmj.313.7048.41 (Published 06 July 1996) Cite this as: BMJ
1996;313:41

J Martin Bland, professor of medical statisticsa, Douglas G Altman, headb

a
Department of Public Health Sciences, St George's Hospital Medical School, London SW17 0RE
b
ICRF Medical Statistics Group, Centre for Statistics in Medicine, Institute of Health Sciences, PO Box 777, Oxford OX3 7LF

Correspondence to: Professor Bland

Measurement error is the variation between measurements of the same quantity on the same individual.1 To
quantify measurement error we need repeated measurements on several subjects. We have discussed the
within-subject standard deviation as an index of measurement error,1 which we like as it has a simple clinical
interpretation. Here we consider the use of correlation coefficients to quantify measurement error.

A common design for the investigation of measurement error is to take pairs of measurements on a group of
subjects, as in table 1. When we have pairs of observations it is natural to plot one measurement against the
other. The resulting scatter diagram (see figure 1 may tempt us to calculate a correlation coefficient between the
first and second measurement. There are difficulties in interpreting this correlation coefficient. In general, the
correlation between repeated measurements will depend on the variability between subjects. Samples containing
subjects who differ greatly will produce larger correlation coefficients than will samples containing similar
subjects. For example, suppose we split this group in whom we have measured forced expiratory volume in one
second (FEV1) into two subsamples, the first 10 subjects and the second 10 subjects. As table 1 is ordered by
the first FEV1 measurement, both subsamples vary less than does the whole sample. The correlation for the first
subsample is r = 0.63 and for the second it is r = 0.31, both less than r = 0.77 for the full sample. The correlation
coefficient thus depends on the way the sample is chosen, and it has meaning only for the population from which
the study subjects can be regarded as a random sample. If we select subjects to give a wide range of the
measurement, the natural approach when investigating measurement error, this will inflate the correlation
coefficient.

Measurement Measurement

Subject No 1st 2nd Subject No 1st 2nd

1 1.19 1.37 11 1.54 1.57

2 1.33 1.32 12 1.59 1.60

 Measurement Measurement

Subject No 1st 2nd Subject No 1st 2nd

3 1.35 1.40 13 1.61 1.53

4 1.36 1.25 14 1.61 1.61

5 1.38 1.29 15 1.62 1.68

6 1.38 1.40 16 1.80 1.82

7 1.38 1.40 17 1.80 1.82

8 1.40 1.38 18 1.85 1.89

9 1.43 1.38 19 1.94 2.10

10 1.43 1.51 20 2.10 2.20

Table 1
Pairs of measurements of FEV1 (litres) a few weeks apart from 20 Scottish schoolchildren, taken from a
larger study (D Strachan,personal communication)

Fig 1
Fig 1
Measurements from pairs of observations plotted against each other

The correlation coefficient between repeated measurements is often called the reliability of the measurement
method. It is widely used in the validation of psychological measures such as scales of anxiety and depression,
where it is known as the test-retest reliability. In such studies it is quoted for different populations (university
students, psychiatric outpatients, etc) because the correlation coefficient differs between them as a result of
differing ranges of the quantity being measured. The user has to select the correlation from the study population
most like the user's own.

Another problem with the use of the correlation coefficient between the first and second measurements is that
there is no reason to suppose that their order is important. If the order were important the measurements would
not be repeated observations of the same thing. We could reverse the order of any of the pairs and get a slightly
different value of the correlation coefficient between repeated measurements. For example, reversing the order
of the even numbered subjects in table 1 gives r = 0.80 instead of r = 0.77. The intra-class correlation coefficient
avoids this problem. It estimates the average correlation among all possible orderings of pairs. It also extends
easily to the case of more than two observations per subject, where it estimates the average correlation between
all possible pairs of observations.

Few computer programs will calculate the intra-class correlation coefficient directly, but when the number of
observations is the same for each subject it can be found from a one way analysis of variance table2 such as
table 2. We need the total sum of squares, SST, and the sum of squares between subjects, SSB.
Then

rI = mSSB - SST/(m - 1) SST

where m is the number of observations per subject. For table II, m = 2 and

rI = 2 1.52981 - 1.74651/(2 - 1) 1.74651 = 0.75

Source of Degrees of Sum of Mean Variance

Probability(P)
variation freedom squares square ratio (F)

Children 19 1.52981 0.08052 7.4 <0.0001

Residual 20 0.21670 0.01086

Total 39 1.74651

Table 2
One way analysis of variance for the data in table 1

In practice, there will usually be little difference between r and rI for true repeated measurements. If, however,
there is a systematic change from the first measurement to the second, as might be caused by a learning effect,
rI will be much less than r. If there was such an effect the measurements would not be made under the same
conditions and so we could not measure reliability.

The correlation coefficient can be used to compare measurements of different quantities, such as different scales
for measuring anxiety. We could make repeated measurements of all the quantities on the same subjects and
calculate intra-class correlations. The measures with the highest correlation between repeated measurements
would discriminate best between individuals; in other words they would carry the most information. For most
applications, however, we prefer the within-subjects standard deviation as an index of measurement error, as it
has a more direct interpretation which can be applied to individual measurements.1

References
1.Bland JM, Altman AD.Measurement error.BMJ 1996;312: 1654.

2.Altman DG, Bland BJ.Comparing several groups using a analysis of variance BMJ 1996;312: 14723.

Statistics notes
Bayesians and frequentists
J Martin Bland, Douglas G Altman,
There are two competing philosophies of statistical
analysis: the Bayesian and the frequentist. The
frequentists are much the larger group, and almost all
the statistical analyses which appear in the BMJ are fre-
quentist. The Bayesians are much fewer and until
recently could only snipe at the frequentists from the
high ground of university departments of mathemati-
cal statistics. Now the increasing power of computers is
bringing Bayesian methods to the fore.
Bayesian methods are based on the idea that
unknown quantities, such as population means and
proportions, have probability distributions. The prob-
ability distribution for a population proportion
expresses our prior knowledge or belief about it, before
we add the knowledge which comes from our data. For
example, suppose we want to estimate the prevalence
of diabetes in a health district. We could use the knowl-
edge that the percentage of diabetics in the United
Kingdom as a whole is about 2%, so we expect the
prevalence in our health district to be fairly similar. It is
unlikely to be 10%, for example. We might have infor-
mation based on other datasets that such rates vary
between 1% and 3%, or we might guess that the preva-
lence is somewhere between these values. We can con-
struct a prior distribution which summarises our
beliefs about the prevalence in the absence of specific
data. We can do this with a distribution having mean 2
and standard deviation 0.5, so that two standard devia-
tions on either side of the mean are 1% and 3%. (The
precise mathematical form of the prior distribution
depends on the particular problem.)
Suppose we now collect some data by a sample
survey of the district population. We can use the data to
modify the prior probability distribution to tell us what
we now think the distribution of the population
percentage is; this is the posterior distribution. For
example, if we did a survey of 1000 subjects and found
15 (1.5%) to be diabetic, the posterior distribution
would have mean 1.7% and standard deviation 0.3%.
We can calculate a set of values, a 95% credible interval
(1.2% to 2.4% for the example), such that there is a
probability of 0.95 that the percentage of diabetics is
within this set. The frequentist analysis, which ignores
the prior information, would give an estimate 1.5%
with standard error 0.4% and 95% confidence interval
0.8% to 2.5%. This is similar to the results of the Baye-
sian method, as is usually the case, but the Bayesian
method gives an estimate nearer the prior mean and a
narrower interval.
Frequentist methods regard the population value
as a fixed, unvarying (but unknown) quantity, without a
probability distribution. Frequentists then calculate
confidence intervals for this quantity, or significance
tests of hypotheses concerning it. Bayesians reasonably
object that this does not allow us to use our wider
knowledge of the problem. Also, it does not provide
what researchers seem to want, which is to be able to
say that there is a probability of 95% that the
BMJ VOLUME 317 24 OCTOBER 1998 www.bmj.com
Education and debate
population value lies within the 95% confidence inter- Department of
Public Health
val, or that the probability that the null hypothesis is Sciences, St
true is less than 5%. It is argued that researchers want Georges Hospital
this, which is why they persistently misinterpret Medical School,
London SW17 0RE
confidence intervals and significance tests in this way.
J Martin Bland,
A major difficulty, of course, is deciding on the professor of medical
prior distribution. This is going to influence the statistics
conclusions of the study, yet it may be a subjective syn- ICRF Medical
thesis of the available information, so the same data Statistics Group,
Centre for Statistics
analysed by different investigators could lead to differ- in Medicine,
ent conclusions. Another difficulty is that Bayesian Institute of Health
methods may lead to intractable computational Sciences, Oxford
OX3 7LF
problems. (All widely available statistical packages use Douglas G Altman,
frequentist methods.) head
Most statisticians have become Bayesians or Correspondence to:
frequentists as a result of their choice of university. Professor Bland
They did not know that Bayesians and frequentists
existed until it was too late and the choice had been BMJ 1998;317:1151
made. There have been subsequent conversions. Some
who were taught the Bayesian way discovered that
when they had huge quantities of medical data to ana-
lyse the frequentist approach was much quicker and
more practical, although they may remain Bayesian at
heart. Some frequentists have had Damascus road con-
versions to the Bayesian view. Many practising
statisticians, however, are fairly ignorant of the
methods used by the rival camp and too busy to have
time to find out.
The advent of very powerful computers has given a
new impetus to the Bayesians. Computer intensive
methods of analysis are being developed, which allow
new approaches to very difficult statistical problems,
such as the location of geographical clusters of cases of
a disease. This new practicability of the Bayesian
approach is leading to a change in the statistical
paradigmand a rapprochement between Bayesians
and frequentists.1 2 Frequentists are becoming curious
about the Bayesian approach and more willing to use
Bayesian methods when they provide solutions to diffi-
cult problems. In the future we expect to see more
Bayesian analyses reported in the BMJ. When this hap-
pens we may try to use Statistics notes to explain them,
though we may have to recruit a Bayesian to do it.
We thank David Spiegelhalter for comments on a draft.
1 Breslow N. Biostatistics and Bayes (with discussion). Statist Sci 1990;5:
269-98.
2 Spiegelhalter DJ, Freedman LS, Parmar MKB. Bayesian approaches to
randomized trials (with discussion). J R Statist Soc A 1994;157:357-416.
Correction
North of England evidence based guidelines development project:
guideline for the primary care management of dementia
An editorial error occurred in this article by Martin Eccles
and colleagues (19 September, pp 802-8). In the list of
authors the name of Moira Livingston [not Livingstone] was
misspelt.
1151
Clinical review

Statistics Notes
Survival probabilities (the Kaplan-Meier method)
J Martin Bland, Douglas G Altman

Department of As we have observed,1 analysis of survival data requires

Public Health
special techniques because some observations are 1.0

Survival probability
Sciences, St
Georges Hospital censored as the event of interest has not occurred for all
Medical School, patients. For example, when patients are recruited over
London SW17 0RE 0.75
J Martin Bland,
two years one recruited at the end of the study may be
professor of medical alive at one year follow up, whereas one recruited at the
statistics start may have died after two years. The patient who died 0.5
ICRF Medical has a longer observed survival than the one who still
Statistics Group,
survives and whose ultimate survival time is unknown.
Centre for Statistics
in Medicine, The table shows data from a study of conception in 0.25
Institute of Health subfertile women.2 The event is conception, and
Sciences, Oxford
OX3 7LF women survived until they conceived. One woman
0
Douglas G Altman, conceived after 16 months (menstrual cycles), whereas 0 6 12 18 24
head several were followed for shorter time periods during Time (months)
Correspondence to: which they did not conceive; their time to conceive was Survival curve showing probability of not conceiving among 38
Professor Bland thus censored. subfertile women after laparoscopy and hydrotubation2
We wish to estimate the proportion surviving (not
BMJ 1998;317:1572
having conceived) by any given time, which is also the
estimated probability of survival to that time for a There are three assumptions in the above. Firstly,
member of the population from which the sample is we assume that at any time patients who are censored
drawn. Because of the censoring we use the have the same survival prospects as those who
continue to be followed. This assumption is not easily
Kaplan-Meier method. For each time interval we
testable. Censoring may be for various reasons. In the
estimate the probability that those who have survived
conception study some women had received hormone
to the beginning will survive to the end. This is a condi-
treatment to promote ovulation, and others had
tional probability (the probability of being a survivor at
stopped trying to conceive. Thus they were no longer
the end of the interval on condition that the subject
part of the population we wanted to study, and their
was a survivor at the beginning of the interval). Survival
survival times were censored. In most studies some
Time (months) to to any time point is calculated as the product of the
subjects drop out for reasons unrelated to the
conception or conditional probabilities of surviving each time
condition under study (for example, emigration) If,
censoring in 38 interval. These data are unusual in representing
sub-fertile women however, for some patients in this study censoring was
months (menstrual cycles); usually the conditional
after laparoscopy related to failure to conceive this would have biased the
probabilities relate to days. The calculations are simpli-
and hydrotubation2 estimated survival probabilities downwards.
fied by ignoring times at which there were no recorded
Secondly, we assume that the survival probabilities
Did not survival times (whether events or censored times).
Conceived conceive are the same for subjects recruited early and late in the
In the example, the probability of surviving for two
1 2 study. In a long term observational study of patients
months is the probability of surviving the first month
1 3 with cancer, for example, the case mix may change over
times the probability of surviving the second month
1 4 the period of recruitment, or there may be an innova-
given that the first month was survived. Of 38 women,
1 7 tion in ancillary treatment. This assumption may be
1 7 32 survived the first month, or 0.842. Of the 32 women tested, provided we have enough data to estimate
1 8 at the start of the second month (at risk of survival curves for different subsets of the data.
2 8 conception), 27 had not conceived by the end of the Thirdly, we assume that the event happens at the
2 9 month. The conditional probability of surviving the time specified. This is not a problem for the conception
2 9 second month is thus 27/32 = 0.844, and the overall data, but could be, for example, if the event were recur-
2 9 probability of surviving (not conceiving) after two rence of a tumour which would be detected at a regu-
2 11 months is 0.842 0.844 = 0.711. We continue in this lar examination. All we would know is that the event
3 24
way to the end of the table, or until we reach the last happened between two examinations. This imprecision
3 24
event. Observations censored at a given time affect the would bias the survival probabilities upwards. When
3
number still at risk at the start of the next month. The the observations are at regular intervals this can be
4
4
estimated probability changes only in months when allowed for quite easily, using the actuarial method.3
4 there is a conception. In practice, a computer is used to Formal methods are needed for testing hypotheses
6 do these calculations. Standard errors and confidence about survival in two or more groups. We shall describe
6 intervals for the estimated survival probabilities can be the logrank test for comparing curves and the more
9 found by Greenwoods method.3 Survival probabilities complex Cox regression model in future Notes.
9 are usually presented as a survival curve (figure). The
1 Altman DG, Bland JM. Time to event (survival) data. BMJ
9 curve is a step function, with sudden changes in the 1997;317:468-9.
10 estimated probability corresponding to times at which 2 Luthra P, Bland JM, Stanton SL. Incidence of pregnancy after
13 laparoscopy and hydrotubation. BMJ 1982;284:1013-4.
an event was observed. The times of the censored data 3 Parmar MKB, Machin D. Survival analysis: a practical approach. Chichester:
16
are indicated by short vertical lines. Wiley, 37, 47-9.

1572 BMJ VOLUME 317 5 DECEMBER 1998 www.bmj.com


Statistics notes
Treatment allocation in controlled trials: why randomise?
Douglas G Altman, J Martin Bland
Since 1991 the BMJ has had a policy of not publishing
trials that have not been properly randomised, except
in rare cases where this can be justified.1 Why?
The simplest approach to evaluating a new
treatment is to compare a single group of patients
given the new treatment with a group previously
treated with an alternative treatment. Usually such
studies compare two consecutive series of patients in
the same hospital(s). This approach is seriously flawed.
Problems will arise from the mixture of retrospective
and prospective studies, and we can never satisfactorily
eliminate possible biases due to other factors (apart
from treatment) that may have changed over time.
Sacks et al compared trials of the same treatments in
which randomised or historical controls were used and
found a consistent tendency for historically controlled
trials to yield more optimistic results than randomised
trials.2 The use of historical controls can be justified
only in tightly controlled situations of relatively rare
conditions, such as in evaluating treatments for
advanced cancer.
The need for contemporary controls is clear, but
there are difficulties. If the clinician chooses which
treatment to give each patient there will probably be
differences in the clinical and demographic character-
istics of the patients receiving the different treatments.
Much the same will happen if patients choose their
own treatment or if those who agree to have a
treatment are compared with refusers. Similar prob-
lems arise when the different treatment groups are at
different hospitals or under different consultants. Such
systematic differences, termed bias, will lead to an over-
estimate or underestimate of the difference between
treatments. Bias can be avoided by using random allo-
cation.
A well known example of the confusion engen-
dered by a non-randomised study was the study of the
possible benefit of vitamin supplementation at the time
of conception in women at high risk of having a baby
with a neural tube defect.3 The investigators found that
the vitamin group subsequently had fewer babies with
neural tube defects than the placebo control group.
The control group included women ineligible for the
trial as well as women who refused to participate. As a
consequence the findings were not widely accepted,
and the Medical Research Council later funded a large
randomised trial to answer to the question in a way that
would be widely accepted.4
The main reason for using randomisation to
allocate treatments to patients in a controlled trial is to
prevent biases of the types described above. We want to
compare the outcomes of treatments given to groups
of patients which do not differ in any systematic way.
Another reason for randomising is that statistical
theory is based on the idea of random sampling. In a
study with random allocation the differences between
treatment groups behave like the differences between
random samples from a single population. We know
BMJ VOLUME 318 1 MAY 1999 www.bmj.com
Education and debate
how random samples are expected to behave and so ICRF Medical
Statistics Group,
can compare the observations with what we would Centre for Statistics
expect if the treatments were equally effective. in Medicine,
Institute of Health
The term random does not mean the same as hap- Sciences, Oxford
hazard but has a precise technical meaning. By random OX3 7LF
allocation we mean that each patient has a known Douglas G Altman,
chance, usually an equal chance, of being given each professor of statistics
in medicine
treatment, but the treatment to be given cannot be pre-
Department of
dicted. If there are two treatments the simplest method Public Health
of random allocation gives each patient an equal Sciences, St
Georges Hospital
chance of getting either treatment; it is equivalent to Medical School,
tossing a coin. In practice most people use either a London SW17 0RE
J Martin Bland,
table of random numbers or a random number
professor of medical
generator on a computer. This is simple randomisa- statistics
tion. Possible modifications include block randomisa-
Correspondence to:
tion, to ensure closely similar numbers of patients in Professor Altman.
each group, and stratified randomisation, to keep the
groups balanced for certain prognostic patient charac- BMJ 1999;318:1209
teristics. We discuss these extensions in a subsequent
Statistics note.
Fifty years after the publication of the first
randomised trial5 the technical meaning of the term
randomisation continues to elude some investigators.
Journals continue to publish randomised trials which
are no such thing. One common approach is to
allocate treatments according to the patients date of
birth or date of enrolment in the trial (such as giving
one treatment to those with even dates and the other to
those with odd dates), by the terminal digit of the hos-
pital number, or simply alternately into the different
treatment groups. While all of these approaches are in
principle unbiasedbeing unrelated to patient
characteristicsproblems arise from the openness of
the allocation system.1 Because the treatment is known
when a patient is considered for entry into the trial this
knowledge may influence the decision to recruit that
patient and so produce treatment groups which are
not comparable.
Of course, situations exist where randomisation is
simply not possible.6 The goal here should be to retain
all the methodological features of a well conducted
randomised trial7 other than the randomisation.
1 Altman DG. Randomisation. BMJ 1991;302:1481-2.
2 Sacks H, Chalmers TC, Smith H. Randomized versus historical controls
for clinical trials. Am J Med 1982;72:233-40.
3 Smithells RW, Sheppard S, Schorah CJ, Seller MJ, Nevin NC, Harris R, et
al. Possible prevention of neural-tube defects by periconceptional vitamin
supplementation. Lancet 1980;i:339-40.
4 MRC Vitamin Study Research Group. Prevention of neural tube defects:
results of the Medical Research Council vitamin study. Lancet
1991;338:131-7.
5 Medical Research Council. Streptomycin treatment of pulmonary tuber-
culosis. BMJ 1948;2:769-82.
6 Black N. Why we need observational studies to evaluate the effectiveness
of health care. BMJ 1996;312:1215-8.
7 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improv-
ing the quality of reporting of randomized controlled trials: the
CONSORT Statement. JAMA 1996;276:637-9.
1209
 General practice

15 Van den Hoogen HJM, Koes BW, van Eijk JT, Bouter LM, Devill W. On
the course of low back pain in general practice: a one year follow up
study. Ann Rheum Dis 1998;57:13-9.
16 Croft PR, Papageorgiou AC, Ferry S, Thomas E, Jayson MIV, Silman AJ.
Psychological distress and low back pain: Evidence from a prospective
study in the general population. Spine 1996;20:2731-7.
17 Papageorgiou AC, Macfarlane GJ, Thomas E, Croft PR, Jayson MIV,
Silman AJ. Psychosocial factors in the work placedo they predict new
episodes of low back pain? Spine 1997;22:1137-42.
18 Main CJ, Wood PL, Hollis S, Spanswick CC, Waddell G. The distress and
risk assessment method. A simple patient classification to identify distress
and evaluate the risk of poor outcome. Spine 1992;17:42-52.
19 Coste J, Delecoeuillerie G, Cohen de Lara A, Le Parc JM, Paolaggi JB.
Clinical course and prognostic factors in acute low back pain: an incep-
tion cohort study in primary care practice. BMJ 1994;308:577-80.
20 Dionne CE, Koepsell TD, Von Korff M, Deyo RA, Barlow WE, Checkoway
H. Predicting long-term functional limitations among back pain patients
in primary care. J Clin Epidemiol 1997;50:31-43.
21 Macfarlane GJ, Thomas E, Papageorgiou AC, Schollum J, Croft PR. The
natural history of chronic pain in the community: a better prognosis than
in the clinic? J Rheumatol 1996;23:1617-20.
22 Troup JDG, Martin JW, Lloyd DCEF. Back pain in industry. A prospective
survey. Spine 1981;6:61-9.
23 Burton AK, Tillotson KM. Prediction of the clinical course of low-back
trouble using multivariable models. Spine 1991;16:7-14.
24 Pope MH, Rosen JC, Wilder DG, Frymoyer JW. The relation between bio-
mechanical and psychological factors in patients with low-back pain.
Spine 1980;5:173-8.
(Accepted 31 March 1999)

Statistics notes
Variables and parameters
Douglas G Altman, J Martin Bland

Like all specialist areas, statistics has developed its own ICRF Medical
80 Statistics Group,
language. As we have noted before,1 much confusion Frequency Centre for Statistics
may arise when a word in common use is also given a in Medicine,
technical meaning. Statistics abounds in such terms, Institute of Health
60 Sciences, Oxford
including normal, random, variance, significant, etc. OX3 7LF
Two commonly confused terms are variable and Douglas G Altman,
parameter; here we explain and contrast them. 40 professor of statistics
in medicine
Information recorded about a sample of individu-
als (often patients) comprises measurements such as Department of
20 Public Health
blood pressure, age, or weight and attributes such as Sciences, St
blood group, stage of disease, and diabetes. Values of Georges Hospital
these will vary among the subjects; in this context Medical School,
0 London SW17 0RE
35 40 45 50 55
blood pressure, weight, blood group and so on are J Martin Bland,
Albumin (g/l)
variables. Variables are quantities which vary from professor of medical
individual to individual. Measurements of serum albumin in 481 white men aged over 20 statistics
By contrast, parameters do not relate to actual (data from Dr W G Miller) Correspondence to:
Professor Altman.
measurements or attributes but to quantities defining a
theoretical model. The figure shows the distribution of (more generally known as regression coefficients) are
BMJ 1999;318:1667
measurements of serum albumin in 481 white men the parameters defining the model. They have no
aged over 20 with mean 46.14 and standard deviation meaning for individuals, although they can be used to
3.08 g/l. For the empirical data the mean and SD are predict an individuals lung function from their height.
called sample estimates. They are properties of the col- In some contexts parameters are values that can be
lection of individuals. Also shown is the normal1 distri- altered to see what happens to the performance of
bution which fits the data most closely. It too has mean some system. For example, the performance of a
46.14 and SD 3.08 g/l. For the theoretical distribution screening programme (such as positive predictive
the mean and SD are called parameters. There is not value or cost effectiveness) will depend on aspects such
one normal distribution but many, called a family of as the sensitivity and specificity of the screening test. If
distributions. Each member of the family is defined by we look to see how the performance would change if,
its mean and SD, the parameters1 which specify the say, sensitivity and specificity were improved, then we
particular theoretical normal distribution with which are treating these as parameters rather than using the
we are dealing. In this case, they give the best estimate values observed in a real set of data.
of the population distribution of serum albumin if we Parameter is a technical term which has only
can assume that in the population serum albumin has recently found its way into general use, unfortunately
a normal distribution. without keeping its correct meaning. It is common in
Most statistical methods, such as t tests, are called medical journals to find variables incorrectly called
parametric because they estimate parameters of some parameters (but not in the BMJ we hope2). Another
underlying theoretical distribution. Non-parametric common misuse of parameter is as a limit or boundary,
methods, such as the Mann-Whitney U test and the log as in within certain parameters. This misuse seems to
rank test for survival data, do not assume any particu- have arisen from confusion between parameter and
lar family for the distribution of the data and so do not perimeter.
estimate any parameters for such a distribution. Misuse of medical terms is rightly deprecated. Like
Another use of the word parameter relates to its other language errors it leads to confusion and the loss
original mathematical meaning as the value(s) defining of valuable distinction. Misuse of non-medical terms
one of a family of curves. If we fit a regression model, should be viewed likewise.
such as that describing the relation between lung func- 1 Altman DG, Bland JM. The normal distribution. BMJ 1995;310:298.
tion and height, the slope and intercept of this line 2 Endpiece: Whats a parameter? BMJ 1998;316:1877.

BMJ VOLUME 318 19 JUNE 1999 www.bmj.com 1667

 Education and debate

Much work still needs to be done to achieve this. To be
useful in health policy at this level, all the targets need
to be elaborated further and clear, practical statements
must be made on their operationespecially the four
targets on health policy and sustainable health systems.
The WHO should stimulate the discussion of these
important targets, but it should also be careful about
being too prescriptive about health systems since this
could be counterproductive.
In addition, more attention should be given to the
usefulness of the targets in member states. One way of
doing this is to rank the countries by target and to
divide them into three groups. A specific level could be
set for each group. For example, for target 2, three such
groups could be distinguished as follows:
x Countries that have already achieved this target
x Countries for which the global target is achievable
and challenging
x Countries that find the global target hard to achieve
and therefore demotivating.
The first group needs stricter target levels, and the
third group less stringent ones. If a breakdown of this
kind is made for each target, some countries may be
classified in different groups for different targets. In this
way, the targets will provide an insight into the health
status of the population and could be useful for policy
makers in member states in encouraging action and
allocating their resources.
We thank Dr J Visschedijk and Professor L J Gunning-Schepers
and other referees of this article for their helpful comments.
Funding: This study was commissioned by Policy Action
Coordination at the WHO and supported by an unrestricted
educational grant from Merck & Co Inc, New Jersey, USA.
Competing interests: None declared.
1 World Health Assembly. Resolution WHA51.7. Health for all policy for the
twenty-first century. Geneva: World Health Organisation, 1998.
2 World Health Association. Health for all in the 21st century. Geneva: WHO,
1998.
3 World Health Association. Global strategy for health for all by the year 2000.
Geneva: WHO, 1981. (WHO Health for All series No 3.)
4 Visschedijk J, Simant S. Targets for health for all in the 21st century.
World Health Stat Q 1998;51:56-67.
5 Van de Water HPA, van Herten LM. Never change a winning team? Review
of WHOs new global policy: health for all in the 21st century. Leiden: TNO
Prevention and Health, 1999.
6 World Health Organisation. Bridging the gaps. Geneva: WHO, 1995.
(World health report.)
7 World Health Organisation. Fighting disease, fostering development. Geneva:
WHO, 1996. (World health report.)
8 World Health Organisation. 1997: Conquering suffering, enriching humanity.
Geneva: WHO, 1997. (World health report.)
9 Murray CJL, Lopez AD, eds. The global burden of disease. Boston: Harvard
University Press, 1996.
10 United Nations. The world population prospects. New York: UN, 1998.
11 United Nations Development Programme. Human development report
1997. New York: Oxford University Press, 1997.
12 World Bank. Poverty reduction and the World Bank: progress and challenges in
the 1990s. New York: World Bank, 1996.
13 World Health Organisation. Third evaluation of health for all by the year
14
2000. Geneva: WHO, 1999. (In press.)
Ad Hoc Committee on Health Research Relating to Future Intervention
Options. Investing in health research and development. Geneva: WHO,1996.
(Document TDR/Gen/96.1.)
15 Taylor CE. Surveillance for equity in primary health care: policy implica-
tions from international experience. Int J Epidemiol 1992;21:1043-9.
16 Frerichs RR. Epidemiologic surveillance in developing countries. Annu
Rev Public Health 1991;12:257-80.
17 World Health Organisation. Health for all renewalbuilding sustainable
health systems: from policy to action. Report of meeting on 17-19 November 1997
in Helsinki, Finland. Geneva: WHO, 1998.
18 World Health Organisation. EMC annual report 1996. Geneva: WHO:
1996.
19 World Health Organisation. Physical status: the use and interpretation of
anthropometry of a WHO expert committee. Geneva: WHO, 1995. (WHO
technical report series No 834.)
20 World Health Organisation. Global database on child growth and malnutri-
tion. Geneva: WHO, 1997.
21 World Health Organisation. Tobacco or health: a global status report. Geneva:
WHO, 1997.
22 Erkens C. Cost-effectiveness of short course chemotherapy in smear-negative
tuberculosis. Utrecht: Netherlands School of Public Health, 1996.
23 Van de Water HPA, van Herten LM. Bulls eye or Achilles heel: WHOs Euro-
pean health for all targets evaluated in the Netherlands. Leiden: Netherlands
Association for Applied Scientific Research (TNO) Prevention and
Health, 1996.
24 Van de Water HPA, van Herten LM. Health policies on target? Review of
health target and priority setting in 18 European countries. Leiden:
Netherlands Association for Applied Scientific Research (TNO) Preven-
tion and Health, 1998.
(Accepted 4 May 1999)

Statistics notes
How to randomise
Douglas G Altman, J Martin Bland

We have explained why random allocation of place to start and also the direction in which to read ICRF Medical
Statistics Group,
treatments is a required feature of controlled trials.1 the table. The first 10 two digit numbers from a starting Centre for Statistics
Here we consider how to generate a random allocation place in column 2 are 85 80 62 36 96 56 17 17 23 87, in Medicine,
sequence. which translate into the sequence A B B B B B A A A Institute of Health
Sciences, Oxford
Almost always patients enter a trial in sequence A for the first 10 patients. We could instead have taken OX3 7LF
over a prolonged period. In the simplest procedure, each digit on its own, or numbers 00 to 49 for A and 50 Douglas G Altman,
simple randomisation, we determine each patients to 99 for B. There are countless possible strategies; it professor of statistics
in medicine
treatment at random independently with no con- makes no difference which is used.
straints. With equal allocation to two treatment groups We can easily generalise the approach. With three Department of
Public Health
this is equivalent to tossing a coin, although in practice groups we could use 01 to 33 for A, 34 to 66 for B, and Sciences, St
coins are rarely used. Instead we use computer gener- 67 to 99 for C (00 is ignored). We could allocate treat- Georges Hospital
Medical School,
ated random numbers. Suitable tables can be found in ments A and B in proportions 2 to 1 by using 01 to 66
London SW17 0RE
most statistics textbooks. The table shows an example2: for A and 67 to 99 for B. J Martin Bland,
the numbers can be considered as either random digits At any point in the sequence the numbers of professor of medical
from 0 to 9 or random integers from 0 to 99. patients allocated to each treatment will probably statistics

For equal allocation to two treatments we could differ, as in the above example. But sometimes we want Correspondence to:
Professor Altman.
take odd and even numbers to indicate treatments A to keep the numbers in each group very close at all
and B respectively. We must then choose an arbitrary times. Block randomisation (also called restricted BMJ 1999;319:7034

BMJ VOLUME 319 11 SEPTEMBER 1999 www.bmj.com 703

Education and debate

compare two alternative treatments for breast cancer it

Excerpt from a table of random digits.2 The numbers used in the
might be important to stratify by menopausal status.
example are shown in bold
Separate lists of random numbers should then be con-
89 11 77 99 94 structed for premenopausal and postmenopausal
35 83 73 68 20
women. It is essential that stratified treatment
84 85 95 45 52
allocation is based on block randomisation within each
56 80 93 52 82
stratum rather than simple randomisation; otherwise
97 62 98 71 39
79 36 13 72 99
there will be no control of balance of treatments within
34 96 98 54 89
strata, so the object of stratification will be defeated.
69 56 88 97 43 Stratified randomisation can be extended to two or
09 17 78 78 02 more stratifying variables. For example, we might want
83 17 39 84 16 to extend the stratification in the breast cancer trial to
24 23 36 44 14 tumour size and number of positive nodes. A separate
39 87 30 20 41 randomisation list is needed for each combination of
75 18 53 77 83 categories. If we had two tumour size groups (say <4
33 93 39 24 81 and > 4cm) and three groups for node involvement (0,
22 52 01 86 71 1-4, > 4) as well as menopausal status, then we have
2 3 2 = 12 strata, which may exceed the limit of what
randomisation) is used for this purpose. For example, if is practical. Also with multiple strata some of the com-
we consider subjects in blocks of four at a time there binations of categories may be rare, so the intended
are only six ways in which two get A and two get B: treatment balance is not achieved.
1: A A B B 2: A B A B 3: A B B A 4: B B A A 5: B A In a multicentre study the patients within each cen-
B A 6: B A A B tre will need to be randomised separately unless there
We choose blocks at random to create the is a central coordinated randomising service. Thus
allocation sequence. Using the single digits of the pre- centre is a stratifying variable, and there may be other
vious random sequence and omitting numbers outside stratifying variables as well.
the range 1 to 6 we get 5 6 2 3 6 6 5 6 1 1. From these In small studies it is not practical to stratify on more
we can construct the block allocation sequence B A B than one or perhaps two variables, as the number of
A / B A A B / A B A B / A B B A / B A A B, and so on. strata can quickly approach the number of subjects.
The numbers in the two groups at any time can never When it is really important to achieve close similarity
differ by more than half the block length. Block size is between treatment groups for several variables
normally a multiple of the number of treatments. minimisation can be usedwe discuss this method in a
Large blocks are best avoided as they control balance separate Statistics note.3
less well. It is possible to vary the block length, again at We have described the generation of a random
random, perhaps using a mixture of blocks of size 2, 4, sequence in some detail so that the principles are clear.
or 6. In practice, for many trials the process will be done by
While simple randomisation removes bias from the computer. Suitable software is available at http://
allocation procedure, it does not guarantee, for exam- www.sghms.ac.uk/phs/staff/jmb/jmb.htm.
ple, that the individuals in each group have a similar We shall also consider in a subsequent note the
age distribution. In small studies especially some practicalities of using a random sequence to allocate
chance imbalance will probably occur, which might treatments to patients.
complicate the interpretation of results. We can use
stratified randomisation to achieve approximate
1 Altman DG, Bland JM. Treatment allocation in controlled trials: why ran-
balance of important characteristics without sacrificing domise? BMJ 1999;318:1209.
the advantages of randomisation. The method is to 2 Altman DG. Practical statistics for medical research. London: Chapman and
Hall, 1990: 540-4.
produce a separate block randomisation list for each 3 Treasure T, MacRae KD. Minimisation: the platinum standard for trials?
subgroup (stratum). For example, in a study to BMJ 1998;317:362-3.

One hundred years ago

Generalisation of salt infusions

The subcutaneous infusion of salt solution has proved of great
benefit in the treatment of collapse after severe operations. The
practice, it may be said, developed from two sources: the new
method of transfusion where water, instead of another persons
blood, is injected into the patients veins; and flushing of the
peritoneum, introduced by Lawson Tait. After flushing, much of
the fluid left in the peritoneum is absorbed into the circulation,
greatly to the patients advantage. Dr. Clement Penrose has tried
the effect of subcutaneous salt infusions as a last extremity in
severe cases of pneumonia. He continues this treatment with
inhalations of oxygen. He has had experience of three cases, all
considered hopeless, and succeeded in saving one. In the other
two the prolongation of life and the relief of symptoms were so
marked that Dr. Penrose regretted that the treatment had not
been employed earlier. Several physicians have adopted Dr.
Penroses method, and with the most gratifying results. The cases
are reported fully in the Bulletin of the Johns Hopkins Hospital for
July last. The infusions of salt solution were administered just as
after an operation. The salt solution, at a little above body
temperature, is poured into a graduated bottle connected by a
rubber tube with a needle. The pressure is regulated by elevating
the bottle, or by means of a rubber bulb with valves; the needle is
introduced into the connective tissue under the breast or under
the integuments of the thighs. There can be no doubt that
subcutaneous saline infusions are increasing in popularity, and
little doubt that their use will be greatly extended in medicine as
well as surgery.
(BMJ 1899;ii:933)

704 BMJ VOLUME 319 11 SEPTEMBER 1999 www.bmj.com

Education and debate

Statistics Notes
The odds ratio
J Martin Bland, Douglas G Altman

Department of In recent years odds ratios have become widely used in The two odds ratios are
Public Health
Sciences, medical reportsalmost certainly some will appear in
St Georges todays BMJ. There are three reasons for this. Firstly,
Hospital Medical
they provide an estimate (with confidence interval) for
School, London
SW17 0RE the relationship between two binary (yes or no) vari- which can both be rearranged to give
J Martin Bland ables. Secondly, they enable us to examine the effects of
professor of medical
statistics
other variables on that relationship, using logistic
regression. Thirdly, they have a special and very
ICRF Medical
Statistics Group, convenient interpretation in case-control studies (dealt If we switch the order of the categories in the rows and
Centre for Statistics with in a future note). the columns, we get the same odds ratio. If we switch
in Medicine,
Institute of Health
The odds are a way of representing probability, the order for the rows only or for the columns only, we
Sciences, Oxford especially familiar for betting. For example, the odds get the reciprocal of the odds ratio, 1/4.89 = 0.204.
OX3 7LF that a single throw of a die will produce a six are 1 to These properties make the odds ratio a useful
Douglas G Altman
professor of statistics
5, or 1/5. The odds is the ratio of the probability that indicator of the strength of the relationship.
in medicine the event of interest occurs to the probability that it The sample odds ratio is limited at the lower end,
Correspondence to: does not. This is often estimated by the ratio of the since it cannot be negative, but not at the upper end,
Professor Bland number of times that the event of interest occurs to and so has a skew distribution. The log odds ratio,2
the number of times that it does not. The table shows however, can take any value and has an approximately
BMJ 2000;320:1468
data from a cross sectional study showing the Normal distribution. It also has the useful property that
prevalence of hay fever and eczema in 11 year old if we reverse the order of the categories for one of the
children.1 The probability that a child with eczema will variables, we simply reverse the sign of the log odds
also have hay fever is estimated by the proportion ratio: log(4.89) = 1.59, log(0.204) = 1.59.
141/561 (25.1%). The odds is estimated by 141/420. We can calculate a standard error for the log odds
Similarly, for children without eczema the probability ratio and hence a confidence interval. The standard
of having hay fever is estimated by 928/14 453 (6.4%) error of the log odds ratio is estimated simply by the
and the odds is 928/13 525. We can compare the square root of the sum of the reciprocals of the four
frequencies. For the example,
groups in several ways: by the difference between the
proportions, 141/561 928/14 453 = 0.187 (or 18.7
percentage points); the ratio of the proportions, (141/
561)/(928/14 453) = 3.91 (also called the relative
risk); or the odds ratio, (141/420)/(928/
13 525) = 4.89.
A 95% confidence interval for the log odds ratio is
obtained as 1.96 standard errors on either side of the
estimate. For the example, the log odds ratio is
Association between hay fever and eczema in 11 year old
loge(4.89) = 1.588 and the confidence interval is
children1
1.5881.960.103, which gives 1.386 to 1.790. We can
Hay fever antilog these limits to give a 95% confidence interval
Eczema Yes No Total for the odds ratio itself,2 as exp(1.386) = 4.00 to
Yes 141 420 561 exp(1.790) = 5.99. The observed odds ratio, 4.89, is not
No 928 13 525 14 453 in the centre of the confidence interval because of the
Total 1069 13 945 15 522
asymmetrical nature of the odds ratio scale. For this
reason, in graphs odds ratios are often plotted using a
logarithmic scale. The odds ratio is 1 when there is no
Now, suppose we look at the table the other way relationship. We can test the null hypothesis that the
round, and ask what is the probability that a child with odds ratio is 1 by the usual 2 test for a two by two
hay fever will also have eczema? The proportion is table.
141/1069 (13.2%) and the odds is 141/928. For a Despite their usefulness, odds ratios can cause diffi-
child without hay fever, the proportion with eczema is culties in interpretation.3 We shall review this debate
420/13 945 (3.0%) and the odds is 420/13 525. Com- and also discuss odds ratios in logistic regression and
paring the proportions this way, the difference is 141/ case-control studies in future Statistics Notes.
1069 420/13 945 = 0.102 (or 10.2 percentage We thank Barbara Butland for providing the data.
points); the ratio (relative risk) is (141/1069)/(420/
13 945) = 4.38; and the odds ratio is (141/928)/(420/ 1 Strachan DP, Butland BK, Anderson HR. Incidence and prognosis of
asthma and wheezing illness from early childhood to age 33 in a national
13 525) = 4.89. The odds ratio is the same whichever British cohort. BMJ. 1996;312:1195-9.
2 Bland JM, Altman DG. Transforming data. BMJ 1996;312:770.
way round we look at the table, but the difference and 3 Sackett DL, Deeks JJ, Altman DG. Down with odds ratios! Evidence-Based
ratio of proportions are not. It is easy to see why this is. Med 1996;1:164-6.

1468 BMJ VOLUME 320 27 MAY 2000 bmj.com

Education and debate
Statistics Notes
Blinding in clinical trials and other studies
Simon J Day, Douglas G Altman
Leo Human behaviour is influenced by what we know or
Pharmaceuticals,
Princes Risborough,
believe. In research there is a particular risk of expecta-
Buckinghamshire tion influencing findings, most obviously when there is
HP27 9RR some subjectivity in assessment, leading to biased
Simon J Day results. Blinding (sometimes called masking) is used to
manager, clinical
biometrics try to eliminate such bias.
ICRF Medical
It is a tenet of randomised controlled trials that the
Statistics Group, treatment allocation for each patient is not revealed
Institute of Health until the patient has irrevocably been entered into the
Sciences, Oxford
OX3 7LF trial, to avoid selection bias. This sort of blinding, better
Douglas G Altman referred to as allocation concealment, will be discussed
professor of statistics in a future statistics note. In controlled trials the term
in medicine
blinding, and in particular double blind, usually
Correspondence to: refers to keeping study participants, those involved
S J Day
with their management, and those collecting and ana-
lysing clinical data unaware of the assigned treatment,
BMJ 2000;321:504
so that they should not be influenced by that
knowledge.
The relevance of blinding will vary according to
circumstances. Blinding patients to the treatment they
have received in a controlled trial is particularly impor-
tant when the response criteria are subjective, such as
alleviation of pain, but less important for objective cri-
teria, such as death. Similarly, medical staff caring for
patients in a randomised trial should be blinded to
treatment allocation to minimise possible bias in
patient management and in assessing disease status.
For example, the decision to withdraw a patient from a
study or to adjust the dose of medication could easily
be influenced by knowledge of which treatment group
the patient has been assigned to.
In a double blind trial neither the patient nor the
caregivers are aware of the treatment assignment.
Blinding means more than just keeping the name of
the treatment hidden. Patients may well see the
treatment being given to patients in the other
treatment group(s), and the appearance of the drug
used in the study could give a clue to its identity. Differ-
ences in taste, smell, or mode of delivery may also
influence efficacy, so these aspects should be identical
for each treatment group. Even colour of medication
has been shown to influence efficacy.1
In studies comparing two active compounds, blind-
ing is possible using the double dummy method. For
example, if we want to compare two medicines, one
presented as green tablets and one as pink capsules, we
could also supply green placebo tablets and pink
placebo capsules so that both groups of patients would
take one green tablet and one pink capsule.
Blinding is certainly not always easy or possible.
Single blind trials (where either only the investigator or
only the patient is blind to the allocation) are
sometimes unavoidable, as are open (non-blind) trials.
In trials of different styles of patient management,
surgical procedures, or alternative therapies, full blind-
ing is often impossible.
In a double blind trial it is implicit that the
assessment of patient outcome is done in ignorance of
504
the treatment received. Such blind assessment of
outcome can often also be achieved in trials which are
open (non-blinded). For example, lesions can be
photographed before and after treatment and assessed
by someone not involved in running the trial. Indeed,
blind assessment of outcome may be more important
than blinding the administration of the treatment,
especially when the outcome measure involves subjec-
tivity. Despite the best intentions, some treatments have
unintended effects that are so specific that their occur-
rence will inevitably identify the treatment received to
both the patient and the medical staff. Blind
assessment of outcome is especially useful when this is
a risk.
In epidemiological studies it is preferable that the
identification of cases as opposed to controls be
kept secret while researchers are determining each
subjects exposure to potential risk factors. In many
such studies blinding is impossible because exposure
can be discovered only by interviewing the study
participants, who obviously know whether or not they
are a case. The risk of differential recall of important
disease related events between cases and controls must
then be recognised and if possible investigated.2 As a
minimum the sensitivity of the results to differential
recall should be considered. Blinded assessment of
patient outcome may also be valuable in other
epidemiological studies, such as cohort studies.
Blinding is important in other types of research
too. For example, in studies to evaluate the perform-
ance of a diagnostic test those performing the test must
be unaware of the true diagnosis. In studies to evaluate
the reproducibility of a measurement technique the
observers must be unaware of their previous measure-
ment(s) on the same individual.
We have emphasised the risks of bias if adequate
blinding is not used. This may seem to be challenging
the integrity of researchers and patients, but bias asso-
ciated with knowing the treatment is often subcon-
scious. On average, randomised trials that have not
used appropriate levels of blinding show larger
treatment effects than blinded studies.3 Similarly, diag-
nostic test performance is overestimated when the ref-
erence test is interpreted with knowledge of the test
result.4 Blinding makes it difficult to bias results
intentionally or unintentionally and so helps ensure
the credibility of study conclusions.
1 De Craen AJM, Roos PJ, de Vries AL, Kleijnen J. Effect of colour of drugs:
systematic review of perceived effect of drugs and their effectiveness. BMJ
1996;313:1624-6.
2 Barry D. Differential recall bias and spurious associations in case/control
studies. Stat Med 1996;15:2603-16.
3 Schulz KF, Chalmers I, Hayes R, Altman DG. Empirical evidence of bias:
dimensions of methodological quality associated with estimates of treat-
ment effects in controlled trials. JAMA 1995;273:408-12.
4 Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der
Meulen JH, et al. Empirical evidence of design-related bias in studies of
diagnostic tests. JAMA 1999;282:1061-6.
BMJ VOLUME 321 19-26 AUGUST 2000 bmj.com
Education and debate
Irrationality, the market, and quality of care
Consider the irrationality of a person who pays extra
so as not to share a hotel room with a colleague while
on a business trip. He does this because he values
privacy but he also scoffs at taking out long term care
insurance to guarantee a private room in a nursing
home. Why is he willing to risk sharing a room for the
rest of his life with a person he does not like? This
common irrationality is often masked by
rationalisations such as I would rather die than have
to live in a nursing home. Yet we know that when the
time comes most prefer the limited pleasures of life in
a nursing home to suicide
their feet. There are even more fundamental reasons
why depending on the rationality of the market will
never work well for quality of care (box). Sensible
policy for providing nursing home care requires a
larger welfare state, a larger regulatory state, and
encouragement of public, non-profit providers.
Australias recent experience shows that to head in the
opposite direction is medically, economically, and
politically irrational.
Statistics Notes
Concealing treatment allocation in randomised trials
Douglas G Altman, Kenneth F Schulz
ICRF Medical We have previously explained why random allocation
Statistics Group,
Centre for Statistics of treatments is a required design feature of controlled
in Medicine, trials1 and explained how to generate a random alloca-
Institute of Health tion sequence.2 Here we consider the importance of
Sciences, Oxford
OX3 7LF concealing the treatment allocation until the patient is
Douglas G Altman entered into the trial.
professor of statistics Regardless of how the allocation sequence has
in medicine
been generatedsuch as by simple or stratified
Family Health
International, randomisation2there will be a prespecified sequence
PO Box 13950, of treatment allocations. In principle, therefore, it is
Research Triangle possible to know what treatment the next patient will
Park, NC 27709,
USA get at the time when a decision is taken to consider the
Kenneth F Schulz patient for entry into the trial.
vice president, The strength of the randomised trial is based on
Quantitative Sciences
aspects of design which eliminate various types of bias.
Correspondence to:
D G Altman
Randomisation of patients to treatment groups
eliminates bias by making the characteristics of the
BMJ 2001;323:4467 patients in two (or more) groups the same on average,
and stratification with blocking may help to reduce
chance imbalance in a particular trial.2 All this good
work can be undone if a poor procedure is adopted to
implement the allocation sequence. In any trial one or
more people must determine whether each patient is
eligible for the trial, decide whether to invite the
patient to participate, explain the aims of the trial and
the details of the treatments, and, if the patient agrees
to participate, determine what treatment he or she will
receive.
Suppose it is clear which treatment a patient will
receive if he or she enters the trial (perhaps because
446
Competing interests: None declared.
1 McCallum J, Geiselhart K. Australias new aged: issues for young and old.
Sydney: Allen and Unwin, 1996.
2 Osborne D, Gaebler T. Reinventing government. New York: Addison-
Wesley, 1992.
3 Jost T. The necessary and proper role of regulation to assure the quality
of health care. Houston Law Review 1988;25:525-98.
4 Tingle L. Moran the big winner as aged care goes private. Sydney Morning
Herald 16 March 2001:2.
5 Lohr R, Head M. Kerosene baths reveal systemic aged care crisis in Aus-
tralia. World Socialist Web Site. www.wsws.org/articles/2000/mar2000/
aged-m10.shtml (accessed 10 Mar 2000).
6 Jenkins A, Braithwaite J. Profits, pressure and corporate lawbreaking.
Crime, Law and Social Change 1993;20:221-32.
7 Braithwaite J, Makkai T, Braithwaite V, Gibson D. Raising the standard: resi-
dent centred nursing home regulation in Australia. Canberra: Department of
Community Services and Health, 1993.
8 Braithwaite J, Braithwaite V. The politics of legalism: rules versus
standards in nursing home regulation. Social and Legal Studies
1995;4:307-41.
9 Black J. Rules and regulators. Oxford: Clarendon Press, 1997.
10 Braithwaite J, Makkai T. Can resident-centred inspection of nursing
homes work with very sick residents? Health Policy 1993;24:19-33.
11 Makkai T, Braithwaite J. Praise, pride and corporate compliance. Int J Soci-
ology Law 1993;21:73-91.
12 Braithwaite J. Restorative justice and responsive regulation. New York: Oxford
University Press (in press).
13 McKibbin H. Accreditation: the on-site audit. The Standard (Newsletter of
the Aged Care Standards Agency) 1999;2(2):2.
14 Power M. The audit society. Oxford: Oxford University Press, 1997.
there is a typed list showing the allocation sequence).
Each of the above steps may then be compromised
because of conscious or subconscious bias. Even when
the sequence is not easily available, there is strong
anecdotal evidence of frequent attempts to discover
the sequence through a combination of a misplaced
belief that this will be beneficial to patients and lack of
understanding of the rationale of randomisation.3
How can the allocation sequence be concealed?
Firstly, the person who generates the allocation
sequence should not be the person who determines
eligibility and entry of patients. Secondly, if possible the
mechanism for treatment allocation should use people
not involved in the trial. A common procedure,
especially in larger trials, is to use a central telephone
randomisation system. Here patient details are
supplied, eligibility confirmed, and the patient entered
into the trial before the treatment allocation is divulged
(and it may still be blinded4). Another excellent alloca-
tion concealment mechanism, common in drug trials,
is to get the allocation done by a pharmacy. The inter-
ventions are sealed in serially numbered containers
(usually bottles) of equal appearance and weight
according to the allocation sequence.
If external help is not available the only other
system that provides a plausible defence against alloca-
tion bias is to enclose assignments in serially
numbered, opaque, sealed envelopes. Apart from
neglecting to mention opacity, this is the method used
in the famous 1948 streptomycin trial (see box). This
BMJ VOLUME 323 25 AUGUST 2001 bmj.com

Description of treatment allocation in the MRC
streptomycin trial5
Determination of whether a patient would be treated
by streptomycin and bed-rest (S case) or by bed-rest
alone (C case) was made by reference to a statistical
series based on random sampling numbers drawn up
for each sex at each centre by Professor Bradford Hill;
the details of the series were unknown to any of the
investigators or to the co-ordinator and were
contained in a set of sealed envelopes, each bearing on
the outside only the name of the hospital and a
number. After acceptance of a patient by the panel,
and before admission to the streptomycin centre, the
appropriate numbered envelope was opened at the
central office; the card inside told if the patient was to
be an S or a C case, and this information was then
given to the medical officer of the centre.
method is not immune to corruption,3 particularly if
poorly executed. However, with care, it can be a good
mechanism for concealing allocation. We recommend
that investigators ensure that the envelopes are opened
sequentially, and only after the participants name and
other details are written on the appropriate envelope.3
If possible, that information should also be transferred
to the assigned allocation by using pressure sensitive
paper or carbon paper inside the envelope. If an inves-
tigator cannot use numbered containers, envelopes
represent the best available allocation concealment
mechanism without involving outside parties, and may
sometimes be the only feasible option. We suspect,
however, that in years to come we will see greater use of
external third party randomisation.
The public health benefits of mobile phones
The bread and butter of public health on call is
identifying contacts in the case of suspected
meningococcal disease. On the whole this is
straightforward but can occasionally cause difficulties.
Most areas that I have worked in include several
universities, and during October it is common to
experience the problem of contact tracing in the
student population.
There are two main problems. The first is how to
define household contacts when the index patient lives
in a hall of residence containing several hundred
students. Finding the appropriate university protocol
and not being too concerned about the different
approaches adopted by neighbouring universities can
reduce the number of sleepless nights. The second
problem is harder. Close kissing contacts among 18
year olds who have been set free from parental control
for the first time is a minefield. My experience suggests
that it is best to assume there will be lots and that names
and contact details will not necessarily have been
obtained. By the end of a weekend on call, you will feel
like a cross between a detective and an agony aunt.
One year I volunteered to cover Christmas weekend
in the belief that at least the students would be gone by
then. I could not have been more mistaken. To add a
further difficulty, the index patient presented to
hospital on the night of the last day of term, and all
contacts had already set off to the far reaches of the
country. I could not believe my luck when the friend
BMJ VOLUME 323 25 AUGUST 2001 bmj.com
Education and debate
The desirability of concealing the allocation was
recognised in the streptomycin trial5 (see box). Yet the
importance of this key element of a randomised trial
has not been widely recognised. Empirical evidence of
the bias associated with failure to conceal the
allocation6 7 and explicit requirement to discuss this
issue in the CONSORT statement8 seem to be leading
to wider recognition that allocation concealment is an
essential aspect of a randomised trial.
Allocation concealment is completely different
from (double) blinding.4 It is possible to conceal the
randomisation in every randomised trial. Also,
allocation concealment seeks to eliminate selection
bias (who gets into the trial and the treatment they are
assigned). By contrast, blinding relates to what happens
after randomisation, is not possible in all trials, and
seeks to reduce ascertainment bias (assessment of
outcome).
1 Altman DG, Bland JM. Treatment allocation in controlled trials: why ran-
domise? BMJ 1999;318:1209.
2 Altman DG, Bland JM. How to randomise. BMJ 1999;319:703-4.
3 Schulz KF. Subverting randomization in controlled trials. JAMA
1995;274:1456-8.
4 Day SJ, Altman DG. Blinding in clinical trials and other studies. BMJ
2000;321:504.
5 Medical Research Council. Streptomycin treatment of pulmonary
tuberculosis: a Medical Research Council investigation. BMJ 1948;2:
769-82.
6 Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does
quality of reports of randomised trials affect estimates of intervention
efficacy reported in meta-analyses. Lancet 1998;352:609-13.
7 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias:
dimensions of methodological quality associated with estimates of treat-
ment effects in controlled trials. JAMA 1995;273:408-12.
8 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improv-
ing the quality of reporting of randomized controlled trials: the
CONSORT statement. JAMA 1996;276:637-9.
accompanying the patient produced both their mobile
phones and confidently reassured me that between the
two of them they would have the mobile numbers of
all 15 household contacts. She was right, and in just
over two hours all of them had been contacted.
There has been much coverage in the medical and
popular press about the potential health hazards of
mobile phones, and if these fears are realised the 100%
ownership among this small sample of students is
worrying. However, in terms of contact tracing for
suspected meningococcal disease, mobile phones have
potential health benefits not just for their owners but
also for the mental health of public health doctors. Of
course, this may not solve the close kissing contact
problem.
Debbie Lawlor senior lecturer in epidemiology and public
health, University of Bristol
We welcome articles up to 600 words on topics such as
A memorable patient, A paper that changed my practice, My
most unfortunate mistake, or any other piece conveying
instruction, pathos, or humour. If possible the article
should be supplied on a disk. Permission is needed
from the patient or a relative if an identifiable patient is
referred to. We also welcome contributions for
Endpieces, consisting of quotations of up to 80 words
(but most are considerably shorter) from any source,
ancient or modern, which have appealed to the reader.
447
 Education and debate

Statistics Notes
Analysing controlled trials with baseline and follow up
measurements
Andrew J Vickers, Douglas G Altman

In many randomised trials researchers measure a con-
tinuous variable at baseline and again as an outcome
assessed at follow up. Baseline measurements are com-
mon in trials of chronic conditions where researchers
want to see whether a treatment can reduce
pre-existing levels of pain, anxiety, hypertension, and
the like.
Statistical comparisons in such trials can be made
in several ways. Comparison of follow up (post-
treatment) scores will give a result such as at the end
of the trial, mean pain scores were 15 mm (95% confi-
dence interval 10 to 20 mm) lower in the treatment
group. Alternatively a change score can be calculated
by subtracting the follow up score from the baseline
score, leading to a statement such as pain reductions
were 20 mm (16 to 24 mm) greater on treatment than
control. If the average baseline scores are the same in
each group the estimated treatment effect will be the
same using these two simple approaches. If the
treatment is effective the statistical significance of the
treatment effect by the two methods will depend on the
correlation between baseline and follow up scores. If
the correlation is low using the change score will add
variation and the follow up score is more likely to show
a significant result. Conversely, if the correlation is high
using only the follow up score will lose information
and the change score is more likely to be significant. It
is incorrect, however, to choose whichever analysis
gives a more significant finding. The method of analy-
sis should be specified in the trial protocol.
Some use change scores to take account of chance
imbalances at baseline between the treatment groups.
However, analysing change does not control for
baseline imbalance because of regression to the
mean1 2: baseline values are negatively correlated with
change because patients with low scores at baseline
generally improve more than those with high scores. A
better approach is to use analysis of covariance
(ANCOVA), which, despite its name, is a regression
method.3 In effect two parallel straight lines (linear
regression) are obtained relating outcome score to
baseline score in each group. They can be summarised
as a single regression equation:
follow up score =
constant + abaseline score + bgroup
where a and b are estimated coefficients and group Integrative
Medicine Service,
is a binary variable coded 1 for treatment and 0 for Biostatistics Service,
control. The coefficient b is the effect of interestthe Memorial
estimated difference between the two treatment Sloan-Kettering
Cancer Center, New
groups. In effect an analysis of covariance adjusts each York, New York
patients follow up score for his or her baseline score, 10021, USA
but has the advantage of being unaffected by baseline Andrew J Vickers
assistant attending
differences. If, by chance, baseline scores are worse in research methodologist
the treatment group, the treatment effect will be
ICRF Medical
underestimated by a follow up score analysis and over- Statistics Group,
estimated by looking at change scores (because of Centre for Statistics
regression to the mean). By contrast, analysis of covari- in Medicine,
Institute of Health
ance gives the same answer whether or not there is Sciences, Oxford
baseline imbalance. OX3 7LF
As an illustration, Kleinhenz et al randomised 52 Douglas G Altman
professor of statistics
patients with shoulder pain to either true or sham acu- in medicine
puncture.4 Patients were assessed before and after
Correspondence to:
treatment using a 100 point rating scale of pain and Dr Vickers
function, with lower scores indicating poorer outcome. vickersa@mskcc.org
There was an imbalance between groups at baseline,
BMJ 2001;323:11234
with better scores in the acupuncture group (see table).
Analysis of post-treatment scores is therefore biased.
The authors analysed change scores, but as baseline
and change scores are negatively correlated (about
r = 0.25 within groups) this analysis underestimates
the effect of acupuncture. From analysis of covariance
we get:
follow up score =
24 + 0.71baseline score + 12.7group
(see figure). The coefficient for group (b) has a use-
ful interpretation: it is the difference between the mean
change scores of each group. In the above example it
can be interpreted as pain and function score
improved by an estimated 12.7 points more on average
in the treatment group than in the control group. A
95% confidence interval and P value can also be calcu-
lated for b (see table).5 The regression equation
provides a means of prediction: a patient with a
baseline score of 50, for example, would be predicted
to have a follow up score of 72.2 on treatment and 59.5
on control.
An additional advantage of analysis of covariance is
that it generally has greater statistical power to detect a
treatment effect than the other methods.6 For example,
a trial with a correlation between baseline and follow

Results of trial of acupuncture for shoulder pain4

Pain scores (mean and SD)
Placebo group Acupuncture group Difference between means
(n=27) (n=25) (95% CI) P value
Baseline 53.9 (14) 60.4 (12.3) 6.5
Analysis
Follow up 62.3 (17.9) 79.6 (17.1) 17.3 (7.5 to 27.1) 0.0008
Change score* 8.4 (14.6) 19.2 (16.1) 10.8 (2.3 to 19.4) 0.014
ANCOVA 12.7 (4.1 to 21.3) 0.005
*Analysis reported by authors.4

BMJ VOLUME 323 10 NOVEMBER 2001 bmj.com 1123

Education and debate
Posttreatment score
100
80
60
40
Acupuncture
Placebo
20
20 40 60 80
Pretreatment score
Pretreatment and post-treatment scores in each group showing fitted
lines. Squares show mean values for the two groups. The estimated
difference between the groups from analysis of covariance is the
vertical distance between the two lines
up scores of 0.6 that required 85 patients for analysis of
follow up scores, would require 68 for a change score
analysis but only 54 for analysis of covariance.
The efficiency gains of analysis of covariance com-
pared with a change score are low when there is a high
correlation (say r > 0.8) between baseline and follow up
measurements. This will often be the case, particularly
in stable chronic conditions such as obesity. In these
A memorable patient
Informed consent
I first met Ivy three years ago when she came for her
29th oesophageal dilatation. She was an 86 year old
spinster, deaf without speech from childhood, and the
only sign language she knew was thumbs up, which she
would use for saying good morning or for showing
happiness. She had no next of kin and had lived in a
residential home for the past 50 years. She developed a
benign oesophageal stricture in 1992 and came to the
endoscopy unit for repeated dilatations. The carers in
the residential home used to say that she enjoyed her
days out at the endoscopy unit.
We would explain the procedure to her in sign
language. She would use the thumbs up sign and make
a cross on the dotted line on the consent form. She
would enter the endoscopy room smiling, put her left
arm out to be cannulated, turn to her left side for
endoscopy, and when fully awake would show her
thumbs up again. Every time after her dilatation the
nursing staff would question why an expandable
oesophageal stent was not being considered. We would
conclude that the indications for an expandable stent
in benign strictures are not well established.
Her need for dilatation was becoming more
frequent, and so on her 46th dilatation we decided to
refer her to our regional centre for the insertion of a
stent. She had an expandable stent inserted, and in his
report the endoscopist mentioned the risk of the stent
migrating down in the stomach beyond the stricture.
Six weeks later she developed a bolus obstruction. At
endoscopy it was noted that the stent had indeed
migrated down. She consented to another stent. Four
weeks later she had another bolus obstruction that
could not be completely removed at the first attempt,
1124
situations, analysis of change scores can be a
reasonable alternative, particularly if restricted
randomisation is used to ensure baseline comparability
between groups.7 Analysis of covariance is the
preferred general approach, however.
As with all analyses of continuous data, the use of
analysis of covariance depends on some assumptions
that need to be tested. In particular, data transforma-
tion, such as taking logarithms, may be indicated.8
Lastly, analysis of covariance is a type of multiple
regression and can be seen as a special type of adjusted
analysis. The analysis can thus be expanded to include
additional prognostic variables (not necessarily con-
tinuous), such as age and diagnostic group.
100
We thank Dr J Kleinhenz for supplying the raw data from his
study.
1 Bland JM, Altman DG. Regression towards the mean. BMJ
1994;308:1499.
2 Bland JM, Altman DG. Some examples of regression towards the mean.
BMJ 1994;309:780.
3 Senn S. Baseline comparisons in randomized clinical trials. Stat Med
1991;10:1157-9.
4 Kleinhenz J, Streitberger K, Windeler J, Gussbacher A, Mavridis G, Mar-
tin E. Randomised clinical trial comparing the effects of acupuncture and
a newly designed placebo needle in rotator cuff tendonitis. Pain
1999;83:235-41.
5 Altman DG, Gardner MJ. Regression and correlation. In: Altman DG,
Machin D, Bryant TN, Gardner MJ, eds. Statistics with confidence. 2nd ed.
London: BMJ Books, 2000:73-92.
6 Vickers AJ. The use of percentage change from baseline as an outcome in
a controlled trial is statistically inefficient: a simulation study. BMC Med
Res Methodol 2001;1:16.
7 Altman DG, Bland JM. How to randomise. BMJ 1999;319:703-4.
8 Bland JM, Altman DG. The use of transformation when comparing two
means. BMJ 1996;312:1153.
and she was brought back the following day for
removal of the bolus by endoscopy.
She came to the endoscopy room but did not have
her familiar smile. She looked around for a minute, got
off her trolley, and walked out. Everyone in the
endoscopy room understood that she was trying to say,
Ive had enough.
She did not come back for a repeat endoscopy, and
she stayed nil by mouth on intravenous fluids. Two
weeks later she died of an aspiration pneumonia. We
think she understood all the procedures she had
agreed to. We also think it was informed consent. I
hope we were right. She gave us a very clear message
without saying a word on her last visit to the
endoscopy room.
Do we really understand what aphasic patients are
trying to tell us when we get informed consent for
invasive procedures? We should try to read the
non-verbal messages very carefully.
I Tiwari associate specialist in gastroenterology, Broomfield
Hospital, Chelmsford
We welcome articles of up to 600 words on topics
such as A memorable patient, A paper that changed my
practice, My most unfortunate mistake, or any other piece
conveying instruction, pathos, or humour. If possible
the article should be supplied on a disk. Permission is
needed from the patient or a relative if an identifiable
patient is referred to. We also welcome contributions
for Endpieces, consisting of quotations of up to 80
words (but most are considerably shorter) from any
source, ancient or modern, which have appealed to
the reader.
BMJ VOLUME 323 10 NOVEMBER 2001 bmj.com
Education and debate
The quality and reliability of health information on
the internet remains of paramount concern in Europe,
as elsewhere. Self regulatory codes of ethics for health
websites abound, yet the quality and practices of many
are highly questionable.
Little progress seems to have been made,
moreover, in assuring consumers that the information
they share with health websites will not be misused.
Several US studies have already concluded that
websites privacy practices do not match their
proclaimed policies.5 In an attempt to counter this ero-
sion of trust in Europe, the European Commissions
guidelines for quality criteria for health related
websites have recognised that there is no shortage of
legislation in the field of privacy and security.6 They
have drawn specific attention to a new recommen-
dation regarding online data collection adopted in
May 2001 that explains how European directives on
issues such as data protection should be applied to the
most common processing tasks carried out via the
internet.7
The challenge facing Europes health professionals
and policymakers is to carefully craft the development
of new approaches to the supervision of medical and
pharmaceutical practice. Their ultimate goal is to raise
Statistics Notes
Validating scales and indexes
J Martin Bland, Douglas G Altman
Papers p 569
An index of quality is a measurement like any other,
whether it is assessing a website, as in todays BMJ,1 a
clinical trial used in a meta-analysis,2 or the quality of a
Department of life experienced by a patient.3 As with all measure-
Public Health ments, we have to decide whether it measures what we
Sciences, St
Georges Hospital want it to measure, and how well.
Medical School, The simplest measurements, such as length and
London SW17 0RE
distance, can be validated by an objective criterion. The
J Martin Bland
professor of medical earliest criteria must have been biological: the length of
statistics a pace, a foot, a thumb. The obvious problem, that the
Cancer Research criterion varies from person to person, was eventually
UK Medical solved by establishing a fundamental unit and defining
Statistics Group,
Centre for Statistics all others in terms of it. Other measurements can then
in Medicine, be defined in terms of a fundamental unit. To define a
Institute for Health
Sciences, Oxford
unit of weight we find a handy substance which
OX3 7LF appears the same everywhere, such as water. The unit
Douglas G Altman of weight is then the weight of a volume of water speci-
professor of statistics
in medicine
fied in the basic unit of length, such as 100 cubic centi-
metres. Such measurements have criterion validity,
Correspondence to:
Professor Bland meaning that we can take some known quantity and
mbland@sghms.ac.uk compare our measurement with it.
For some measurements no such standard is possi-
BMJ 2002;324:6067 ble. Cardiac stroke volume, for example, can be
measured only indirectly. Direct measurement, by
collecting all the blood pumped out of the heart over a
series of beats, would involve rather drastic interference
with the system. Our criterion becomes agreement
with another indirect measurement. Indeed, we some-
times have to use as a standard a method which we
know produces inaccurate measurements.
606
consumers confidence in online healthcare. They must
ensure that the mechanisms are put in place whereby
health professionals themselves can benefit from using
the internet, while still ensuring the highest standards
of medical practice.
Avienda was formerly known as the Centre for Law Ethics and
Risk in Telemedicine.
Competing interests: None declared.
1 http://news.bbc.co.uk/hi/english/uk/england/newsid_1752000/
1752670.stm (accessed 5 Feb 2002).
2 Case C-322/01: Reference for a preliminary ruling by the Landgericht
Frankfurt am Main by order of that court of 10 August 2001 in the case
of Deutscher Apothekerverband e.V. against DocMorris NV and Jacques
Waterval. Official Journal of the European Communities No C 2001
December 8:348/10.
3 Council Directive 1992/28/EEC of 31 March 1992 on the advertising of
medicinal products for human use. (Articles 1(3) and 3(1).) Official Journal
of the European Communities No L 1995 11 February:32/26.
4 Directive 2000/31/EC on mutual recognition of primary medical and
specialist medical qualifications and minimum standards of training. Offi-
cial Journal of the European Communities No L 2001 July 31:206/1-51.
5 Schwartz J. Medical websites faulted on privacy. Washington Post 2000
February 1.
6 http://europa.eu.int/information_society/eeurope/ehealth/quality/
draft_guidelines/index_en.htm (accessed 5 Feb 2002).
7 European Commission. Recommendation 2/2001 on certain minimum
requirements for collecting personal data on-line in the European
Union. Adopted on 17 May 2001. http://europa.eu.int/comm/
internal_market/en/dataprot/wpdocs/wp43en.htm (accessed 25 Jan
2002).
Some quantities are even more difficult to measure
and evaluate. Cardiac stroke volume does at least have
an objective reality; a physical quantity of blood is
pumped out of the heart when it beats. Anxiety and
depression do not have a physical reality but are useful
artificial constructs. They are measured by question-
naire scales, where answers to a series of questions
related to the concept we want to measure are
combined to give a numerical score. Website quality is
similar. We are measuring a quantity which is not pre-
cisely defined, and there is no instrument with which
we can compare any measure we might devise. How
are we to assess the validity of such a scale?
The relevant theory was developed in the social sci-
ences in the context of questionnaire scales.4 First we
might ask whether the scale looks right, whether it asks
about the sorts of thing which we think of as being
related to anxiety or website quality. If it appears to be
correct, we call this face validity. Next we might ask
whether it covers all the aspects which we want to
measure. A phobia scale which asked about fear of
dogs, spiders, snakes, and cats but ignored height, con-
fined spaces, and crowds would not do this. We call
appropriate coverage of the subject matter content
validity.
Our scale may look right and cover the right things,
but what other evidence can we bring to the question
of validity? One question we can ask is whether our
score has the relationships with other variables that we
would expect. For example, does an anxiety measure
BMJ VOLUME 324 9 MARCH 2002 bmj.com

distinguish between psychiatric patients and medical
patients? Do we get different anxiety scores from
students before and after an examination? Does a
measure of depression predict suicide attempts? We
call the property of having appropriate relationships
with other variables construct validity.
We can also ask whether the items which together
compose the scale are related to one another: does the
scale have internal consistency? If not, do the items really
measure the same thing? On the other hand, if the
items are too similar, some may be redundant. Highly
correlated items in a scale may make the scale over-
long and may lead to some aspects being overempha-
sised, impairing the content validity. A handy summary
measure for this feature is Cronbachs alpha.5
A scale must also be repeatable and be sufficiently
objective to give similar results for different observers.
If a measurement is repeatable, in that someone who
has a high score on one occasion tends to have a high
score on another, it must be measuring something.
With physical measurements, it is often possible for the
same observer (or different observers) to make
repeated measurements in quick succession. When
there is a subjective element in the measurement the
observer can be blinded from their first measurement,
Honour a physician with the honour due unto him
A few years ago my general practitioner told me that
anyone aged over 40 with upper abdominal discomfort
needed investigating. At the local teaching hospital, a
pleasant young doctor did a gastroscopy, which
showed a mass in my stomach wall. I was sent for a
barium meal. A consultant radiologist took the x ray
films, instructing me briskly to turn this way and that
but not otherwise paying me any attention. He told me
to wait a few minutes while he checked the films to see
if all the views were satisfactory. I sat alone in the room
for about five minutes.
From the moment the consultant re-entered I could
see that he was slightly agitated. Im terribly sorry, he
called out as he came through the door at the far end.
And then again, Im terribly sorry. Perhaps these
words of regret, coupled with the concern on his face,
might not have had the effect they did had I not been a
man with an abdominal mass on his mind. At this
moment of truth and reckoning, certain visions swam
before my eyes.
Three strides later, he was in front of me and
looking me full in the face: Im terribly sorry, I hadnt
realised you were a doctor. In his hand was the request
form, and I could see that my general practitioner had
written ex-SR here in one corner. He must have
spotted this when checking the form as he looked at
the preliminary plates. Though no further x rays were
needed, he proceeded a little breathlessly to deliver
three or four minutes of almost a caricature of caring,
empathic interest in a patient. What branch of
medicine was I in, and where did I work? Good
heavens, that must be tough. Is that an Australian
accent I hear? A St Marys old boy, ah yes. What did I
think about. . .?
I dont mean to imply that this was insincere, merely
splendidly different from his earlier matter of factness
and economy of word. I had thought nothing of this at
the time: in such a bread and butter procedure I had
BMJ VOLUME 324 9 MARCH 2002 bmj.com
Education and debate
and different observers can make simultaneous
measurements. In assessing the reliability of a website
quality scale, it is easy to get several observers to apply
the scale independently. With websites, repeat assess-
ments need to be close in time because their content
changes frequently (as does bmj.com). With question-
naires, either self administered or recorded by an
observer, repeat measurements need to be far enough
apart in time for the earlier responses to be forgotten,
yet not so far apart that the underlying quantity being
measured might have changed. Such data enable us to
evaluate test-retest reliability. If two measures have com-
parable face, content, and construct validity the more
repeatable one may be preferred for the study of a
given population.
1 Gagliardi A, Jadad AR. Examination of instruments used to rate quality of
health information on the internet: chronicle of a voyage with an unclear
destination. BMJ 2002;324:569-73.
2 Jni P, Altman DG, Egger M. Assessing the quality of controlled clinical
trials. BMJ 2001;323:42-6.
3 Muldoon MF, Barger SD, Flory JD, Manuck B. What are quality of life
measurements measuring? BMJ 1998;316:542-5.
4 Streiner DL, Norman GR. Health measurement scales: a practical
guide to their development and use. 2nd ed. Oxford: Oxford University Press,
1996.
5 Bland JM, Altman DG. Cronbachs alpha. BMJ 1997;314:572.
no more reason to expect the doctor to engage with
me as a person than I would the phlebotomist taking a
routine blood sample. Clearly, this consultant saw
things similarly as a rule, but when the patient was a
doctor the aesthetics of the encounter changed. He
had apologised three times for what he felt was a lapse
on his part, arising from his failure to notice what was
written in the corner of the request form. Perhaps he
thought I knew that my general practitioner had
written this and that I expected this of a medical
referral, and thus expected to be recognised by him
not just as a patient but also as a colleague. He seemed
to see this as my due. (As it happens, I didnt.)
I had forgotten this incident, but it was brought back
to me by the aftermath of the Bristol cardiac surgery
debacle, and by the publicity surrounding other recent
medical scandals. These have all put a spotlight on
relations between doctors, who seem to offer each
other acknowledgement and empathy, as my
consultant had sought belatedly to do to me. The
general public may be coming to suspect that this
collegiate solidarity is somehow not in their interests,
associating it with mutual protectiveness and thus with
cover-ups of medical malpractice. It is too soon to say
how the profession will react, but my consultant was an
older man and my guess is that, with younger
generations of doctors, we will see the waning of a
tradition whose roots lie with Hippocrates. For it was
his oath that bound doctors to look well on each other
(and not charge each other for their services).
Its another story, but I found out later that the mass
was the gastroscopy instrument itself distorting the
stomach wall, misdiagnosed by an inexperienced
registrar. No special treatment there, anyway.
Derek Summerfield consultant psychiatrist, CASCAID,
South London and Maudsley NHS Trust, London
607
 Education and debate

Statistics Notes
Interaction revisited: the difference between two estimates
Douglas G Altman, J Martin Bland

We often want to compare two estimates of the same 0.2206 (row 8). From these two values we can test the Cancer Research
UK Medical
quantity derived from separate analyses. Thus we might interaction and estimate the ratio of the relative risks Statistics Group,
want to compare the treatment effect in subgroups in a (with confidence interval). The test of interaction is the Centre for Statistics
randomised trial, such as two age groups. The term for ratio of d to its standard error: z= 0.2726/ in Medicine,
Institute for Health
such a comparison is a test of interaction. In earlier Sta- 0.2206= 1.24, which gives P=0.2 when we refer it to a Sciences, Oxford
tistics Notes we discussed interaction in terms of hetero- table of the normal distribution. The estimated OX3 7LF
geneity of treatment effect.13 Here we revisit interaction interaction effect is exp( 0.2726)=0.76. (This value can Douglas G Altman
professor of statistics
and consider the concept more generally. also be obtained directly as 0.67/0.88=0.76.) The in medicine
The comparison of two estimated quantities, such as confidence interval for this effect is 0.7050 to 0.1598
Department of
means or proportions, each with its standard error, is a on the log scale (row 9). Transforming back to the rela- Public Health
general method that can be applied widely. The two esti- tive risk scale, we get 0.49 to 1.17 (row 12). There is thus Sciences,
mates should be independent, not obtained from the no good evidence to support a different treatment effect St Georges
Hospital Medical
same individualsexamples are the results from in younger and older women. School, London
subgroups in a randomised trial or from two independ- The same approach is used for comparing odds SW17 0RE
ent studies. The samples should be large. If the estimates ratios. Comparing means or regression coefficients is J Martin Bland
professor of medical
are E1 and E2 with standard errors SE(E1) and SE(E2), simpler as there is no log transformation. The two esti- statistics
then the difference d=E1 E2 has standard error mates must be independent: the method should not be
Correspondence to:
SE(d)=[SE(E1)2 + SE(E2)2] (that is, the square root of the used to compare a subset with the whole group, or two D G Altman
sum of the squares of the separate standard errors). This estimates from the same patients. doug.altman@
cancer.org.uk
formula is an example of a well known relation that the There is limited power to detect interactions, even in
variance of the difference between two estimates is the a meta-analysis combining the results from several stud-
BMJ 2003;326:219
sum of the separate variances (here the variance is the ies. As this example illustrates, even when the two
square of the standard error). Then the ratio z=d/SE(d) estimates and P values seem very different the test of
gives a test of the null hypothesis that in the population interaction may not be significant. It is not sufficient for
the difference d is zero, by comparing the value of z to the relative risk to be significant in one subgroup and
the standard normal distribution. The 95% confidence not in another. Conversely, it is not correct to assume
interval for the difference is d1.96SE(d) to d+1.96SE(d). that when two confidence intervals overlap the two esti-
We illustrated this for means and proportions,3 mates are not significantly different.6 Statistical analysis
although we did not show how to get the standard should be targeted on the question in hand, and not
error of the difference. Here we consider comparing based on comparing P values from separate analyses.2
relative risks or odds ratios. These measures are always
analysed on the log scale because the distributions of 1 Altman DG, Matthews JNS. Interaction 1: Heterogeneity of effects. BMJ
1996;313:486.
the log ratios tend to be those closer to normal than of 2 Matthews JNS, Altman DG. Interaction 2: Compare effect sizes not P
the ratios themselves. values. BMJ 1996;313:808.
3 Matthews JNS, Altman DG. Interaction 3: How to examine heterogeneity.
In a meta-analysis of non-vertebral fractures in ran- BMJ 1996;313:862.
domised trials of hormone replacement therapy the 4 Torgerson DJ, Bell-Syer SEM. Hormone replacement therapy and
estimated relative risk from 22 trials was 0.73 (P=0.02) in prevention of nonvertebral fractures. A meta-analysis of randomized
trials. JAMA 2001;285:2891-7.
favour of hormone replacement therapy.4 From 14 trials 5 Bland JM, Altman DG. Logarithms. BMJ 1996;312:700.
of women aged on average < 60 years the relative risk 6 Bland M, Peacock J. Interpreting statistics with confidence. Obstetrician
and Gynaecologist (in press).
was 0.67 (95% confidence interval 0.46 to 0.98; P=0.03).
From eight trials of women aged >60 the relative risk
was 0.88 (0.71 to 1.08; P=0.22). In other words, in Calculations for comparing two estimated relative risks
younger women the estimated treatment benefit was a
Group 1 Group 2
33% reduction in risk of fracture, which was statistically 1 RR 0.67 0.88
significant, compared with a 12% reduction in older 2 *log RR 0.4005 (E1) 0.1278 (E2)
women, which was not significant. But are the relative 3 95% CI for RR 0.46 to 0.98 0.71 to 1.08
risks from the subgroups significantly different from 4 *95% CI for log RR 0.7765 to 0.0202 0.3425 to 0.0770
each other? We show how to answer this question using 5 Width of CI 0.7563 0.4195
just the summary data quoted. 6 SE[=width/(21.96)] 0.1929 0.1070
Because the calculations were made on the log scale, Difference between log relative risks
comparing the two estimates is complex (see table). We 7 d[=E1E2] 0.4005(0.1278)=0.2726
need to obtain the logs of the relative risks and their 8 SE(d) (0.19292+ 0.10702)=0.2206
confidence intervals (rows 2 and 4).5 As 95% confidence 9 CI(d) 0.2726 1.960.2206
or 0.7050 to 0.1598
intervals are obtained as 1.96 standard errors either side
10 Test of interaction z=0.2726/0.2206=1.24 (P=0.2)
of the estimate, the SE of each log relative risk is Ratio of relative risks (RRR)
obtained by dividing the width of its confidence interval 11 RRR=exp(d) exp(0.2726)=0.76
by 21.96 (row 6). The estimated difference in log 12 CI(RRR) exp(0.7050) to exp(0.1598), or 0.49 to 1.17
relative risks is d=E1 E2= 0.2726 and its standard error *Values obtained by taking natural logarithms of values on preceding row.

BMJ VOLUME 326 25 JANUARY 2003 bmj.com 219

 Education and debate

Statistics Notes
The logrank test
J Martin Bland, Douglas G Altman

We often wish to compare the survival experience of Department of

1.0 Health Sciences,

Survival proportion
two (or more) groups of individuals. For example, the Astrocytoma University of York,
table shows survival times of 51 adult patients with Glioblastoma York YO10 5DD
0.8
recurrent malignant gliomas1 tabulated by type of J Martin Bland
tumour and indicating whether the patient had died or professor of health
0.6 statistics
was still alive at analysisthat is, their survival time was
Cancer Research
censored.2 As the figure shows, the survival curves dif- 0.4 UK/NHS Centre
fer, but is this sufficient to conclude that in the popula- for Statistics in
tion patients with anaplastic astrocytoma have worse Medicine, Institute
0.2
of Health Sciences,
survival than patients with glioblastoma? Oxford OX3 7LF
We could compute survival curves3 for each group 0 Douglas G Altman
0 52 104 156 208
and compare the proportions surviving at any specific professor of statistics
Time (weeks) in medicine
time. The weakness of this approach is that it does not
provide a comparison of the total survival experience of Correspondence to:
Survival curves for women with glioma by diagnosis
Professor Bland
the two groups, but rather gives a comparison at some
arbitrary time point(s). In the figure the difference in BMJ 2004;328:1073
survival is greater at some times than others and eventu-
ally becomes zero. We describe here the logrank test, the
2 1 = 1. From a table of the
2 distribution we get
most popular method of comparing the survival of
P < 0.01, so that the difference between the groups is
groups, which takes the whole follow up period into
statistically significant. There is a different method of
account. It has the considerable advantage that it does
calculating the test statistic,4 but we prefer this Weeks to death or
not require us to know anything about the shape of the
approach as it extends easily to several groups. It is censoring in 51
survival curve or the distribution of survival times.
also possible to test for a trend in survival across adults with
The logrank test is used to test the null hypothesis
ordered groups.4 Although we have shown how the recurrent gliomas1
that there is no difference between the populations in (A=astrocytoma,
calculation is made, we strongly recommend the use of
the probability of an event (here a death) at any time G=glioblastoma)
statistical software.
point. The analysis is based on the times of events
The logrank test is based on the same assumptions A G
(here deaths). For each such time we calculate the
as the Kaplan Meier survival curve3namely, that cen- 6 10
observed number of deaths in each group and the 13 10
soring is unrelated to prognosis, the survival probabili-
number expected if there were in reality no difference 21 12
ties are the same for subjects recruited early and late in
between the groups. The first death was in week 6, 30 13
the study, and the events happened at the times speci-
when one patient in group 1 died. At the start of this 31* 14
fied. Deviations from these assumptions matter most if
week, there were 51 subjects alive in total, so the risk of 37 15
they are satisfied differently in the groups being
death in this week was 1/51. There were 20 patients in 38 16
compared, for example if censoring is more likely in
group 1, so, if the null hypothesis were true, the 47* 17
one group than another.
expected number of deaths in group 1 is 20 1/51 = 49 18
The logrank test is most likely to detect a difference 50 20
0.39. Likewise, in group 2 the expected number of
between groups when the risk of an event is 63 24
deaths is 31 1/51 = 0.61. The second event
consistently greater for one group than another. It is 79 24
occurred in week 10, when there were two deaths.
unlikely to detect a difference when survival curves 80* 25
There were now 19 and 31 patients at risk (alive) in the
cross, as can happen when comparing a medical with a 82* 28
two groups, one having died in week 6, so the
surgical intervention. When analysing survival data, the 82* 30
probability of death in week 10 was 2/50. The
survival curves should always be plotted. 86 33
expected numbers of deaths were 19 2/50 = 0.76 98 34*
Because the logrank test is purely a test of
and 31 2/50 = 1.24 respectively. 149* 35
significance it cannot provide an estimate of the size of
The same calculations are performed each time an 202 37
the difference between the groups or a confidence
event occurs. If a survival time is censored, that 219 40
interval. For these we must make some assumptions
individual is considered to be at risk of dying in the 40
about the data. Common methods use the hazard ratio,
week of the censoring but not in subsequent weeks. 40*
including the Cox proportional hazards model, which
This way of handling censored observations is the 46
we shall describe in a future Statistics Note.
same as for the Kaplan-Meier survival curve.3 48
From the calculations for each time of death, the Competing interests: None declared. 70*

total numbers of expected deaths were 22.48 in group 76

81
1 and 19.52 in group 2, and the observed numbers of 1 Rostomily RC, Spence AM, Duong D, McCormick K, Bland M, Berger
MS. Multimodality management of recurrent adult malignant gliomas: 82
deaths were 14 and 28. We can now use a
2 test of the results of a phase II multiagent chemotherapy study and analysis of
91
null hypothesis. The test statistic is the sum of (O cytoreductive surgery. Neurosurgery 1994;35:378-8.
2 Altman DG, Bland JM. Time to event (survival) data. BMJ 112
E)2/E for each group, where O and E are the totals of 1998;317:468-9. 181
the observed and expected events. Here (14 22.48)2 3 Bland JM, Altman DG. Survival probabilities. The Kaplan-Meier method.
BMJ 1998;317:1572. *Censored survival
/ 22.48 + (28 19.52)2 / 19.52 = 6.88. The degrees of 4 Altman DG. Practical statistics for medical research. London: Chapman & time (still alive at
freedom are the number of groups minus one, i.e. Hall, 1991: 371-5. follow up).

BMJ VOLUME 328 1 MAY 2004 bmj.com 1073

Education and debate

Statistics Notes
Diagnostic tests 4: likelihood ratios
Jonathan J Deeks, Douglas G Altman

Screening and Test The properties of a diagnostic or screening test are often
Evaluation
Program, School of described using sensitivity and specificity or predictive Calculation of post-test probabilities using
Public Health, values, as described in previous Notes.1 2 Likelihood likelihood ratios
University of
Sydney, NSW 2006,
ratios are alternative statistics for summarising diag-
Pretest probability = p1 = 0.1
Australia nostic accuracy, which have several particularly powerful pretest odds = p1/(1 p1) = 0.1/0.9 = 0.11
Jonathan J Deeks properties that make them more useful clinically than post-test odds = pretest oddslikelihood ratio
senior research
biostatistician
other statistics.3 post-test odds = o2 = 0.1120.43 = 2.27
Each test result has its own likelihood ratio, which Post-test probability = o2/(1+ o2) = 2.27/3.37 = 0.69
Cancer Research
UK/NHS Centre summarises how many times more (or less) likely
for Statistics in patients with the disease are to have that particular
Medicine, Institute
for Health Sciences, result than patients without the disease. More formally, Likelihood ratios are ratios of probabilities, and can
Oxford OX3 7LF it is the ratio of the probability of the specific test result be treated in the same way as risk ratios for the
Douglas G Altman in people who do have the disease to the probability in
professor of statistics
purposes of calculating confidence intervals.6
in medicine people who do not. For a test with only two outcomes, likelihood ratios
Correspondence to:
A likelihood ratio greater than 1 indicates that the can be calculated directly from sensitivities and specifi-
Mr Deeks test result is associated with the presence of the disease, cities.1 For example, if smoking habit is dichotomised
Jon.Deeks@ whereas a likelihood ratio less than 1 indicates that the as above or below 40 pack years, the sensitivity is 28.4%
cancer.org.uk
test result is associated with the absence of disease. The (42/148) and specificity 98.6% (142/144). The positive
BMJ 2004;329:1689 further likelihood ratios are from 1 the stronger the likelihood ratio is the proportion with obstructive
evidence for the presence or absence of disease. Likeli- airway disease who smoked more than 40 pack years
hood ratios above 10 and below 0.1 are considered to (sensitivity) divided by the proportion without disease
provide strong evidence to rule in or rule out who smoked more than 40 pack years (1specificity),
diagnoses respectively in most circumstances.4 When 28.4/1.4 = 20.3, as before. The negative likelihood
tests report results as being either positive or negative ratio is the proportion with disease who smoked less
the two likelihood ratios are called the positive than 40 pack years (1sensitivity) divided by the
proportion without disease who smoked less than 40
likelihood ratio and the negative likelihood ratio.
pack years (specificity), 71.6/98.6 = 0.73. However,
The table shows the results of a study of the value of
unlike sensitivity and specificity, computation of
a history of smoking in diagnosing obstructive airway
likelihood ratios does not require dichotomisation of
disease.5 Smoking history was categorised into four
groups according to pack years smoked (packs per day
years smoked). The likelihood ratio for each category Pre-test Post-test
probability probability
is calculated by dividing the percentage of patients with
0.001 0.999
obstructive airway disease in that category by the
0.002 0.998
percentage without the disease in that category. For 0.003 0.997
example, among patients with the disease 28% had 40+ 0.005 0.995
0.007 0.993
smoking pack years compared with just 1.4% of 0.01 0.99
patients without the disease. The likelihood ratio is Likelihood
0.02 ratio 0.98
thus 28.4/1.4 = 20.3. A smoking history of more than 0.03 1000 0.97
500
40 pack years is strongly predictive of a diagnosis of 0.05 0.95
0.07 200 0.93
obstructive airway disease as the likelihood ratio is sub- 0.1 100 0.9
50
stantially higher than 10. Although never smoking or 20 0.8
0.2
smoking less than 20 pack years both point to not hav- 10
0.3 5 0.7
ing obstructive airway disease, their likelihood ratios 0.4 2 0.6
0.5 1 0.5
are not small enough to rule out the disease with 0.6 0.5 0.4
confidence. 0.7 0.2 0.3
0.1
0.8 0.05 0.2
0.02
0.9 0.01 0.1
Likelihood ratios are ratios of probabilities, and can be treated in the same way as risk 0.93 0.005 0.07
0.95 0.05
ratios for the purposes of calculating confidence intervals6 0.002
0.97 0.001 0.03
Obstructive airway disease 0.98 0.02
Smoking habit
(pack years) Yes (n (%)) No (n (%)) Likelihood ratio 95% CI 0.99 0.01
40 42 (28.4) 2 (1.4) (42/148)/(2/144)=20.4 5.04 to 82.8 0.993 0.007
0.995 0.005
20-40 25 (16.9) 24 (16.7) (25/148)/(24/144)=1.01 0.61 to 1.69 0.003
0.997
0-20 29 (19.6) 51 (35.4) (29/148)/51/144)=0.55 0.37 to 0.82 0.998 0.002
Never smoked or smoked 52 (35.1) 67 (46.5) (52/148)/67/144)=0.76 0.57 to 1.00
0.999 0.001
for <1 yr
Total 148 (100) 144 (100)
Use of Fagans nomogram for calculating post-test probabilities7

168 BMJ VOLUME 329 17 JULY 2004 bmj.com


test results. Forcing dichotomisation on multicategory
test results may discard useful diagnostic information.
Likelihood ratios can be used to help adapt the
results of a study to your patients. To do this they make
use of a mathematical relationship known as Bayes
theorem that describes how a diagnostic finding
changes our knowledge of the probability of abnor-
mality.3 The post-test odds that the patient has the
disease are estimated by multiplying the pretest odds
by the likelihood ratio. The use of odds rather than
risks makes the calculation slightly complex (box) but a
nomogram can be used to avoid having to make
conversions between odds and probabilities (figure).7
Both the figure and the box illustrate how a prior
probability of obstructive airway disease of 0.1 (based,
say, on presenting features) is updated to a probability
of 0.7 with the knowledge that the patient had smoked
for more than 40 pack years.
In clinical practice it is essential to know how a
particular test result predicts the risk of abnormality.
Sensitivities and specificities1 do not do this: they
describe how abnormality (or normality) predicts
particular test results. Predictive values2 do give
probabilities of abnormality for particular test results,
but depend on the prevalence of abnormality in the
A memorable patient
Living history
I was finally settling down at my desk when the pager
bleeped: it was the outpatients department. An extra
patient had been added to the afternoon listwould I
see him?
The patient was a slightly built man in his 60s. He
had brought recent documentation from another
hospital. I asked about his presenting complaint.
Well, Ill try, but I wasnt aware of everything that
happened. Thats why Ive brought my wifeshe was
with me at the time.
This was turning out to be one of those perfect
neurological consultations: documents from another
hospital, a witness account, an articulate patient. The
only question would be whether it was seizure,
syncope, or transient ischaemic attack. As we went
through his medical history, I studied his records and
for the first time noticed the phrase Tetralogy of
Fallot.
Yes, my lifelong diagnosis, he smiled. I was
operated on.
I saw the faint dusky blue colour of his lips.
Blalock-Taussig shunt? I asked, as dim memories
from medical school somehow came back into focus.
Yes, twice. By Blalock. He paused. In those days
the operation was only done at Johns Hopkinsall the
patients went there. I remember my second operation
well. I was 12 at the time.
And the doctors? I asked.
Yes, especially Dr Taussig. She would come around
with her entourage every so often. Deaf as a post, she
was.
What? Taussig deaf?
Yes. She had an amplifier attached to her
stethoscope to examine patients.
I asked to examine him, only too aware that it was
more for my benefit than his.
A half smile suggested that he had read my
thoughts: Of course.
BMJ VOLUME 329 17 JULY 2004 bmj.com
Education and debate
study sample and can rarely be generalised beyond the
study (except when the study is based on a suitable
random sample, as is sometimes the case for
population screening studies). Likelihood ratios pro-
vide a solution as they can be used to calculate the
probability of abnormality, while adapting for varying
prior probabilities of the chance of abnormality from
different contexts.
1 Altman DG, Bland JM. Diagnostic tests 1: sensitivity and specificity. BMJ
1994;308:1552.
2 Altman DG, Bland JM. Diagnostic tests 2: predictive values. BMJ
1994;309:102.
3 Sackett DL, Straus S, Richardson WS, Rosenberg W, Haynes RB. Evidence-
based medicine. How to practise and teach EBM. 2nd ed. Edinburgh: Churchill
Livingstone, 2000:67-93.
4 Jaeschke R, Guyatt G, Lijmer J. Diagnostic tests. In: Guyatt G, Rennie D,
eds. Users guides to the medical literature. Chicago: AMA Press,
2002:121-40.
5 Straus SE, McAlister FA, Sackett DL, Deeks JJ. The accuracy of patient
history, wheezing, and laryngeal measurements in diagnosing obstructive
airway disease. CARE-COAD1 Group. Clinical assessment of the
reliability of the examination-chronic obstructive airways disease. JAMA
2000;283:1853-7.
6 Altman DG. Diagnostic tests. In: Altman DG, Machin D, Bryant TN,
Gardner MJ, eds. Statistics with confidence. 2nd ed. London: BMJ Books,
2000:105-19.
7 Fagan TJ. Letter: Nomogram for Bayes theorem. N Engl J Med
1975;293:257.
To even my unpractised technique, his
cardiovascular signs were a museum piece: absent left
subclavian pulse, big arciform scars on the anterior
chest created by the surgeon who had saved his life,
central cyanosis, right-sided systolic murmurs, loud
pulmonary valve closure sound (iatrogenic pulmonary
hypertension, I reasoned), pulsatile liverall these and
undoubtedly more noted by the physician who first
understood his condition.
That night, I read about his doctors. Helen Taussig
indeed had had substantial hearing impairment, a
disability that would have meant the end of a career in
cardiology for a less able clinician. I also learnt of the
greater challenges of sexual prejudice that she fought
and overcame all her life. I learnt about Alfred Blalock,
the young doctor denied a residency at Johns Hopkins
only to be invited back in his later years to head its
surgical unit.
The experiences of a life in medicine are sometimes
overwhelming. For weeks, I reflected on the
perspectives opened to me by this unassuming patient.
The curious irony of a man with a life threatening
condition who had outlived his saviours; the
extraordinary vision of his all-too-human doctors; the
opportunity to witness history played out in the course
of a half-hour consultation. And memories were
jogged, too: the words of my former professor of
medicine, who showed us cases of Fallots and first
told us about Taussig, the woman cardiologist; the
portrait of Blalock adorning a surgical lecture theatre
in medical college.
(And my patients neurological examination and
investigations? Non-contributory. I still dont know
whether it was seizure, syncope, or transient ischaemic
attack.)
Giridhar P Kalamangalam clinical fellow in epilepsy,
department of neurology, Cleveland Clinic Foundation,
Cleveland OH, USA
169
 Education and debate

Statistics Notes
Treatment allocation by minimisation
Douglas G Altman, J Martin Bland

In almost all controlled trials treatments are allocated Cancer Research

Table 2 Hypothetical distribution of baseline characteristics after UK Medical
by randomisation. Blocking and stratification can be Statistics Group,
40 patients had been enrolled in the trial
used to ensure balance between groups in size and Centre for Statistics
patient characteristics.1 But stratified randomisation Behavioural Nutrition in Medicine, Oxford
counselling counselling OX3 7LF
using several variables is not effective in small trials. (n=20) (n=20) Douglas G Altman
The only widely acceptable alternative approach is Women 12 11 professor of statistics
minimisation,2 3 a method of ensuring excellent Age >50 7 5 in medicine
balance between groups for several prognostic factors, Ethnicity: Department of
even in small samples. With minimisation the White 15 15 Health Sciences,
University of York,
treatment allocated to the next participant enrolled in Black 4 5
York YO10 5DD
the trial depends (wholly or partly) on the characteris- Asian 1 0
J Martin Bland
tics of those participants already enrolled. The aim is Current smokers 6 8 professor of health
statistics
that each allocation should minimise the imbalance
across multiple factors. ling would increase the imbalance; allocation to nutri- Correspondence to:
tion would decrease it. D Altman
Table 1 shows some baseline characteristics in a doug.altman@
controlled trial comparing two types of counselling in At this point there are two options. The chosen cancer.org.uk
relation to dietary intake.4 Minimisation was used for treatment could simply be taken as the one with the
the four variables shown, and the two groups were lower score; or we could introduce a random element. BMJ 2005;330:843

clearly very similar in all of these variables. Such good
balance for important prognostic variables helps the
credibility of the comparisons. How is it achieved?
Minimisation is based on a different principle from
randomisation. The first participant is allocated a treat-
ment at random. For each subsequent participant we
determine which treatment would lead to better
balance between the groups in the variables of interest.
The dietary behaviour trial used minimisation
based on the four variables in table 1. Suppose that
after 40 patients had entered this trial the numbers in
each subgroup in each treatment group were as shown
in table 2. (Note that two or more categories need to be
constructed for continuous variables.)
The next enrolled participant is a black woman
aged 52, who is a non-smoker. If we were to allocate her
to behavioural counselling, the imbalance would be
increased in sex distribution (12+1 v 11 women), in age
(7+1 v 5 aged > 50), and in smoking (14+1 v 12
non-smoking) and decreased in ethnicity (4+1 v 5
black). We formalise this by summing over the four
variables the numbers of participants with the same
characteristics as this new recruit already in the trial:
Behavioural: 12 (sex) +7 (age) +4 (ethnicity) +14 (smoking) = 37
Nutrition: 11+5+5+12 = 33
Imbalance is minimised by allocating this person to
the group with the smaller total (or at random if the
totals are the same). Allocation to behavioural counsel-
Table 1 Baseline characteristics in two groups4
Behavioural Nutrition
counselling counselling
Women 82 84
Mean (SD) age (years) 43.3 (13.8) 43.2 (14.0)
Ethnicity:
White 94 96
Black 37 32
Asian 3 5
Current smokers 47 44
We use weighted randomisation so that there is a high
chance (eg 80%) of each participant getting the
treatment that minimises the imbalance. The use of a
random element will slightly worsen the overall imbal-
ance between the groups, but balance will be much
better for the chosen variables than with simple
randomisation. A random element also makes the allo-
cation more unpredictable, although minimisation is a
secure allocation system when used by an independent
person.
After the treatment is determined for the current
participant the numbers in each group are updated
and the process repeated for each subsequent
participant. If at any time the totals for the two groups
are the same, then the choice should be made using
simple randomisation. The method extends to trials of
more than two treatments.
Minimisation is a valid alternative to ordinary ran-
domisation,2 3 5 and has the advantage, especially in
small trials, that there will be only minor differences
between groups in those variables used in the
allocation process. Such balance is especially desirable
where there are strong prognostic factors and modest
treatment effects, such as oncology. Minimisation is
best performed with the aid of softwarefor example,
minim, a free program.6 Its use makes trialists think
about prognostic factors at the outset and helps ensure
adherence to the protocol as a trial progresses.7
1 Altman DG, Bland JM. How to randomise. BMJ 1999;319:703-4.
2 Treasure T, MacRae KD. Minimisation: the platinum standard for trials.
BMJ 1998;317:362-3.
3 Scott NW, McPherson GC, Ramsay CR, Campbell MK. The method of
minimization for allocation to clinical trials: a review. Control Clin Trials
2002;23:662-74.
4 Steptoe A, Perkins-Porras L, McKay C, Rink E, Hilton S, Cappuccio FP.
Behavioural counselling to increase consumption of fruit and vegetables
in low income adults: randomised trial. BMJ 2003;326:855-8.
5 Buyse M, McEntegart D. Achieving balance in clinical trials: an
unbalanced view from the European regulators. Applied Clin Trials
2004;13:36-40.
6 Evans S, Royston P, Day S. Minim: allocation by minimisation in clinical
trials. http://www-users.york.ac.uk/zmb55/guide/minim.htm. (accessed
24 October 2004).
7 Day S. Commentary: Treatment allocation by the method of
minimisation. BMJ 1999;319:947-8.

BMJ VOLUME 330 9 APRIL 2005 bmj.com 843


6 The Cardiac Arrhythmia Suppression Trial I. Preliminary Report. Effect of
encainide and flecainide on mortality in a randomized trial of arrhythmia
suppression after myocardial infarction. N Engl J Med 1989;321:406-12.
7 Dickert N, Grady C. Whats the price of a research subject? Approaches to
payment for research participation. N Engl J Med 1999;341:198-203.
8 Grady C. Money for research participation: does it jeopardize informed
consent? Am J Bioethics 2001;1:40-4.
9 Macklin R. Due and undue inducements: On paying money to
research subjects. IRB: a review of human subjects research 1981;3:1-6.
10 McGee G. Subject to payment? JAMA 1997;278:199-200.
11 McNeil P. Paying people to participate in research. Bioethics
1997;11:390-6.
12 Wilkenson M, Moore A. Inducement in research. Bioethics 1997;11:
373-89.
13 Halpern SD, Karlawish JHT, Casarett D, Berlin JA, Asch DA. Empirical
assessment of whether moderate payments are undue or unjust induce-
ments for participation in clinical trials. Arch Intern Med 2004;164:801-3.
14 Bentley JP, Thacker PG. The influence of risk and monetary payment on
the research participation decision making process. J Med Ethics
2004;30:293-8.
15 Viens AM. Socio-economic status and inducement to participate. Am J
Bioethics 2001;1.
Statistics Notes
Standard deviations and standard errors
Douglas G Altman, J Martin Bland
The terms standard error and standard deviation
are often confused.1 The contrast between these two
terms reflects the important distinction between data
description and inference, one that all researchers
should appreciate.
The standard deviation (often SD) is a measure of
variability. When we calculate the standard deviation of a
sample, we are using it as an estimate of the variability of
the population from which the sample was drawn. For
data with a normal distribution,2 about 95% of individu-
als will have values within 2 standard deviations of the
mean, the other 5% being equally scattered above and
below these limits. Contrary to popular misconception,
the standard deviation is a valid measure of variability
regardless of the distribution. About 95% of observa-
tions of any distribution usually fall within the 2 standard
deviation limits, though those outside may all be at one
end. We may choose a different summary statistic, how-
ever, when data have a skewed distribution.3
When we calculate the sample mean we are usually
interested not in the mean of this particular sample, but
in the mean for individuals of this typein statistical
terms, of the population from which the sample comes.
We usually collect data in order to generalise from them
and so use the sample mean as an estimate of the mean
for the whole population. Now the sample mean will
vary from sample to sample; the way this variation
occurs is described by the sampling distribution of the
mean. We can estimate how much sample means will
vary from the standard deviation of this sampling distri-
bution, which we call the standard error (SE) of the esti-
mate of the mean. As the standard error is a type of
standard deviation, confusion is understandable.
Another way of considering the standard error is as a
measure of the precision of the sample mean.
The standard error of the sample mean depends
on both the standard deviation and the sample size, by
the simple relation SE = SD/(sample size). The stand-
ard error falls as the sample size increases, as the extent
of chance variation is reducedthis idea underlies the
sample size calculation for a controlled trial, for
BMJ VOLUME 331 15 OCTOBER 2005 bmj.com
Education and debate
16 Beauchamp TL, Childress JF. Respect for autonomy. Principles of biomedical
ethics. 4th ed. New York: Oxford University Press, 1994:120-88.
17 Roberts LW. Evidence-based ethics and informed consent in mental
illness research. Arch Gen Psychiatry 2000;57:540-2.
18 Bayer R, Oppenheimer GM. Toward a more democratic medicine: sharing the
burden of ignorance. AIDS Doctors: voices from the epidemic. New York: Oxford
University Press, 2000:156-69.
19 Coulter A, Rozansky D. Full engagement in health. BMJ 2004;329:1197-8.
20 Joffe S, Manocchia M, Weeks JC, Cleary PD. What do patients value in
their hospital care? An empirical perspective on autonomy centred
bioethics. J Med Ethics 2003;29:103-8.
21 Heesen C, Kasper J, Segal J, Kopke S, Muhlhauser I. Decisional role pref-
erences, risk knowledge and information interests in patients with multi-
ple sclerosis. Mult Scler 2004;10:643-50.
22 Azoulay E, Pochard F, Chevret S, et al. Half the family members of inten-
sive care unit patients do not want to share in the decision-making proc-
ess: a study in 78 French intensive care units. Crit Care Med
2004;32:1832-8.
23 Dunn LB, Gordon NE. Improving informed consent and enhancing
recruitment for research by understanding economic behavior. JAMA
2005;293:609-12.
example. By contrast the standard deviation will not Cancer Research
UK/NHS Centre
tend to change as we increase the size of our sample. for Statistics in
So, if we want to say how widely scattered some Medicine, Wolfson
measurements are, we use the standard deviation. If we College, Oxford
OX2 6UD
want to indicate the uncertainty around the estimate of
Douglas G Altman
the mean measurement, we quote the standard error of professor of statistics
the mean. The standard error is most useful as a means in medicine
of calculating a confidence interval. For a large sample, Department of
a 95% confidence interval is obtained as the values Health Sciences,
University of York,
1.96SE either side of the mean. We will discuss confi- York YO10 5DD
dence intervals in more detail in a subsequent Statistics J Martin Bland
Note. The standard error is also used to calculate P val- professor of health
statistics
ues in many circumstances.
The principle of a sampling distribution applies to Correspondence to:
Prof Altman
other quantities that we may estimate from a sample, doug.altman@
such as a proportion or regression coefficient, and to cancer.org.uk
contrasts between two samples, such as a risk ratio or
BMJ 2005;331:903
the difference between two means or proportions. All
such quantities have uncertainty due to sampling vari-
ation, and for all such estimates a standard error can be
calculated to indicate the degree of uncertainty.
In many publications a sign is used to join the
standard deviation (SD) or standard error (SE) to an
observed meanfor example, 69.49.3 kg. That
notation gives no indication whether the second figure
is the standard deviation or the standard error (or
indeed something else). A review of 88 articles
published in 2002 found that 12 (14%) failed to
identify which measure of dispersion was reported
(and three failed to report any measure of variability).4
The policy of the BMJ and many other journals is to
remove signs and request authors to indicate clearly
whether the standard deviation or standard error is
being quoted. All journals should follow this practice.
Competing interests: None declared.
1 Nagele P. Misuse of standard error of the mean (SEM) when reporting
variability of a sample. A critical evaluation of four anaesthesia journals.
Br J Anaesthesiol 2003;90:514-6.
2 Altman DG, Bland JM. The normal distribution. BMJ 1995;310:298.
3 Altman DG, Bland JM. Quartiles, quintiles, centiles, and other quantiles.
BMJ 1994;309:996.
4 Olsen CH. Review of the use of statistics in Infection and Immunity. Infect
Immun 2003;71:6689-92.
903
Practice
Statistics Notes
The cost of dichotomising continuous variables
Douglas G Altman, Patrick Royston
Cancer Research Measurements of continuous variables are made in all
UK/NHS Centre
for Statistics in
branches of medicine, aiding in the diagnosis and
Medicine, Wolfson treatment of patients. In clinical practice it is helpful to
College, Oxford label individuals as having or not having an attribute,
OX2 6UD
such as being hypertensive or obese or having riously significant result; the difference in the outcome
Douglas G Altman
professor of statistics high cholesterol, depending on the value of a
in medicine continuous variable.
MRC Clinical Trials Categorisation of continuous variables is also com-
Unit, London mon in clinical research, but here such simplicity is
NW1 2DA
gained at some cost. Though grouping may help data
Patrick Royston
professor of statistics presentation, notably in tables, categorisation is unnec-
Correspondence to: essary for statistical analysis and it has some serious
Professor Altman drawbacks. Here we consider the impact of converting
doug.altman@
continuous data to two groups (dichotomising), as this
cancer.org.uk
is the most common approach in clinical research.1
BMJ 2006;332:1080 What are the perceived advantages of forcing all
individuals into two groups? A common argument is
that it greatly simplifies the statistical analysis and leads
to easy interpretation and presentation of results. A
binary splitfor example, at the medianleads to a
comparison of groups of individuals with high or low
values of the measurement, leading in the simplest case
to a t test or
2 test and an estimate of the difference
between the groups (with its confidence interval). There
is, however, no good reason in general to suppose that
there is an underlying dichotomy, and if one exists there
is no reason why it should be at the median.2
Dichotomising leads to several problems. Firstly,
much information is lost, so the statistical power to
detect a relation between the variable and patient out-
come is reduced. Indeed, dichotomising a variable at
the median reduces power by the same amount as
would discarding a third of the data.2 3 Deliberately dis-
carding data is surely inadvisable when research
studies already tend to be too small. Dichotomisation
may also increase the risk of a positive result being a
false positive.4 Secondly, one may seriously underesti-
mate the extent of variation in outcome between
groups, such as the risk of some event, and
considerable variability may be subsumed within each
group. Individuals close to but on opposite sides of the
cutpoint are characterised as being very different
rather than very similar. Thirdly, using two groups
conceals any non-linearity in the relation between the
variable and outcome. Presumably, many who dichot-
omise are unaware of the implications.
If dichotomisation is used where should the
cutpoint be? For a few variables there are recognised
cutpoints, such as > 25 kg/m2 to define overweight
based on body mass index. For some variables, such as
age, it is usual to take a round number, usually a mul-
tiple of five or 10. The cutpoint used in previous stud-
ies may be adopted. In the absence of a prior cutpoint
the most common approach is to take the sample
median. However, using the sample median implies
that various cutpoints will be used in different studies
so that their results cannot easily be compared,
seriously hampering meta-analysis of observational
studies.5 Nevertheless, all these approaches are
1080
preferable to performing several analyses and
choosing that which gives the most convincing result.
Use of this so called optimal cutpoint (usually that
giving the minimum P value) runs a high risk of a spu-
variable between the groups will be overestimated,
perhaps considerably; and the confidence interval will
be too narrow. This strategy should never be used.6 7
When regression is being used to adjust for the effect
of a confounding variable, dichotomisation will run the
risk that a substantial part of the confounding remains.4 7
Dichotomisation is not much used in epidemiological
studies, where the use of several categories is preferred.
Using multiple categories (to create an ordinal
variable) is generally preferable to dichotomising. With
four or five groups the loss of information can be quite
small, but there are complexities in analysis.
Instead of categorising continuous variables, we pre-
fer to keep them continuous. We could then use
linear regression rather than a two sample t test, for
example. If we were concerned that a linear regression
would not truly represent the relation between the
outcome and predictor variable, we could explore
whether some transformation (such as a log transforma-
tion) would be helpful.7 8 As an example, in a regression
analysis to develop a prognostic model for patients with
primary biliary cirrhosis, a carefully developed model
with bilirubin as a continuous explanatory variable
explained 31% more of the variability in the data than
when bilirubin distribution was split at the median.7
Competing interests: None declared.
1 Del Priore G, Zandieh P, Lee MJ. Treatment of continuous data as
categoric variables in obstetrics and gynecology. Obstet Gynecol
1997;89:351-4.
2 MacCallum RC, Zhang S, Preacher KJ, Rucker DD. On the practice of
dichotomization of quantitative variables. Psychol Meth 2002;7:19-40.
3 Cohen J. The cost of dichotomization. Appl Psychol Meas 1983;7:249-53.
4 Austin PC, Brunner LJ. Inflation of the type I error rate when a continu-
ous confounding variable is categorized in logistic regression analyses.
Stat Med 2004;23:1159-78.
5 Buettner P, Garbe C, Guggenmoos-Holzmann I. Problems in defining
cutoff points of continuous prognostic factors: example of tumor
thickness in primary cutaneous melanoma. J Clin Epidemiol
1997;50:1201-10.
6 Altman DG, Lausen B, Sauerbrei W, Schumacher M. Dangers of using
optimal cutpoints in the evaluation of prognostic factors. J Natl Cancer
Inst 1994;86:829-35.
7 Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous
predictors in multiple regression: a bad idea. Stat Med 2006;25:127-41.
8 Royston P, Sauerbrei W. Building multivariable regression models with
continuous covariates in clinical epidemiologywith an emphasis on
fractional polynomials. Methods Inf Med 2005;44:561-71.
Endpiece
Reading and reflecting
Reading without reflecting is like eating without
digesting.
Edmund Burke
Kamal Samanta, retired general practitioner,
Denby Dale, West Yorkshire
BMJ VOLUME 332 6 MAY 2006 bmj.com
 PRACTICE
1
Cancer Research UK/NHS Centre
for Statistics in Medicine, Oxford
OX2 6UD
2
Department of Health Sciences,
University of York, York YO10 5DD
Correspondence to: Professor
Altman
doug.altman@cancer.org.uk
bmj 2007;334:424
doi:10.1136/bmj.38977.682025.2C
424 BMJ | 24 february 2007 | Volume 334
Statistics Notes
Missing data
Douglas G Altman1, J Martin Bland2
Almost all studies have some missing observations.
Yet textbooks and software commonly assume that
data are complete, and the topic of how to handle
missing data is not often discussed outside statistics
journals.
There are many types of missing data and different
reasons for data being missing. Both issues affect the
analysis. Some examples are:
(1) In a postal questionnaire survey not all the
selected individuals respond;
(2) In a randomised trial some patients are lost to
follow-up before the end of the study;
(3) In a multicentre study some centres do not
measure a particular variable;
(4) In a study in which patients are assessed
frequently some data are missing at some time
points for unknown reasons;
(5) Occasional data values for a variable are missing
because some equipment failed;
(6) Some laboratory samples are lost in transit or
technically unsatisfactory;
(7) In a magnetic resonance imaging study some very
obese patients are excluded as they are too large
for the machine;
(8) In a study assessing quality of life some patients
die during the follow-up period.
The prime concern is always whether the available
data would be biased. If the fact that an observation
is missing is unrelated both to the unobserved value
(and hence to patient outcome) and the data that are
available this is called missing completely at ran-
dom. For cases 5 and 6 above that would be a safe
assumption. Sometimes data are missing in a predict-
able way that does not depend on the missing value
itself but which can be predicted from other dataas
in case 3. Confusingly, this is known as missing at
random. In the common cases 1 and 2, however, the
missing data probably depend on unobserved values,
called missing not at random, and hence their lack
may lead to bias.
In general, it is important to be able to examine
whether missing data may have introduced bias. For
example, if we know nothing at all about the non-
responders to a survey then we can do little to explore
possible bias. Thus a high response rate is necessary for
reliable answers.1 Sometimes, though, some informa-
tion is available. For example, if the survey sample is
chosen from a register that includes age and sex, then
the responders and non-responders can be compared
on these variables. At the very least this gives some
pointers to the representativeness of the sample. Non-
responders often (but not always) have a worse medical
prognosis than those who respond.
A few missing observations are a minor nuisance,
but a large amount of missing data is a major threat
to a studys integrity. Non-response is a particular
problem in pair-matched studies, such as some case-
control studies, as it is unclear how to analyse data
from the unmatched individuals. Loss of patients
also reduces the power of the trial. Where losses are
expected it is wise to increase the target sample size
to allow for losses. This cannot eliminate the poten-
tial bias, however.
Missing data are much more common in retro-
spective studies, in which routinely collected data are
subsequently used for a different purpose.2 When infor-
mation is sought from patients medical notes, the notes
often do not say whether or not a patient was a smoker
or had a particular procedure carried out. It is tempt-
ing to assume that the answer is no when there is no
indication that the answer is yes, but this is generally
unwise.
No really satisfactory solution exists for missing data,
which is why it is important to try to maximise data
collection. The main ways of handling missing data in
analysis are: (a) omitting variables which have many
missing values; (b) omitting individuals who do not
have complete data; and (c) estimating (imputing) what
the missing values were.
Omitting everyone without complete data is known
as complete case (or available case) analysis and is
probably the most common approach. When only a
very few observations are missing little harm will be
done, but when many are missing omitting all patients
without full data might result in a large proportion of
the data being discarded, with a major loss of statisti-
cal power. The results may be biased unless the data
are missing completely at random. In general it is
advisable not to include in an analysis any variable
that is not available for a large proportion of the sam-
ple. The main alternative approach to case deletion is
imputation, whereby missing values are replaced by
some plausible value predicted from that individuals
available data. Imputation has been the topic of much
recent methodological work; we will consider some of
the simpler methods in a separate Statistics Note.
Competing interests: None declared.
1 Evans SJW. Good surveys guide. BMJ 1991;302:302-3.
2 Burton A, Altman DG. Missing covariate data within cancer prognostic
studies: a review of current reporting and proposed guidelines. Br J
Cancer 2004;91:4-8.
This is part of a series of occasional articles on statistics and handling
data in research.
This site uses cookies. More info Close By continuing to browse the site you are agreeing to our use of
cookies. Find out more here Close

Practice Statistics Notes

Parametric v non-parametric methods for data

analysis
BMJ 2009; 338 doi: https://doi.org/10.1136/bmj.a3167 (Published 02 April 2009) Cite this as: BMJ
2009;338:a3167

Douglas G Altman, professor of statistics in medicine1, J Martin Bland, professor of health statistics2
1
Centre for Statistics in Medicine, University of Oxford, Wolfson College Annexe, Oxford OX2 6UD
2
Department of Health Sciences, University of York, York YO10 5DD

Correspondence to: Professor Altman doug.altman@csm.ox.ac.uk

Continuous data arise in most areas of medicine. Familiar clinical examples include blood pressure, ejection
fraction, forced expiratory volume in 1 second (FEV ), serum cholesterol, and anthropometric measurements.
1
Methods for analysing continuous data fall into two classes, distinguished by whether or not they make
assumptions about the distribution of the data.

Theoretical distributions are described by quantities called parameters, notably the mean and standard
deviation.1 Methods that use distributional assumptions are called parametric methods, because we estimate the
parameters of the distribution assumed for the data. Frequently used parametric methods include t tests and
analysis of variance for comparing groups, and least squares regression and correlation for studying the relation
between variables. All of the common parametric methods (t methods) assume that in some way the data follow
a normal distribution and also that the spread of the data (variance) is uniform either between groups or across
the range being studied. For example, the two sample t test assumes that the two samples of observations come
from populations that have normal distributions with the same standard deviation. The importance of the
assumptions for t methods diminishes as sample size increases.

Alternative methods, such as the sign test, Mann-Whitney test, and rank correlation, do not require the data to
follow a particular distribution. They work by using the rank order of observations rather than the measurements
themselves. Methods which do not require us to make distributional assumptions about the data, such as the
rank methods, are called non-parametric methods. The term non-parametric applies to the statistical method
used to analyse data, and is not a property of the data.1 As tests of significance, rank methods have almost as
much power as t methods to detect a real difference when samples are large, even for data which meet the
distributional requirements.

Non-parametric methods are most often used to analyse data which do not meet the distributional requirements
of parametric methods. In particular, skewed data are frequently analysed by non-parametric methods, although
data transformation can often make the data suitable for parametric analyses.2

Data that are scores rather than measurements may have many possible values, such as quality of life scales or
data from visual analogue scales, while others have only a few possible values, such as Apgar scores or stage of
disease. Scores with many values are often analysed using parametric methods, whereas those with few values
tend to be analysed using rank methods, but there is no clear boundary between these cases.

To compensate for the advantage of being free of assumptions about the distribution of the data, rank methods
have the disadvantage that they are mainly suited to hypothesis testing and no useful estimate is obtained, such
as the average difference between two groups. Estimates and confidence intervals are easy to find with t
methods. Non-parametric estimates and confidence intervals can be calculated, however, but depend on extra
assumptions which are almost as strong as those for t methods.3 Rank methods have the added disadvantage of
not generalising to more complex situations, most obviously when we wish to use regression methods to adjust
for several other factors.

Rank methods can generate strong views, with some people preferring them for all analyses and others believing
that they have no place in statistics. We believe that rank methods are sometimes useful, but parametric
methods are generally preferable as they provide estimates and confidence intervals and generalise to more
complex analyses.

The choice of approach may also be related to sample size, as the distributional assumptions are more important
for small samples. We consider the analysis of small data sets in a subsequent Statistics Note.

Notes
Cite this as: BMJ 2009;338:a3167

Footnotes
Competing interests: None declared.

Provenance and peer review: Commissioned, not externally peer reviewed.

References
1. Altman DG, Bland JM. Variables and parameters. BMJ1999;318:1667.
2. Bland JM, Altman DG. Transforming data. BMJ1996;312:770.

3. Campbell MJ, Gardner MJ. Medians and their differences. In: Altman DG, Machin D, Bryant TN, Gardner MJ, eds.
Statistics with confidence. 2nd ed. London: BMJ Books, 2000:36-44.
This site uses cookies. More info Close By continuing to browse the site you are agreeing to our use of
cookies. Find out more here Close

Practice Statistics Notes

Analysis of continuous data from small samples

BMJ 2009; 338 doi: https://doi.org/10.1136/bmj.a3166 (Published 06 April 2009) Cite this as: BMJ
2009;338:a3166

J Martin Bland, professor of health statistics 1, Douglas G Altman, professor of statistics in medicine2
1
Department of Health Sciences, University of York, York YO10 5DD
2
Centre for Statistics in Medicine, University of Oxford, Wolfson College Annexe, Oxford OX2 6UD

Correspondence to: Professor Bland mb55@york.ac.uk

Studies with small numbers of measurements are rare in the modern BMJ, but they used to be common and
remain plentiful in specialist clinical journals. Their analysis is often more problematic than that for large samples.

Parametric methods, including t tests, correlation, and regression, require the assumption that the data follow a
normal distribution and that variances are uniform between groups or across ranges.1 In small samples these
assumptions are particularly important, so this setting seems ideal for rank (non-parametric) methods, which
make no assumptions about the distribution of the data; they use the rank order of observations rather than the
measurements themselves.1 Unfortunately, rank methods are least effective in small samples. Indeed, for very
small samples, they cannot yield a significant result whatever the data. For example, when using the Mann-
Witney test for comparing two samples of fewer than four observations a statistically significant difference is
impossible: any data give P>0.05. Similarly, the Wilcoxon paired test, the sign test, and Spearmans and
Kendalls rank correlation coefficients cannot produce P<0.05 for fewer than six observations. Methods based on
the t distribution do not have this problem and can detect differences in samples as small as two for paired
differences and three for two groups, or detect correlations in samples of three.

For example, we were recently asked about the data in table 1, which shows before and after measurements of
pudendal nerve terminal motor latency. Should we use the Wilcoxon or the sign test? MB replied that the
Wilcoxon would be acceptable, giving P<0.05 (actually P=0.047), and so would the paired t test, which gave
P=0.04. The questioner also asked whether the Wilcoxon test could be used for the second group of four
observations alone, for patients who had received a slightly different intervention. Here all the differences are in
the same direction, but the Wilcoxon test gives P=0.125. It is not possible for it to give a significant difference.
The paired t test gives P=0.04, a significant difference.

Subgroup Pudendal nerve terminal motor latency (ms)

Initial Follow-up Change

1 2.2 2.3 0.1

 Subgroup Pudendal nerve terminal motor latency (ms)

1 2.3 1.6 0.7

1 2.1 2.2 0.1

1 2.4 2.3 0.1

2 2.3 2.1 0.2

2 2.4 1.8 0.6

2 2.4 1.9 0.5

2 2.6 1.6 1.0

Table 1 Five year follow-up of patients receiving hyperbaric oxygen therapy for faecal incontinence

On the other hand, using t methods when their assumptions are greatly violated can also be misleading. Table 2
shows concentration of antibody to type II group B Streptococcus in 20 volunteers before and after
immunisation.2 3 The comparison of the antibody levels was summarised in the report of this study as t=1.8;
P>0.05. The paired t test is not suitable for these data, because the differences clearly have a very skewed
distribution. There are 8 zero differences, forming a clump at one end of the distribution, which would remain
whatever transformation we used. We could consider the Wilcoxon paired sample test, but this method assumes
that the differences have a symmetrical distribution, which they do not. The sign test is preferred here; it tests the
null hypothesis that non-zero differences are equally likely to be positive or negative, using the binomial
distribution. We have 1 negative and 11 positive differences, which gives P=0.006. Hence the original authors
failed to detect a difference because they used an inappropriate analysis.

Antibody to type II group B Streptococcus (g/ml)

Before After Change

0.4 0.4 0.0

0.4 0.5 0.1

0.4 0.5 0.1

0.4 0.9 0.5

0.5 0.5 0.0

 Antibody to type II group B Streptococcus (g/ml)

0.5 0.5 0.0

0.5 0.5 0.0

0.5 0.5 0.0

0.5 0.5 0.0

0.6 0.6 0.0

0.6 12.2 11.6

0.7 1.1 0.4

0.7 1.2 0.5

0.8 0.8 0.0

0.9 1.2 0.3

0.9 1.9 1.0

1.0 0.9 0.1

1.0 2.0 1.0

1.6 8.1 6.5

2.0 3.7 1.7

We have often come across the idea that we should not use t distribution methods for small samples but should
instead use rank based methods. The statement is sometimes that we should not use t methods at all for
samples of fewer than six observations.4 But, as we noted, rank based methods cannot produce anything useful
for such small samples.

The aversion to parametric methods for small samples may arise from the inability to assess the distribution
shape when there are so few observations. How can we tell whether data follow a normal distribution if we have
only a few observations? The answer is that we have not only the data to be analysed, but usually also
experience of other sets of measurements of the same thing. In addition, general experience tells us that body
size measurements are usually approximately normal, as are the logarithms of many blood concentrations and
the square roots of counts.
Notes
Cite this as: BMJ 2009;338:a3166

Footnotes
We thank Jonathan Cowley for the data in table 1.

Competing interests: None declared.

Provenance and peer review: Commissioned, not externally peer reviewed.

References
1. Altman DG, Bland JM. Parametric v non-parametric methods for data analysis. BMJ 2009;338:a3167.
2. Baker CJ, Kasper DL, Edwards MS, Schiffman G. Influence of preimmunization antibody levels on the specificity of the
immune response to related polysaccharide antigens. N Engl J Med1980;303:173-8.
3. Altman DG. Practical statistics for medical research. London: Chapman and Hall, 1991: 224-5.
4. Siegel S. Nonparametric statistics for the behavioral sciences. 1st ed. Tokyo: McGraw-Hill Kogakusha, 1956:32.
This site uses cookies. More info Close By continuing to browse the site you are agreeing to our use of
cookies. Find out more here Close

Research Methods & Reporting Statistics Notes

Correlation in restricted ranges of data

BMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d556 (Published 11 March 2011) Cite this as: BMJ 2011;342:d556

J Martin Bland, professor of health statistics1, Douglas G Altman, professor of statistics in medicine2
1
Department of Health Sciences, University of York, York YO10 5DD
2
Centre for Statistics in Medicine, University of Oxford, Oxford OX2 6UD

Correspondence to: Professor M Bland martin.bland@york.ac.uk

In a study of 150 adult diabetic patients there was a strong correlation between abdominal circumference and
body mass index (BMI) (r = 0.85).1 The authors went on to report that the correlation differed in different BMI
categories as shown in the table.

BMI group r

<25 0.62

25 to 30 0.50

30 to 35 0.09

>35 0.86

All patients 0.85

Correlation between abdominal circumference and body mass indeed (BMI) in 1450 adult patients with
diabetes
2
The authors interpretation of these data was that in patients with low or high BMI values (BMI <25 kg/m and
2 2
BMI >35 kg/m ) the correlation was strong, but in those with BMI values between 25 and 35 kg/m the correlation
was weak or missing. They concluded that measuring abdominal circumference is of particular importance in
2
subjects with the most frequent BMI category (25 to 35 kg/m ).

When we restrict the range of one of the variables, a correlation coefficient will be reduced. For example, fig 1
shows some BMI and abdominal circumference measurements from a different population. Although these
people are from a rather thinner population, the correlation coefficient is very similar, r = 0.82 (P<0.0001). When
2
we divide the sample into the same four restricted ranges of BMI at 20, 25, and 30 kg/m , the correlation
coefficient in each interval is smaller than the correlation coefficient for the whole sample. This phenomenon is
to be expected; it is a result of restricting the range of data, not any particular property of BMI and abdominal
circumference.
 Figure1
BMI and abdominal circumference in 202 men and women, with correlation coefficients in four restricted
ranges and overall
2
One interpretation of the correlation coefficient r is that r is the proportion of the variation in abdominal
circumference explained or predicted by the variation in BMI. If we restrict the range of BMI values we reduce the
variation in BMI, which will explain less variation in abdominal circumference, and r will fall. If we further reduce
the variation in BMI until all remaining patients have the same BMI, then we cannot explain any variation in
abdominal circumference and the correlation must be zero. (By contrast within any of the sections of fig 1 the
fitted regression line would be the same, apart from random variation.)

For another example, fig 2 shows the weights and heights of the same sample, with different symbols for men
and women. Clearly, the lower end of the height range for men is higher than the lower end of the range for
women, but the upper ends of the ranges are very similar. The mens heights (SD 6.0 cm) are less variable than
those of the women (SD 8.9 cm) or the heights of both sexes combined (also SD 8.9 cm). The correlation
coefficients for women and for both men and women are very similar and considerably larger than that for men
alone.

Figure2
Weight and height in 202 men and women, with correlation coefficients

The same phenomenon can arise when the sample is restricted using another variable related to the ones being
studies. For example, the correlation between weight and height of schoolchildren will increase as the age range
is increased. But a spurious correlation may also be seen in such a situation, for example between shoe size and
spelling ability.2 Such an example illustrates the well worn phrase that an observed association does not imply
causation.

Correlation coefficients are a property of the variables and also the population in which they are measured. If we
look at a restricted population, we should not conclude that there is little or no relation between the variables
because the correlation coefficient is small. But given a clear relation in the whole group, we see no point in
looking within categories of one of the variables. In any case, regression is generally the preferred approach to
considering the relation between two continuous variables.

Notes
Cite this as: BMJ 2011;342:d556

Footnotes
Acknowledgements: The data are taken from a student elective project by Dr Malcolm Savage.

Contributors: JMB and DGA jointly wrote and agreed the text, JMB did the statistical analysis.

Competing interests: All authors have completed the Unified Competing Interest form at
www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no
support from any organisation for the submitted work; no financial relationships with any organisations that
might have an interest in the submitted work in the previous 3 years; no other relationships or activities that
could appear to have influenced the submitted work.

References
1. Ndas J, Putz Z, Kolev G, Nagy S, Jermendy G. Intraobserver and interobserver variability of measuring waist
circumference. Med Sci Monit2008;14:CR15-8.
2. Goodwin LD, Leech NL. Understanding correlation: factors that affect the size of r. J Exp Educ2006;74:251-66.
BMJ 2011;342:d561 doi: 10.1136/bmj.d561
Research Methods & Reporting
RESEARCH METHODS & REPORTING
STATISTICS NOTES
Comparisons within randomised groups can be very
misleading
J Martin Bland professor of health statistics 1, Douglas G Altman professor of statistics in medicine 2
1
Department of Health Sciences, University of York, York YO10 5DD; 2Centre for Statistics in Medicine, University of Oxford, Oxford OX2 6UD
When we randomise trial participants into two or more
intervention groups, we do this to remove bias; the groups will,
on average, be comparable in every respect except the treatment
which they receive. Provided the trial is well conducted, without
other sources of bias, any difference in the outcome of the
groups can then reasonably be attributed to the different
interventions received. In a previous note we discussed the
analysis of those trials in which the primary outcome measure
is also measured at baseline. We discussed several valid
analyses, observing that analysis of covariance (a regression
method) is the method of choice.1
Rather than comparing the randomised groups directly, however,
researchers sometimes look at the change in the measurement
between baseline and the end of the trial; they test whether there
was a significant change from baseline, separately in each
randomised group. They may then report that this difference is
significant in one group but not in the other, and conclude that
this is evidence that the groups, and hence the treatments, are
different. One such example was a recent trial in which
participants were randomised to receive either an anti-ageing
cream or the vehicle as a placebo.2 A wrinkle score was recorded
at baseline and after six months. The authors gave the results
of significance tests comparing the score with baseline for each
group separately, reporting the active treatment group to have
a significant difference (P=0.013) and the vehicle group not
(P=0.11). Their interpretation was that the cosmetic cream
resulted in significant clinical improvement in facial wrinkles.
But we cannot validly draw this conclusion, because the lack
of a significant difference in the vehicle group does not provide
good evidence that the anti-ageing product is superior.3
The essential feature of a randomised trial is the comparison
between groups. Within group analyses do not address a
meaningful question: the question is not whether there is a
change from baseline, but whether any change is greater in one
group than the other. It is not possible to draw valid inferences
by comparing P values. In particular, there is an inflated risk of
a false positive result, which we shall illustrate with a simulation.
Correspondence to: Professor M Bland martin.bland@york.ac.uk
Reprints: http://journals.bmj.com/cgi/reprintform
Page 1 of 3
The table shows simulated data for a randomised trial with two
groups of 30 participants. Data were drawn from the same
population, so there is no systematic difference between the two
groups. The true baseline measurements had a mean of 10.0
with standard deviation (SD) 2.0, and the outcome measurement
was equal to the baseline plus an increase of 0.5 and a random
element with SD 1.0. The difference between mean outcomes
is 0.22 (95% confidence interval 0.75 to 0.34, P=0.5), adjusting
for the baseline by analysis of covariance.1 The difference is
not statistically significant, which is not surprising because we
know that the null hypothesis of no difference in the population
is true. If we compare baseline with outcome for each group
using a paired t test, however, for group A the difference is
statistically significant, P=0.03, for group B it is not significant,
P = 0.2. These results are quite similar to those of the anti-ageing
cream trial.2
We would not wish to draw any conclusions from one
simulation. In 1000 runs, the difference between groups had
P<0.05 in the analysis of covariance 47 times, or for 4.7% of
samples, very close to the 5% we expect. Of the 2000
comparisons between baseline and outcome, 1500 (75%) had
P<0.05. In this simulation, where there is no difference
whatsoever between the two treatments, the probability of a
significant difference in one group but not the other was 38%,
not 5%. Hence a significant difference in one group but not the
other is not good evidence of a significant difference between
the groups. Even when there is a clear benefit of one treatment
over the other, separate P values are not the way to analyse such
studies.4
How many pairs of tests will have one significant and one
non-significant difference depends on the size of the change
from baseline to final measurement. If the population difference
from baseline is very large, nearly all the within group tests will
be significant, and if the population difference is small, nearly
all tests will be not significant, so there will be few samples
with only one significant difference. If the difference is such
that half the samples would show a significant change from
baseline, as it would be in our simulation if the underlying
Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe
BMJ 2011;342:d561 doi: 10.1136/bmj.d561
difference were 0.37 rather than 0.5, we would expect 50% of
samples to have just one significant difference.
The anti-ageing trial is not the only one where we have seen
this misleading approach applied to randomised trial data.3 We
even found it once in the BMJ!5
Contributors: JMB and DGA jointly wrote and agreed the text, JMB did
the statistical analysis.
Competing interests: All authors have completed the Unified Competing
Interest form at www.icmje.org/coi_disclosure.pdf (available on request
from the corresponding author) and declare: no support from any
organisation for the submitted work; no financial relationships with any
Reprints: http://journals.bmj.com/cgi/reprintform
Page 2 of 3
RESEARCH METHODS & REPORTING
organisations that might have an interest in the submitted work in the
previous 3 years; no other relationships or activities that could appear
to have influenced the submitted work.
1 Vickers AJ, Altman DG. Analysing controlled trials with baseline and follow-up
measurements. BMJ 2001;323:1123-4.
2 Watson REB, Ogden S, Cotterell LF, Bowden JJ, Bastrilles JY, Long SP, et al. A cosmetic
anti-ageing product improves photoaged skin: a double-blind, randomized controlled
trial. Br J Dermatol 2009;161:419-26.
3 Bland JM. Evidence for an anti-ageing product may not be so clear as it appears. Br J
Dermatol 2009;161:1207-8.
4 Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ
2003;326:219.
5 Bland JM, Altman DG. Informed consent. BMJ 1993;306:928.
Cite this as: BMJ 2011;342:d561
Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe
BMJ 2011;342:d561 doi: 10.1136/bmj.d561 Page 3 of 3

RESEARCH METHODS & REPORTING

Table

Table 1| Simulated data from a randomised trial comparing two groups of 30 patients, with a true change from baseline but no difference
between groups (sorted by baseline values within each group)

Group A Group B
Baseline 6 months Change Baseline 6 months Change
1 6.4 7.1 0.7 1 6.8 7.9 1.1
2 6.6 5.6 -1.0 2 7.2 7.5 0.3
3 7.3 8.3 1.0 3 7.2 6.9 -0.3
4 7.7 9.1 1.4 4 7.4 6.9 -0.5
5 7.7 9.5 1.8 5 7.5 8.3 0.8
6 7.9 9.6 1.7 6 7.5 9.4 1.9
7 8.0 8.5 0.5 7 8.3 9.0 0.7
8 8.0 8.5 0.5 8 8.4 8.8 0.4
9 8.1 9.1 1.0 9 8.7 8.0 -0.7
10 9.2 9.6 0.4 10 9.0 7.2 -1.8
11 9.3 8.7 -0.6 11 9.2 7.1 -2.1
12 9.6 10.7 1.1 12 9.6 10.6 1.0
13 9.7 9.0 -0.7 13 9.9 11.0 1.1
14 9.8 9.0 -0.8 14 10.1 11.5 1.4
15 9.8 8.0 -1.8 15 10.2 10.4 0.2
16 10.2 11.1 0.9 16 10.3 11.0 0.7
17 10.3 11.5 1.2 17 10.4 9.9 -0.5
18 10.6 9.1 -1.5 18 10.5 11.3 0.8
19 10.6 12.0 1.4 19 10.7 9.9 -0.8
20 10.7 13.2 2.5 20 10.8 10.7 -0.1
21 10.9 9.7 -1.2 21 10.8 11.8 1.0
22 11.1 12.2 1.1 22 11.1 10.0 -1.1
23 11.2 10.8 -0.4 23 11.1 13.2 2.1
24 11.8 11.9 0.1 24 11.4 11.8 0.4
25 12.3 12.2 -0.1 25 11.6 12.1 0.5
26 12.4 12.6 0.2 26 11.7 11.5 -0.2
27 13.1 15.0 1.9 27 12.0 12.7 0.7
28 13.2 13.8 0.6 28 12.3 13.7 1.4
29 13.3 14.1 0.8 29 13.7 12.6 -1.1
30 13.7 14.2 0.5 30 13.9 13.7 -0.2
Mean 10.02 10.46 0.44 Mean 9.98 10.21 0.24
SD 2.06 2.29 1.06 SD 1.90 2.09 1.02

Reprints: http://journals.bmj.com/cgi/reprintform Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe

BMJ 2011;343:d2090 doi: 10.1136/bmj.d2090 (Published 8 August 2011)
Research Methods & Reporting
RESEARCH METHODS & REPORTING
How to obtain the confidence interval from a P value
Douglas G Altman professor of statistics in medicine , J Martin Bland professor of health statistics
1
Centre for Statistics in Medicine, University of Oxford, Oxford OX2 6UD; 2Department of Health Sciences, University of York, Heslington, York
YO10 5DD
Confidence intervals (CIs) are widely used in reporting statistical
analyses of research data, and are usually considered to be more
informative than P values from significance tests.1 2 Some
published articles, however, report estimated effects and P
values, but do not give CIs (a practice BMJ now strongly
discourages). Here we show how to obtain the confidence
interval when only the observed effect and the P value were
reported.
The method is outlined in the box below in which we have
distinguished two cases.
(a) Calculating the confidence interval for
a difference
We consider first the analysis comparing two proportions or
two means, such as in a randomised trial with a binary outcome
or a measurement such as blood pressure.
For example, the abstract of a report of a randomised trial
included the statement that more patients in the zinc group
than in the control group recovered by two days (49% v 32%,
P=0.032).5 The difference in proportions was Est = 17
percentage points, but what is the 95% confidence interval (CI)?
Following the steps in the box we calculate the CI as follows:
z = 0.862+ [0.743 2.404log(0.032)] = 2.141;
SE = 17/2.141 = 7.940, so that 1.96SE = 15.56 percentage
points;
95% CI is 17.0 15.56 to 17.0 + 15.56, or 1.4 to 32.6
percentage points.
(b) Calculating the confidence interval for
a ratio (log transformation needed)
The calculation is trickier for ratio measures, such as risk ratio,
odds ratio, and hazard ratio. We need to log transform the
estimate and then reverse the procedure, as described in a
previous Statistics Note.6
For example, the abstract of a report of a cohort study includes
the statement that In those with a [diastolic blood pressure]
reading of 95-99 mm Hg the relative risk was 0.30 (P=0.034).7
What is the confidence interval around 0.30?
Correspondence to: D G Altman doug.altman@csm.ox.ac.uk
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 1 of 2
1 2
Following the steps in the box we calculate the CI as follows:
z = 0.862+ [0.743 2.404log(0.034)] = 2.117;
Est = log (0.30) = 1.204;
SE = 1.204/2.117 = 0.569 but we ignore the minus sign,
so SE = 0.569, and 1.96SE = 1.115;
95% CI on log scale = 1.204 1.115 to 1.204 + 1.115 =
2.319 to 0.089;
95% CI on natural scale = exp (2.319) = 0.10 to exp
(0.089) = 0.91.
Hence the relative risk is estimated to be 0.30 with 95% CI
0.10 to 0.91.
Limitations of the method
The methods described can be applied in a wide range of
settings, including the results from meta-analysis and regression
analyses. The main context where they are not correct is in small
samples where the outcome is continuous and the analysis has
been done by a t test or analysis of variance, or the outcome is
dichotomous and an exact method has been used for the
confidence interval. However, even here the methods will be
approximately correct in larger studies with, say, 60 patients or
more.
P values presented as inequalities
Sometimes P values are very small and so are presented as
P<0.0001 or something similar. The above method can be
applied for small P values, setting P equal to the value it is less
than, but the z statistic will be too small, hence the standard
error will be too large and the resulting CI will be too wide.
This is not a problem so long as we remember that the estimate
is better than the interval suggests.
When we are told that P>0.05 or the difference is not significant,
things are more difficult. If we apply the method described here,
using P=0.05, the confidence interval will be too narrow. We
must remember that the estimate is even poorer than the
confidence interval calculated would suggest.
Subscribe: http://www.bmj.com/subscribe
BMJ 2011;343:d2090 doi: 10.1136/bmj.d2090 (Published 8 August 2011) Page 2 of 2

RESEARCH METHODS & REPORTING

Steps to obtain the confidence interval (CI) for an estimate of effect from the P value and the estimate (Est)
(a) CI for a difference
1 calculate the test statistic for a normal distribution test, z, from P3: z = 0.862 + [0.743 2.404log(P)]
2 calculate the standard error: SE = Est/z (ignoring minus signs)
3 calculate the 95% CI: Est 1.96SE to Est + 1.96SE.

(b) CI for a ratio

For a ratio measure, such as a risk ratio, the above formulas should be used with the estimate Est on the log scale (eg, the log risk ratio). Step
3 gives a CI on the log scale; to derive the CI on the natural scale we need to exponentiate (antilog) Est and its CI.4
Notes
All P values are two sided.
All logarithms are natural (ie, to base e). 4
For a 90% CI, we replace 1.96 by 1.65; for a 99% CI we use 2.57.

1 Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather 6 Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ
than hypothesis testing. BMJ 1986;292:746-50. 2003;326:219.
2 Moher D, Hopewell S, Schulz KF, Montori V, Gtzsche PC, Devereaux PJ, et al. 7 Lindblad U, Rstam L, Rydn L, Ranstam J, Isacsson S-O, Berglund G. Control of blood
CONSORT 2010. Explanation and Elaboration: updated guidelines for reporting parallel pressure and risk of first acute myocardial infarction: Skaraborg hypertension project.
group randomised trials. BMJ 2010;340:c869. BMJ 1994;308:681.
3 Lin J-T. Approximating the normal tail probability and its inverse for use on a pocket
calculator. Appl Stat 1989;38:69-70.
4 Bland JM, Altman DG. Statistics Notes. Logarithms. BMJ 1996;312:700. Cite this as: BMJ 2011;343:d2090
5 Roy SK, Hossain MJ, Khatun W, Chakraborty B, Chowdhury S, Begum A, et al. Zinc
BMJ Publishing Group Ltd 2011
supplementation in children with cholera in Bangladesh: randomised controlled trial. BMJ
2008;336:266-8.

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
BMJ 2011;343:d2304 doi: 10.1136/bmj.d2304 Page 1 of 2

Research Methods & Reporting

RESEARCH METHODS & REPORTING

STATISTICS NOTES

How to obtain the P value from a confidence interval

Douglas G Altman professor of statistics in medicine 1, J Martin Bland professor of health statistics 2

1
Centre for Statistics in Medicine, University of Oxford, Oxford OX2 6UD; 2Department of Health Sciences, University of York, Heslington, York
YO10 5DD

We have shown in a previous Statistics Note1 how we can
calculate a confidence interval (CI) from a P value. Some
published articles report confidence intervals, but do not give
corresponding P values. Here we show how a confidence interval
can be used to calculate a P value, should this be required. This
might also be useful when the P value is given only imprecisely
(eg, as P<0.05). Wherever they can be calculated, we are
advocates of confidence intervals as much more useful than P
values, but we like to be helpful.
The method is outlined in the box below in which we have
distinguished two cases.
(a) P from CI for a difference (no
transformation needed)
The simple case is when we have a CI for the difference between
two means or two proportions. For example, participants in a
trial received antihypertensive treatment with or without
pravastatin. The authors report that pravastatin performed
slightly worse than a placebo. The estimated difference between
group means was 1.9 (95% CI 0.6 to 4.3) mm Hg.4 What was
the P value?
Following the steps in the box above we calculate P as follows:
SE = [4.3 (0.6)]/(21.96) = 1.25;
z = 1.9/1.25 = 1.52;
P = exp(0.7171.52 0.4161.522) = 0.13.
In this paper, the authors did indeed publish a P value of 0.13,4
as we have estimated from their confidence interval.
(b) CI for a ratio (log transformation
needed)
The calculation is trickier for ratio measures, such as risk ratio,
odds ratio, and hazard ratio. We need to log transform the
estimate and confidence limits, so that Est, l, and u in the box
are the logarithms of the published values.
For example, in a meta-analysis of several studies comparing
Taggart et al presented a hazard ratio of 0.81; 95% CI 0.70 to
0.94.5 They did not quote the P value.
Following the steps in the box we calculate P as follows:
Est = log(0.81) = 0.211
l = log(0.70) = 0.357, u = log (0.94) = 0.062
SE = [0.062 (0.357)]/(21.96) = 0.0753.
z = 0.211/0.0753 = 2.802. We take the positive value of
z, 2.802.
P = exp(0.7172.802 0.4162.8022) = 0.005.
Limitations of the method
The formula for P is unreliable for very small P values and if
your P value is smaller than 0.0001, just report it as P<0.0001.
The methods described can be applied in a wide range of
settings, including the results from meta-analysis and regression
analyses. The main context where they are not correct is small
samples where the outcome is continuous and the analysis has
been done by a t test or analysis of variance, or the outcome is
dichotomous and an exact method has been used for the
confidence interval. However, even here the methods will be
approximately correct in larger studies with, say, 60 patients or
more.
Contributors: JMB and DGA jointly wrote and agreed the text.
Competing interests: All authors have completed the Unified Competing
Interest form at www.icmje.org/coi_disclosure.pdf (available on request
from the corresponding author) and declare: no support from any
organisation for the submitted work; no financial relationships with any
organisations that might have an interest in the submitted work in the
previous 3 years; no other relationships or activities that could appear
to have influenced the submitted work.
1 Altman DG, Bland JM. How to obtain a confidence interval from a P value. BMJ
2
2011;342:d2090.
Lin J-T. Approximating the normal tail probability and its inverse for use on a pocket
calculator. Appl Stat 1989;38:69-70.

single versus bilateral mammary artery coronary bypass grafts

Correspondence to: D G Altman doug.altman@csm.ox.ac.uk

Reprints: http://journals.bmj.com/cgi/reprintform Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe

BMJ 2011;343:d2304 doi: 10.1136/bmj.d2304 Page 2 of 2

RESEARCH METHODS & REPORTING

Steps to obtain the P value from the CI for an estimate of effect (Est)

(a) P from CI for a difference

If the upper and lower limits of a 95% CI are u and l respectively:
1 calculate the standard error: SE = (u l)/(21.96)
2 calculate the test statistic: z = Est/SE
3 calculate the P value2: P = exp(0.717z 0.416z2).

(b) P from CI for a ratio

For a ratio measure, such as a risk ratio, the above formulas should be used with the estimate Est and the confidence
limits on the log scale (eg, the log risk ratio and its CI).
Notes
All P values are two sided.
All logarithms are natural (ie, to base e).3
exp is the exponential function.
The formula for P works only for positive z, so if z is negative we remove the minus sign.
For a 90% CI, we replace 1.96 by 1.65; for a 99% CI we use 2.57.

3 Mancia G, Parati G, Revera M, Bilo G, Giuliano A, Veglia F, et al. Statins, antihypertensive 5 Bland JM, Altman DG. Logarithms. BMJ 1996;312:700.
treatment, and blood pressure control in clinic and over 24 hours: evidence from PHYLLIS
randomised double blind trial. BMJ 2010;340:c1197.
4 Taggart DP, DAmico R, Altman DG. Effect of arterial revascularisation on survival: a Cite this as: BMJ 2011;343:d2304
systematic review of studies comparing bilateral and single internal mammary arteries.
Lancet 2001;358:870-5.

Reprints: http://journals.bmj.com/cgi/reprintform Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe

BMJ 2011;343:d570 doi: 10.1136/bmj.d570
Research Methods & Reporting
RESEARCH METHODS & REPORTING
STATISTICS NOTES
Brackets (parentheses) in formulas
Douglas G Altman professor of statistics in medicine 1, J Martin Bland professor of health statistics 2
1
Centre for Statistics in Medicine, University of Oxford, Oxford OX2 6UD; 2Department of Health Sciences, University of York, York YO10 5DD
Each year, new health sciences postgraduate students in York
are given a simple maths test. Each year the majority of them
fail to calculate 20 3 5 correctly. According to the
conventional rules of arithmetic, division and multiplication are
done before addition and subtraction, so 20 3 5 = 20 15
= 5. Many students work from left to right and calculate 20
3 5 as 17 5 = 85. If that was what was actually meant, we
would need to use brackets: (20 3) 5 = 17 5 = 85. Brackets
tell us that the enclosed part must be evaluated first. That
convention is part of various mnemonic acronyms which indicate
the order of operations, such as BODMAS (Brackets, Of (that
is, power of), Divide, Multiply, Add, Subtract) and PEMDAS
(Parentheses, Exponentiation, Multiplication, Division, Addition,
Subtraction).1
Schoolchildren learn the basic rules about how to construct and
interpret mathematical formulas.1 The conventions exist to
ensure that there is absolutely no ambiguity, as mathematics
(unlike prose) has no redundancy, so any mistake may have
serious consequences. Our experience is that mistakes are quite
common when formulas are presented in medical journal
articles. A particular concern is that brackets are often omitted
or misused. The following examples are typical and we mean
nothing personal by choosing them.
Example 1
In a discussion of methods for analysing diagnostic test
accuracy, Collinson 2 wrote:
Sensitivity = TP/TP + FN
where TP = true positive and FN = false negative. The formula
should, of course, be:
Sensitivity = TP/(TP + FN).
Example 2
For a non-statistical example, Leyland 3 wrote that the total
optical power of the cornea is:
P = P1 + P2 t/n2(P1P2)
Correspondence to: DG Altman doug.altman@csm.ox.ac.uk
Reprints: http://journals.bmj.com/cgi/reprintform
Page 1 of 2
where P1 = n2 n1/r1 and P2 = n3 n2/r2. Here n1, n2, and n3 are
refractive indices, r1, r2, and t are distances in metres, and P, P1,
and P2, are powers in dioptres. But he should have written P1 =
(n2 n1)/r1 and P2 = (n3 n2)/r2. P1 = n2 n1/r1 is clearly wrong
dimensionally, as P1 is dioptres, 1/metre, n2 and n1 are ratios
and so pure numbers, and r1 is in metres. Also, it is not clear
whether t/n2 (P1P2) means (t/n2) P1P2, which it does, or t/(n2P1P2).
Do such errors matter? Certainly. In our experience the
calculations are usually correct in the paper, but anyone using
the published formula would go wrong. Sometimes, however,
the incorrect formula was used, as in the following case.
Example 3
In their otherwise exemplary evaluation of the chronic ankle
instability scale, Eechaute et al4 made a mistake in their formula
for the minimal detectable change (MDC) or repeatability
coefficient,5 writing: MDC = 2.04 (2 SEM). Here SEM is
the standard error of a measurement or within subject standard
deviation.5 This formula uses 2.04 where 2 or 1.96 is more
usual,6 but, much more seriously, the SEM should not be
included within the square root, as the brackets indicate. This
might be dismissed as a simple typographical error, but the
authors actually used this incorrect formula. Their value of SEM
was 2.7 points, so they calculated the minimal detectable change
as 2.04 (2 2.7) = 4.7. They should have calculated 2.04
(2) 2.7 = 7.8. Their erroneous formula makes the scale appear
considerably more reliable than it actually is.6 The formula is
also wrong in terms of dimensions, because the minimum
clinical difference should be in the same units as the
measurement, not in square root units.
Some mistakes in formulas may be present in a submitted
manuscript, but others might be introduced in the publication
process. For example, problems sometimes arise when a
displayed formula is converted to an in-text formula as part
of the editing, and the implications are not realised or not noticed
by either editing staff or authors. Often it is necessary to insert
brackets when reformatting a formula. So the simple formula:
Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe
BMJ 2011;343:d570 doi: 10.1136/bmj.d570
should be changed to p/(1 p) if moved to the text.
Formulas in published articles may be used by others, so
mistakes may lead to substantive errors in research. It is essential
that authors and editors check all formulas carefully.
Acknowledgements: We are very grateful to Phil McShane for pointing
out a mistake in an earlier version of this statistics note.
Contributors: DGA and JMB jointly wrote and agreed the text.
Competing interests: All authors have completed the Unified Competing
Interest form at www.icmje.org/coi_disclosure.pdf (available on request
Reprints: http://journals.bmj.com/cgi/reprintform
Page 2 of 2
RESEARCH METHODS & REPORTING
from the corresponding author) and declare: no support from any
organisation for the submitted; no financial relationships with any
organisations that might have an interest in the submitted work in the
previous 3 years; no other relationships or activities that could appear
to have influenced the submitted work
1 Wikipedia. Order of operations. [cited 2010 Nov 23]. http://en.wikipedia.org/wiki/Order_
of_operations
2 Collinson P. Of bombers, radiologists, and cardiologists: time to ROC. Heart
1998;80:215-7.
3 Leyland M. Validation of Orbscan II posterior corneal curvature measurement for intraocular
lens power calculation. Eye 2004;18:357-60.
4 Eechaute C, Vaes P, Duquet W. The chronic ankle instability scale: Clinimetric properties
of a multidimensional, patient-assessed instrument. Phys Ther Sport 2008;9:57-66.
5 Bland JM, Altman DG. Statistics Notes. Measurement error. BMJ 1996;312:744.
6 Bland JM. Minimal detectable change. Phys Ther Sport 2009;10:39.
Cite this as: BMJ 2011;343:d570
Subscribe: http://resources.bmj.com/bmj/subscribers/how-to-subscribe
BMJ 2013;346:f3438 doi: 10.1136/bmj.f3438 (Published 6 June 2013)
Research Methods & Reporting
RESEARCH METHODS & REPORTING
Statistics Notes: Missing outcomes in randomised
trials
Andrew J Vickers attending research methodologist , Douglas G Altman professor of statistics in
2
medicine
Department of Epidemiology and Biostatistics, Memorial Sloan-Kettering Cancer Center, New York, NY 10065, USA; 2Centre for Statistics in
1
Medicine, University of Oxford, Oxford OX2 6UD, UK
In most randomised trials, some patients fail to provide data for
study endpoints.1 We have previously described the analysis of
a trial of acupuncture versus sham acupuncture for the treatment
of shoulder pain.2 All 52 randomised patients provided baseline
data on pain and range of motion, but only 45 returned for
follow-up testing. The statistical question is how to handle those
seven patients with missing data. The most straightforward
approach is simply to ignore the seven patients and do what is
known as an available case analysis (often confusingly known
as complete case analysis). As not all randomised patients are
included in the analysis, this leads to reduced statistical power.1
A method that attempts to include all randomised patients is
last observation carried forward, in which the last
measurement obtained from the patient is used for all data points
that were subsequently missed. This method is attractive because
it is simple, but it has little else to recommend it. Substituting
a missing data point with a value is known as imputation,1
and the data analyst needs a clear rationale for the type of
imputation used. That a patients responses would remain the
same after drop-out is generally implausible. This is most
obvious in chronic degenerative diseases. For instance, cognitive
function scores decrease over time in dementia, so last
observation carried forward gives overoptimistic scores for
patients who drop out (figure). If a treatment was associated
with toxicity, and this led to earlier drop-out than in the control
group, the method would give results biased in favour of the
experimental arm.3 By contrast, shoulder pain generally gets
better over time, either because treatment is effective or because
of the placebo effect and regression to the mean.4 In the
randomised trial, patients in the control group improved by a
mean of 9.8 points out of 100 from baseline to post-treatment
follow-up, whereas patients who received acupuncture improved
by 21.5 points. So assuming that patients lost to follow-up
experienced precisely zero change in pain scores makes little
sense. Last observation carried forward may also underestimate
the standard deviation of the endpoint, especially in cases in
which the last observation is the baseline, leading to confidence
intervals that are too narrow.
Correspondence to: A Vickers vickersa@mskcc.org
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 1 of 4
1
A more sophisticated approach to missing data is known as
multiple imputation, which uses a regression model to predict
missing values.5 In randomised trials, the strongest predictors
of future outcome are often the scores provided by the patient
so far, but other variables can be included. To avoid
underestimating the width of the confidence interval, multiple
imputation involves a form of random sampling. For a given
patient with a missing outcome, regression is used to predict
the mean value of the missing outcome for similar patients and
also the variability around the mean; a value is then selected at
random from this distribution. The results from several
imputations (hence multiple) are combined using a method
known as Rubins rules.5 6 Multiple imputation is widely
believed to be the preferred approach to missing data, not just
for randomised trials.7 It is computationally complex, however,
and needs to be implemented by special software, such as the
ice command in Stata (see www.multiple-imputation.com).
The table shows the results of the shoulder pain study analysed
by each method. The estimates for available case and multiple
imputation do not differ much, although multiple imputation
has a slightly narrower confidence interval. Last observation
carried forward appears to be biasedit underestimates the
effects of acupunctureand gives a confidence interval that is
too narrow.
Multiple imputation works best when good predictors of
outcome are available. In the shoulder pain example, baseline
score was only moderately correlated with follow-up score
(r0.4). Had outcome been assessed halfway through treatment,
this measure would have been more highly correlated with
post-treatment score, markedly improving the properties of the
multiple imputation.
Multiple imputation has several important strengths, but it does
not adjust for the sort of bias created if patients were less likely
to return for follow-up if they were in a lot of pain; this is an
inherent limitation to missing data analysis. We cannot know
whether patients pain levels affect the chance that they will
complete a pain questionnaire because, obviously enough, we
do not have the pain scores of non-respondents.
Subscribe: http://www.bmj.com/subscribe
BMJ 2013;346:f3438 doi: 10.1136/bmj.f3438 (Published 6 June 2013)
Sometimes simple common sense is more important than
complex statistics. In the shoulder pain trial, three of the seven
drop-outs were in the acupuncture group and four were controls,
so it seems implausible that their omission had materially
affected the results of the trial. If drop-out rates were very
different between the two arms of a trial, that may raise concerns
about bias. Above all, analysis of missing data teaches us the
importance of avoiding missing data in the first place: an
informed guess, even using a technique as sophisticated as
multiple imputation, is still a guess.
Contributors: AJV and DGA jointly wrote and agreed the text.
Competing interests: All authors have completed the ICMJE uniform
disclosure form at www.icmje.org/coi_disclosure.pdf (available on
request from the corresponding author) and declare: no support from
any organisation for the submitted work; no financial relationships with
any organisations that might have an interest in the submitted work in
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 2 of 4
RESEARCH METHODS & REPORTING
the previous three years; no other relationships or activities that could
appear to have influenced the submitted work.
Provenance and peer review: Not commissioned; not externally peer
reviewed.
1 Altman DG, Bland JM. Missing data. BMJ 2007;334:424.
2 Vickers AJ, Altman DG. Statistics notes: analysing controlled trials with baseline and
follow up measurements. BMJ 2001;323:1123-4.
3 Molnar FJ, Man-Son-Hing M, Hutton B, Fergusson DA. Have
last-observation-carried-forward analyses caused us to favour more toxic dementia
therapies over less toxic alternatives? A systematic review. Open Med 2009;3:e31-50.
4 Bland JM, Altman DG. Regression towards the mean. BMJ 1994;308:1499.
5 Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation
for missing data in epidemiological and clinical research: potential and pitfalls. BMJ
2009;338:b2393.
6 White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and
guidance for practice. Stat Med 2011;30:377-99.
7 White IR, Horton NJ, Carpenter J, Pocock SJ. Strategy for intention to treat analysis in
randomised trials with missing outcome data. BMJ 2011;342:d40.
Cite this as: BMJ 2013;346:f3438
BMJ Publishing Group Ltd 2013
Subscribe: http://www.bmj.com/subscribe
BMJ 2013;346:f3438 doi: 10.1136/bmj.f3438 (Published 6 June 2013) Page 3 of 4

RESEARCH METHODS & REPORTING

Table

Table 1| Analysis of shoulder pain trial with three statistical methods

Analysis Effect of acupuncturedifference in points (95% CI) Standard error P value

Available case 14.2 (5.1 to 23.4) 4.53 0.003
Last observation carried forward* 12.6 (3.9 to 21.3) 4.33 0.005
Multiple imputation 14.3 (5.4 to 23.3) 4.45 0.002

*Missing final value replaced by baseline value (n=7).

Using baseline score and treatment group.

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
BMJ 2013;346:f3438 doi: 10.1136/bmj.f3438 (Published 6 June 2013) Page 4 of 4

RESEARCH METHODS & REPORTING

Figure

Function scores over time for patient with chronic degenerative disease

For personal use only: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe
BMJ 2014;349:g7064 doi: 10.1136/bmj.g7064 (Published 25 November 2014)
Research Methods & Reporting
RESEARCH METHODS & REPORTING
STATISTICS NOTES
Uncertainty and sampling error
Douglas G Altman professor of statistics in medicine , J Martin Bland professor of health statistics
Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford
1
OX3 7LD, UK; 2Department of Health Sciences, University of York, York YO10 5DD, UK
Medical research is conducted to help to reduce uncertainty.
For example, randomised controlled trials aim to answer
questions relating to treatment choices for a particular group of
patients. Rarely, however, does a single study remove
uncertainty. There are two reasons for this: sampling error and
other (non-sampling) sources of uncertainty. The word error
comes from a Latin root meaning to wander, and we use it in
its statistical sense of meaning variation from the average, not
mistake. Sampling error arises because any sample may not
behave quite the same as the larger population from which it
was drawn. Non-sampling error arises from the many ways a
research study may deviate from addressing the question that
the researcher wants to answer.
Sampling error is very much the concern of the statistician, who
imagines that the group of people in the study is just one of the
many possible samples from the population of interest. Despite
it being widely condemned,1 the dominant way of summarising
the evidence from a research study is by the P value. It should
be obvious that the evidence from a research study cannot
reasonably be summarised as just a single number, but the use
of P values remains unshakeable. Further, the practice of
labelling P values as significant or not significant leads not only
to dichotomous decisions but often also to the belief that the
research question has been answered.
P values represent the probability that the observed data (or a
more extreme result) could have arisen when the true effect of
interest is zerofor example, the true treatment effect in a
randomised trial. It is common to interpret P<0.05 (significant)
as clear evidence that there is a real effect, and P>0.05 (not
significant) as evidence that there is no effect. However, the
former interpretation may be unwise, and the latter is wrong.
Although 0.05 is the conventional decision point, P<0.05 is far
from representing certainty. One in 20 studies could have a
difference of the observed size if there were really no difference
in the population. Not significant indicates that we found
insufficient evidence to conclude that there is a real effect, not
that we have shown that there is not one.2 Referring to results
as statistically significant, or not, only helps a bit.
Correspondence to: D G Altman doug.altman@csm.ox.ac.uk
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 1 of 2
1 2
Interpretation of a studys results should be primarily based on
the estimated effect and a measure of its uncertainty. In
mainstream statistics, the uncertainty of estimates is indicated
by the use of confidence intervals. Before the mid-1980s,
confidence intervals were rarely seen in clinical research articles.
Around 1986 things changed,3 and these days almost all clinical
research articles in major journals include confidence intervals.
The confidence interval is a range of uncertainty around the
estimate of interest, such as the treatment effect in a controlled
trial.
So, for example, in a study of the impact of a mental health
worker on the management of depression in primary care, it
was reported that After adjustment for baseline depression,
mean depression score was 1.33 PHQ-9 points lower (95%
confidence interval 0.35 to 2.31, P=0.009) in participants
receiving collaborative care than in those receiving usual care
at four months.4 This means that we estimate that, in the
population which these trial participants represent, the average
difference in mean depression score if all were offered
collaborative care would be between 0.35 and 2.31 scale points
less than if all were treated in the usual way. It is only an
estimate. For 2.5% of studies the confidence interval will be
entirely below the true population difference, and 2.5% will
have the interval entirely above it. We dont think P=0.009
adds much to this, but researchers can seldom bear to do without
it. The inevitable uncertainty from sampling error can be reduced
by increasing the sample size, but usually only modestly. To
halve the width of the confidence interval we would need to
quadruple the sample size.
A common mistake is to believe that the confidence interval
expresses all the uncertainty. Rather, the confidence interval
expressed uncertainty from just one causenamely the
uncertainty due to having taken a sample from the population
defined by the inclusion criteria. Often there are other sources
of uncertainty that may be even more important to consider, in
particular relating to possibly biased results. We address these
in our linked statistics note.5
Contributors: DGA and JMB jointly wrote and agreed the text.
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;349:g7064 doi: 10.1136/bmj.g7064 (Published 25 November 2014)
Competing interests: All authors have completed the ICMJE uniform
disclosure form at www.icmje.org/coi_disclosure.pdf (available on
request from the corresponding author) and declare: no support from
any organisation for the submitted work; no financial relationships with
any organisations that might have an interest in the submitted work in
the previous three years; no other relationships or activities that could
appear to have influenced the submitted work.
Provenance and peer review: Not commissioned; not externally peer
reviewed.
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 2 of 2
RESEARCH METHODS & REPORTING
1 Sterne JA, Davey Smith G. Sifting the evidencewhats wrong with significance tests?
BMJ 2001;322:226-31.
2 Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ
1995;311:485.
3 Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather
than hypothesis testing. BMJ 1986;292:746-50.
4 Richards DA, Hill JJ, Gask L, Lovell K, Chew-Graham C, Bower P, et al. Clinical
effectiveness of collaborative care for depression in UK primary care (CADET): cluster
randomised controlled trial. BMJ 2013;347:f4913.
5 Altman DG, Bland JM. Uncertainty beyond sampling error. BMJ 2014;349:g7065.
Cite this as: BMJ 2014;349:g7064
BMJ Publishing Group Ltd 2014
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;349:g7065 doi: 10.1136/bmj.g7065 (Published 25 November 2014)
Research Methods & Reporting
RESEARCH METHODS & REPORTING
STATISTICS NOTES
Uncertainty beyond sampling error
Douglas G Altman professor of statistics in medicine , J Martin Bland professor of health statistics
Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford
1
OX3 7LD, UK; 2Department of Health Sciences, University of York, York YO10 5DD, UK
Statistical analysis of research results mainly uses confidence
intervals and hypothesis tests to capture the uncertainty rising
from our study being on a sample of participants drawn from a
much larger population, in which our interest mainly lies.1 But
beyond the issue of sampling variation there are other sources
of uncertainty that may be even more important to consider. In
measurement, a distinction is made between precision, which
is how variable are measurements on the same person by the
same method made at the same time, and accuracy, which is
how close the measurement is to what we actually want to know.
For example, if we were to ask a group of patients on two
occasions how much alcohol they typically consume, this would
enable us to estimate precision, how repeatable answers are, but
not how close these answers are to how much they actually
drink, which we might suspect to be higher. In the same way,
a confidence interval tells us about the precision of research
results, what would happen if we were to repeat the same study,
not their accuracy, which is how close the study is to the truth.
In general, beyond the imprecision or uncertainty of numerical
results arising from sampling, the main concern is the possibility
that the study results are biased. Recent developments in
appraising published randomised trials have switched from
considering quality (essentially undefinable) to assessing
explicitly the risk of bias in relation to the way the study was
done.2 Here sources of possible bias include lack of blinding
and losses to follow up (missing data). But bias is especially
relevant in non-randomised studies, where there will be major
concerns about possible confounding, where an apparent
relationship between two things may be the result of the
relationships of both to a third.
A further source of uncertainty concerns the extent to which the
results of research conducted in a particular setting with selected
participants can be taken as applying equally to a wider group
of patients in a different location. Judgement of generalisability
(also known as external validity) is challenging.3 In a clinical
trial, the part of the larger population we particularly care about
does not yet exist. We want to know what would happen to
future patients if we were to apply either of the trial treatments.
Changes over time in the nature of disease, in the fitness of the
Correspondence to: D G Altman doug.altman@csm.ox.ac.uk
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 1 of 2
1 2
patient, in nutrition, in ancillary treatments, and so on, may all
make our sample unrepresentative as a guide for future action.
For example, the UK review of evidence relating to
mammographic screening outlined three sources of uncertainty
in relation to the pooled estimated effect from a meta-analysis
of the results of all the randomised trials.4 First, there was
uncertainty due to sampling variation, as previously discussed.1
Second, there was uncertainty from some methodological
weaknesses of the trials. Third was uncertainty about whether
the results from the trials were still relevant 25 years later, after
major changes in cancer incidence, management of and
treatments for breast cancer, and the technology of
mammography. Unlike sampling variation, which is quantified
in the confidence interval and is uncontroversial, the uncertainty
from the other causes cannot easily be quantified and remains
the source of fierce debate.
Large numbers, increasingly common in this era of big data,
will produce narrow confidence intervals. These can create an
illusion of accuracy, but they ignore all sources of possible bias
that are not affected by sample size, and so these other sources
become relatively more important.5 6 A recent example of a very
precise but seriously wrong answer purported to show that skin
cancer was protective for heart attacks and all-cause mortality.7
So, although confidence intervals are a valuable way of depicting
uncertainty, they are always too narrow in the sense that they
reflect only statistical uncertainty, precision rather than accuracy.
Many journals require authors to consider in their discussion
the limitations of their studysome even require this in the
articles abstract. Issues raised there help readers to judge what
extra uncertainty might apply to the study, including whether
observed effects may be affected by bias. It is common for
authors to say that their results should be interpreted with
caution (including >700 in the BMJ), but who knows what that
means in practice? The GRADE group have developed a
framework for a more structured approach to assessing the
reliability of research findings that addresses the aspects outlined
above.8
Subscribe: http://www.bmj.com/subscribe
BMJ 2014;349:g7065 doi: 10.1136/bmj.g7065 (Published 25 November 2014)
Contributors: DGA and JMB jointly wrote and agreed the text.
Competing interests: All authors have completed the ICMJE uniform
disclosure form at www.icmje.org/coi_disclosure.pdf (available on
request from the corresponding author) and declare: no support from
any organisation for the submitted work; no financial relationships with
any organisations that might have an interest in the submitted work in
the previous three years; no other relationships or activities that could
appear to have influenced the submitted work.
Provenance and peer review: Not commissioned; not externally peer
reviewed.
1 Altman DG, Bland JM. Uncertainty and sampling error. BMJ 2014;349:g7064.
2 Higgins JP, Altman DG, Gtzsche PC, Jni P, Moher D, Oxman AD, et al. The Cochrane
Collaborations tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 2 of 2
RESEARCH METHODS & REPORTING
3 Rothwell PM. External validity of randomised controlled trials: to whom do the results of
this trial apply?. Lancet 2005;365:82-93.
4 Marmot MG, Altman DG, Cameron DA, Dewar JA, Thompson SG, Wilcox M. The benefits
and harms of breast cancer screening: an independent review. Br J Cancer
2013;108:2205-40.
5 Greenland S. Interval estimation by simulation as an alternative to and extension of
confidence intervals. Int J Epidemiol 2004;33:1389-97.
6 Kaplan RM, Chambers DA, Glasgow RE. Big data and large sample size: a cautionary
note on the potential for bias. Clin Transl Sci 2014;7:342-6.
7 Lange T, Keiding N. Skin cancer as a marker of sun exposure: a case of serious immortality
bias. Int J Epidemiol 2014;43:971.
8 Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE:
an emerging consensus on rating quality of evidence and strength of recommendations.
BMJ 2008;336:924-6.
Cite this as: BMJ 2014;349:g7065
BMJ Publishing Group Ltd 2014
Subscribe: http://www.bmj.com/subscribe

open access
1Department of Health Sciences,
University of York, York YO10
5DD, UK
2Centre for Statistics in
Medicine, Nuffield Department
of Orthopaedics, Rheumatology
and Musculoskeletal Sciences,
University of Oxford, Oxford
OX3 7LD, UK;
Correspondence to: J M Bland
martin.bland@york.ac.uk
Cite this as: BMJ 2015;350:h2622
doi: 10.1136/bmj.h2622
No of samplings
Histogram of 1000
resampling estimates of the
treatment difference from
the CADET data, with
corresponding normal
distribution curve, mean,
and 2.5 and 97.5 centiles
thebmj|BMJ 2015;350:h2622|doi: 10.1136/bmj.h2622
Statistics Notes: Bootstrap resampling methods
J Martin Bland,1 Douglas G Altman2
In medical research we study a sample of individuals to
make inferences about a target population. Estimates of
interest, such as a mean or a difference in proportions,
are calculated, usually accompanied by a confidence
interval derived from the standard error. The data from a
single sample are used here to quantify the variation in
the estimate of interest across (hypothetical) multiple
samples from the same population.1 As we have only
one sample we need to make assumptions about the
data. Most methods of analysis are called parametric
because they incorporate assumptions about the distri-
bution ofthe data, such as that observations follow a
normal distribution. Non-parametric methods avoid
assumptions about distributions but generally provide
only P values and not estimates of quantities of interest.2
For a given dataset the assumptions may not be met. In
such cases there is an alternative way to estimate stan-
dard errors and confidence intervals without any reli-
ance on assumed probability distributions. We use the
sample dataset and apply a resampling procedure called
the bootstrap. (In general language, a bootstrap method
is a self sustaining process that needs no e xternal input.)
The clever idea behind the bootstrap is to create mul-
tiple datasets from the real dataset without needing to
make any assumptions. Our observed sample is repre-
sentative of a population about which we wish to make
inferences, so a set of randomly chosen observations
from our sample will be equally representative of the
original population. We can generate a sample of the
same size as the original data set by randomly choosing
real observations one at a time. Each observation has an
equal chance of being chosen each time, so some obser-
vations will be picked more than once and some wont
be picked at all. That doesnt matter; the new boot-
strap sample is comparable to the original data set and
is equally representative of the target population.
For an example, CADET3 was a cluster randomised
trial comparing collaborative care for depression
detected in primary care with treatment as usual. The
outcome measure was the PHQ-9 depression scale,
and data were available for 505 participants. The esti-
mated mean difference (collaborative care minus
2.5% Mean 97.5%
200
150
100
50
0
-4 -3 -2 -1 0 1
Estimated treatment difference
Research Methods & ReportinG
treatment as usual) was 1.33 points on the PHQ-9
scale (95% confidence interval 2.31 to 0.35) adjusted
for baseline PHQ-9, age, the list size, index of multiple
deprivation, city of the practice, and clustering.
We created another sample of 505 by resampling as
described above, the full original sample being avail-
able for each of the 505 choices. The resulting new sam-
ple of 505 observations included 313 of the original 505
participants, some once, some more than once, a maxi-
mum of five times. The same regression analysis which
produced the original treatment effect estimate was
repeated for this new sample resulting in a slightly dif-
ferent estimated treatment difference of 1.25 points.
Instead of resampling once, we should do it many times
and use the variability of the results to obtain a confidence
interval. The distribution of the estimated treatment effect
from 1000 resamplings of the CADET data is shown in the
figure. The mean and standard deviation of this distribu-
tion are 1.353 and 0.565. This standard deviation provides
an alternative estimate of the standard error of the mean
difference between the treatments, which does not make
use of any theory about the distribution of the data. There
are two ways to use the bootstrap estimates to find a confi-
dence interval. If the resampling distribution is close to
normal, as is the case here, the 95% confidence interval
will be 1.353(1.960.565) to 1.353+(1.960.565), or
2.46 to 0.25. This interval is similar to that obtained
using the standard error from the least squares regression
on the real data. The other approach is to take the 95%
confidence interval directly from the 2.5th and 97.5th cen-
tiles of the distribution. For these data the bootstrap confi-
dence interval calculated this way is 2.44 to 0.26. This
second approach can be used regardless of the distribu-
tion of the bootstrap estimates.
Clearly we need enough repetitions so that the estimates
are stableusually thousands of bootstrap samples are
used, especially when using the observed centiles of the
distribution of estimates. A repetition of the whole boot-
strap analysis for CADET produced almost i dentical values
of the mean (1.335) and standard deviation (0.567).
This note gives the general idea of the bootstrap;
there are many variations.4 We can get a bootstrap esti-
mate for any quantity we can calculate from any sam-
ple. Bootstrap methods are particularly favoured by
health economists, because cost data tend to be highly
skewed and unsuited to conventional approaches.5
They are also useful for complex d atasetsfor exam-
ple, when the observations arent independent.
Contributors: JMB and DGA jointly wrote and agreed the text.
Competing interests: We have read and understood the BMJ Group policy
on declaration of interests and have no relevant interests to declare.
Provenance and peer review: Not commissioned; not externally peer
reviewed.
1 Altman DG, Bland JM. Standard deviations and standard errors.
BMJ 2005;331:903.
2 Altman DG, Bland JM. Parametric v nonparametric methods for data
analysis. BMJ 2009;338:a3167.
1
RESEARCH
3 Richards DA, Hill JJ, Gask L, etal. Clinical effectiveness of collaborative
care for depression in UK primary care (CADET): cluster randomised
controlled trial. BMJ 2013;347:f4913.
4 Carpenter J, Bithell J. Bootstrap confidence intervals: when, which,
what? A practical guide for medical statisticians. Stat Med
2000;19:1141-64.
No commercial reuse: See rights and reprints http://www.bmj.com/permissions
5 Schroeder E, Petrou S, Patel N, etal. Cost effectiveness of alternative
planned places of birth in woman at low risk of complications:
evidence from the Birthplace in England national prospective cohort
study. BMJ 2012;344:e2292.
BMJ Publishing Group Ltd 2015
Subscribe: http://www.bmj.com/subscribe
BMJ 2016;352:i189 doi: 10.1136/bmj.i189 (Published 15 January 2016)
Research Methods & Reporting
RESEARCH METHODS & REPORTING
STATISTICS NOTES
Inverse probability weighting
Mohammad Ali Mansournia assistant professor of epidemiology , Douglas G Altman professor of
2
statistics in medicine
1
Department of Epidemiology and Biostatistics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran; 2Centre for Statistics
in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
Statistical analysis usually treats all observations as equally
important. In some circumstances, however, it is appropriate to
vary the weight given to different observations. Well known
examples are in meta-analysis, where the inverse variance
(precision) weight given to each contributing study varies, and
in the analysis of clustered data.1
Differential weighting is also used when different parts of the
population are sampled with unequal probabilities of selection.
Two examples of intentional unbalanced sampling are:
1.Surveys with unequal probabilities of selectionIn a
national survey of hypertension prevalence, certain groups
with relatively rare characteristics (such as people aged
65 years) were oversampled to improve the precision of
estimates for those groups.2
2.Two-phase prevalence studiesIn the first phase of a
two-phase prevalence study of mental health status, the
sampled patients completed a short screening questionnaire.
In the second phase, a subsample was selected for a
definitive diagnostic test with oversampling of the
screen-positive cases to ensure precise estimates for
diagnostic prevalence.3
In such cases the ordinary unweighted sample quantities,
such as means or proportions, are likely to be biased
estimates of their corresponding population quantities. This
selection bias can be eliminated by performing a
weighted estimation, giving each individuals data a weight
inversely proportional to their probability of selection.
Intuitively, the weighting is used to deflate the weight for
those individuals who are oversampled. The weighted
analysis can be thought of as creating a study with no
differential selection.
Inverse probability weighting can also be used when
individuals vary in their probability of having missing
information. Two contexts where there may be
unintentional unbalanced selection are:
3.Studies with missing outcome dataIn surveys such as
that mentioned in example 1, the response rates will be
Correspondence to: M A Mansournia mansournia_ma@yahoo.com
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 1 of 2
1
affected by availability or willingness to participate.
Likewise in a cohort study of the effect of obesity on
hypertension, some individuals are censored due to loss to
follow-up (such as emigration) or competing risks (such
as death from other causes).4 In each case the amount of
missing information will vary across subgroups.
4.Randomised trials with crossing over from one arm to the
otherIn a randomised trial 8010 postmenopausal women
with early breast cancer were assigned to tamoxifen
(n=2459) or letrozole (n=2463) for five years or to
sequential treatment with two years of one of these agents
followed by three years of the other. There was a selective
crossover to letrozole of 619 patients in the tamoxifen arm
after significant benefit was reported for letrozole compared
with tamoxifen during the study. These 619 women may
be artificially censored at the time they crossed over for
analysis.5
In these situations, missing outcomes are unlikely to happen
at random so that estimates will be biased. While the
selection probabilities in examples 1 and 2 are known, the
response or non-censoring probabilities in examples 3 and
4 are unknown. Inverse probability weighting can be used
with weights estimated from a logistic regression model
for predicting non-response or censoring. As in the first
scenario, this application of the method aims to remove
bias, but it is more controversial. Its validity relies on a
correctly specified model including all prognostic variables
associated with non-response or censoring, which cannot
be assured.
In the breast cancer trial (example 4), although the
intention-to-treat hazard ratio for overall survival (which
ignores selective crossover) was 0.87 (95% confidence
interval 0.77 to 1.00) in favour of letrozole, the adjusted
hazard ratio using inverse probability of selection weights
was 0.79 (0.69 to 0.90), suggesting that the true effect is
greater than the intention-to-treat estimate.5
Subscribe: http://www.bmj.com/subscribe
BMJ 2016;352:i189 doi: 10.1136/bmj.i189 (Published 15 January 2016)
In observational studies, the probability of exposure can
depend on external factors (called confounders) that also
affect the outcome. The causal effect of interest is then
confused with the effects of confounders. Such confounding
can be thought as a type of selection bias, because
confounding essentially means that some causes of the
outcome also influence selection for the exposure. A
particularly important context is:
5.Non-randomised studies comparing different
treatmentsIn a cohort study 12 552 warfarin-naive
patients with atrial fibrillation admitted to hospital for
ischaemic stroke and treated with warfarin were compared
with patients who received no oral anticoagulant at
discharge.6
Outside randomised trials the choice of treatment is likely to be
influenced by predictors of outcome, so called confounding
by indication.7 Various strategies are used to try to remove the
bias in non-randomised treatment comparisons. The
conventional approach is to use multivariable regression, but a
recent alternative is inverse probability of treatment weighting.
Here the weights are based on each individuals probability of
receiving a specific treatment given the confounders, which is
known as the propensity score (PS). The weights are 1/PS for
the treated participants and 1/(1PS) for the untreated
participants.8 The weights can be estimated from a logistic
regression model for predicting treatment. Key assumptions are
that all confounders have been measured and properly modelled
in this treatment model. In the warfarin study (example 5) the
unadjusted hazard ratio for cardiac events was 0.73 (99%
confidence interval 0.67 to 0.80) in favour of warfarin, whereas
the adjusted estimate using inverse probability of treatment
weighting was 0.87 (0.78 to 0.98), about half the effect size.6 If
the cohort is also affected by censoring (see example 3 above),
one can adjust simultaneously for confounding and selection
bias due to censoring.4 8
Although helpful for bias reduction, estimates weighted by
design weights (examples 1 and 2) tend to be less precisely
estimated than the unweighted estimates, which is not
For personal use only: See rights and reprints http://www.bmj.com/permissions
Page 2 of 2
RESEARCH METHODS & REPORTING
necessarily true for examples 3-5. The ordinary 95% confidence
interval for inverse probability weighted estimates may not
provide the correct coverage and should be avoided. Instead,
robust sandwich variance estimators or non-parametric
bootstrapping should be used to provide valid confidence
intervals.8 Deeper discussion of inverse probability weighting
methods can be found elsewhere.8 9
Contributors: MAM and DGA jointly wrote and agreed the text.
Competing interests: We have read and understood the BMJ Group
policy on declaration of interests and have no relevant interests to
declare.
Provenance and peer review: Not commissioned; not externally peer
reviewed.
1 Kerry SM, Bland JM.. Analysis of a trial randomised in clusters. BMJ
1998;316: 54. 9451271
2 Korn EL, Graubard BI.. Epidemiologic studies utilizing surveys: accounting for the sampling
3
design. Am J Public Health 1991;81: 1166-73. 1951829
Vzquez-Barquero JL, Garca J, Simn JA et al. Mental health in primary care. An
epidemiological study of morbidity and use of health resources. Br J Psychiatry
4
1997;170: 529-35. 9330019
Alonso A, Segu-Gmez M, de
Irala J, Snchez-Villegas A, Beunza JJ, Martnez-Gonzalez MA.. Predictors of follow-up
and assessment of selection bias from dropouts using inverse probability weighting in a
cohort of university graduates. Eur J Epidemiol 2006;21: 351-8. 16736275
5 Regan MM, Neven P, Giobbie-Hurder A et al. BIG 1-98 Collaborative Group International
Breast Cancer Study Group (IBCSG). Assessment of letrozole and tamoxifen alone and
in sequence for postmenopausal women with steroid hormone receptor-positive breast
cancer: the BIG 1-98 randomised clinical trial at 81 years median follow-up. Lancet Oncol
2011;12: 1101-8. 22018631
6 Xian Y, Wu J, OBrien EC et al. Real world effectiveness of warfarin among ischemic
stroke patients with atrial fibrillation: observational analysis from Patient-Centered Research
into Outcomes Stroke Patients Prefer and Effectiveness Research (PROSPER) study.
BMJ 2015;351: h3786. 26232340
7 Freemantle N, Marston L, Walters K, Wood J, Reynolds MR, Petersen I.. Making
inferences on treatment effects from real world data: propensity scores, confounding by
indication, and other perils for the unwary in observational research. BMJ
2013;347: f6409. 24217206
8 Robins JM, Hernn MA, Brumback B.. Marginal structural models and causal inference
in epidemiology. Epidemiology 2000;11: 550-60. 10955408
9 Seaman SR, White IR.. Review of inverse probability weighting for dealing with missing
data. Stat Methods Med Res 2013;22: 278-95. 21220355
Accepted: 05 01 2016
Published by the BMJ Publishing Group Limited. For permission to use (where not already
granted under a licence) please go to http://group.bmj.com/group/rights-licensing/
permissions
Subscribe: http://www.bmj.com/subscribe