Professional Documents
Culture Documents
Treating Individuals 1
External validity of randomised controlled trials:
To whom do the results of this trial apply?
Lancet 2005; 365: 8293
See Comment page 13
Stroke Prevention Research
Unit, University Department of
Clinical Neurology, Radcliffe
Inrmary, Oxford OX2 6HE, UK
(P M Rothwell FRCP)
peter.rothwell@clneuro.ox.ac.uk
Peter M Rothwell
In making treatment decisions, doctors and patients must take into account relevant randomised controlled trials
(RCTs) and systematic reviews. Relevance depends on external validity (or generalisability)ie, whether the results
can be reasonably applied to a denable group of patients in a particular clinical setting in routine practice. There is
concern among clinicians that external validity is often poor, particularly for some pharmaceutical industry trials, a
perception that has led to underuse of treatments that are effective. Yet researchers, funding agencies, ethics
committees, the pharmaceutical industry, medical journals, and governmental regulators alike all neglect external
validity, leaving clinicians to make judgments. However, reporting of the determinants of external validity in trial
publications and systematic reviews is usually inadequate. This review discusses those determinants, presents a
checklist for clinicians, and makes recommendations for greater consideration of external validity in the design and
reporting of RCTs.
Between measurements based on RCTs and benet . . .
in the community there is a gulf which has been much
under-estimated
A L Cochrane, 19711
At its best a trial shows what can be accomplished with
a medicine under careful observation and certain
restricted conditions. The same results will not
invariably or necessarily be observed when the medicine
passes into general use.
Austin Bradford Hill, 19842
For personal use. Only reproduce with permission from Elsevier Ltd
Series
For personal use. Only reproduce with permission from Elsevier Ltd
83
Series
Events/Patients
Medical
Surgical
Events/Patients
Surgical
Medical
Fast centres
Slow centres
17/174
29/206
21/127
11/139
83
56
01 to 167
126 to 14
8/162
25/173
25/106
25/113
199
76
107 to 291
20 to 171
Total
46/380
32/266
09
45 to 64
33/335
50/219
135
69 to 202
10
10
20
10
30
10
20
30
10
20
30
B
Fast centres
Slow centres
21/174
38/206
28/127
25/139
118
03
24 to 212
86 to 93
Total
59/380
53/266
57
08 to 122
10
10
20
11/162
35/173
31/106
30/113
236
62
137 to 335
43 to 166
46/335
61/219
147
74 to 219
30
10
Figure 1: Absolute reductions in 5-year risks of ipsilateral ischaemic stroke (upper) and any stroke or death (lower) with surgery in ECST60 centres
Median delay from last symptomatic event to randomisation in these centres was 50 days (fast centres) compared with centres with a longer delay (slow centres). Data are shown separately for
patients with moderate (5069% [left of gure]) and severe (7099% [(right of gure]) carotid stenosis. A: ipsilateral ischaemic stroke or operative stroke/death. B: any stroke or operative death.
Trial setting
There is often concern about the generalisability of
trials done in secondary or tertiary care to practice in
primary care,2629 but there are several other ways in
which the setting of an RCT can affect external validity.
The country
Even if the health-care systems are similar, other
national differences can still affect generalisability.
Continuing with the example of cerebrovascular
disease, there are many differences between countries
in methods of diagnosis and management,62 as well as
84
For personal use. Only reproduce with permission from Elsevier Ltd
Series
Use of patch
Operations in which
patch used (%)
100
47
p<00001
37
36
35
32
40
21
15
17
20
13
2
De
nm
ar
k
Fin
lan
d
Fr
an
ce
UK
Ne
th
er
lan
ds
Ita
ly
en
Po
rtu
ga
l
ed
in
Sw
Sp
a
lgi
u
Be
No
rw
ay
Ge
rm
an
y
Country
6 (110)
2 (44) 6 (57) 8 (70) 4 (96) 1 (31) 12 (152) 27 (630) 11 (266)11 (153) 3 (40) 2 (80)
Use of shunt
100
89
80
67
58
50
60
45
p<00001
41
30
40
19
15
20
10
12
1
Fr
an
ce
De
nm
ar
k
ly
Ita
ain
Po
rtu
ga
l
Ne
th
er
lan
ds
Sp
Fin
lan
d
UK
No
rw
ay
Be
lgi
um
Ge
rm
an
y
Sw
ed
en
Country
Number of 6 (110) 4 (96)
centres
(patients)
6 (57) 2 (44) 27 (630) 2 (80) 8 (70) 1 (31) 11 (266) 12 (152) 3 (40) 11 (153)
Figure 2: Variation in rate of use of two ancillary surgical techniques used during carotid endarterectomy
(patch angioplasty [upper] and use of an intraoperative shunt [lower]) by country in the ECST60
Events
77
60
Number of
centres
(patients)
Selection of patients
Only a small proportion of patients with a specic
disorder participate in a particular trial. For example,
only about one in every 200300 patients undergoing
carotid endarterectomy in North America at the time
got into the large, multicentre RCTs,72 and similar
proportions have been reported in breast cancer trials.73
These low rates mean that trials take many years to
recruit but are not a problem for external validity as long
as the patients randomised in the participating centres
are representative of the whole. As outlined below this
is not always the case.
81
80
Operations in which
shunt used (%)
Risk (%)
95% CI
1/724
014
0004
127/11686
111
0913
Patients
Death
ACAS
46 published series
11/724
150
0624
8 published series
92/2122
430
3552
0
Figure 3: Overall results of a meta-analysis of the operative risk of death (upper) and stroke and death (lower)
from all studies published between 1990 and 2000 inclusive that reported risks due to carotid
endarterectomy for asymptomatic stenosis71 compared with the same risks in ACAS69
For personal use. Only reproduce with permission from Elsevier Ltd
85
Series
Proportion
excluded at
each stage
Available patients
Cumulative
proportion
remaining
Affected patients
known to primary care
in the catchment of a
centre participating
in the RCT
Worst
case
Best
case
Patients referred
to secondary care
10%
90%
Patients referred
to unit doing
RCT
2%
72%
Patients under
care of trial
clinician
004%
58%
Eligible
0005%
52%
Not referred
10%90%
Referred to hospital
or unit not taking
part in the RCT
20%80%
Not eligible
10%90%
Not invited to
participate or
declined to
participate
20%80%
Run-in periods
Randomised
0001%
42%
Figure 4: Schematic diagram illustrating effect of multiple stages of selection inherent in clinical practice on
proportion of patients in catchment of participating centre entered into an RCT done in secondary care
Worst case assumes proportion of patients excluded at each stage is at top of range and best case is based on
lowest proportion of patients excluded.
Enrichment strategies
Patients who are likely to respond well to treatment are
sometimes actively recruited.99101 For example, some
trials of antipsychotic drugs have selectively recruited
patients who had previously had a good response to
such drugs.102 Other trials have excluded non-responders
in a run-in phase. One RCT of the acetylcholinesterase
inhibitor tacrine in Alzheimers disease recruited 632
patients to a 6-week enrichment phase in which patients
were given different doses of tacrine or placebo.103 After a
www.thelancet.com Vol 365 January 1, 2005
For personal use. Only reproduce with permission from Elsevier Ltd
Series
Surgery (n=1807)
Not operated (n=62)
Demography
Male sex
36 (58%)
Age in years
641 (87)
Cerebrovascular events within past 6 months
Hemispheric TIA or stroke
56 (90%)
Ocular event only
6 (10%)
Residual neurological signs
2 (3%)
Days since last symptoms
74 (56)
Other clinical data
Previous stroke
2(3%)
Systolic BP (mm Hg)
154 (272)
Diastolic BP (mm Hg)
890 (130)
Angina
15 (24%)
Previous myocardial infarction
7 (11%)
Previous coronary artery surgery
2 (3%)
Peripheral vascular disease
7 (11%)
Diabetes
8 (13%)
Current cigarette smoking
25 (40%)
Blood cholesterol (mmol/L)
64 (16)
Mean symptomatic carotid stenosis
60% (25)
Mean contralateral carotid stenosis
37% (26)
Operated (n=745)
No surgery
(n=1211)
1263 (72%)
625 (81)
869 (72%)
623 (80)
1495 (85%)
250 (15%)
106 (6%)
62 (53)
1038 (86%)
173 (14%)
78 (7%)
62 (52)
101 (6%)
151 (223)
862 (114)
305 (18%)
219 (13%)
47 (3%)
292 (16%)
208 (12%)
844 (48%)
64 (14)
62% (21)
42% (26)
78 (7%)
1502 (213)
863 (108)
190 (16%)
136 (11%)
23 (2%)
203 (17%)
145 (12%)
557 (46%)
64 (14)
59% (22)
37% (27)
Patients who were randomised to surgery but did not have the operation are compared with those who did and with patients
who were randomised to medical treatment only. TIA=transient ischaemic attack. BP=blood pressure. Data are mean (SD) or
number (%).
SPIRIT (n=651)
EAFT (n=225)
66%
47%
39%
9%
9%
11%
7%
55%
81%
48%
11%
7%
12%
14%
33 (11)
735
27
26
29 (07)
507
0*
13
190 (2.4250)
19 (0847)
p<00001
p=015
For personal use. Only reproduce with permission from Elsevier Ltd
87
Series
Disorder
Surrogate outcome
Clinical outcome
Treatment
Fluoride
Osteoporosis
Increase in bone density112
Antiarrhythmic drugs Postmyocardial infarction Reduction in ECG abnormalities113
Interferon
Multiple sclerosis
70% reduction in new
brain lesions using MRI 115118
Milrinone and
Heart failure
Improved exercise
Epoprostanol
tolerance 119,120
Ibopamine
Heart failure
Improved ejection fraction,
and heart rate variability123
ECG=electrocardiogram.
25 (59)
7 (28)
Physical function
22 (52)
20 (80)
17 (40)
21 (84)
16 (38)
2 (8)
General health
14 (34)
4 (16)
Social function
14 (34)
13 (52)
Vitality
13 (32)
3 (12)
5 (12)
4 (16)
Bodily pain
Surrogate outcomes
Many trials use surrogate outcomes, usually biological or
imaging markers that are believed to be indirect
measures of the effect of treatment on clinical outcomes.
However, as well as being of questionable clinical
relevance, surrogate outcomes are often misleading.
Each of the treatments in table 3 had a major benecial
effect on a surrogate outcome, but even though each
surrogate outcome was correlated with a relevant clinical
outcome in observational studies, the treatments proved
ineffective or harmful in subsequent large RCTs that
used these clinical outcomes.112124
Scales
001
01
10
10
100
Figure 5: Comparison of the elements of quality of life of greatest concern to patients with multiple sclerosis
with those elements that their clinicians believed most important
Based on selection of three of eight aspects of health-related quality of life assessed in short form-36 measures.130
88
For personal use. Only reproduce with permission from Elsevier Ltd
Series
Patient-centred outcomes
70
p <00001
60
50
40
p=090
30
20
10
80
0
S
M
Randomised
S*
M*
Non-randomised
For personal use. Only reproduce with permission from Elsevier Ltd
89
Series
90
For personal use. Only reproduce with permission from Elsevier Ltd
Series
References
1
Cochrane AL. Effectiveness and Efficiency: random reections on
Health Services. London: Nufeld Provincial Hospitals Trust, 1972.
2
Horton R. Common sense and gures: the rhetoric of validity in
medicine (Bradford Hill Memorial Lecture 1999). Stat Med 2000;
19: 314964.
3
Pocock SJ. Clinical trials: a practical approach. Chichester: John
Wiley, 1983.
4
Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical
trials. 3rd edn. New York: Springer 1998.
5
Black D. The limitations of evidence. J R Coll Physicians Lond 1998;
32: 2326.
6
Hampton JR. Size isnt everything. Stat Med 2002; 21: 280714.
7
Caplan LR. Evidence based medicine: concerns of a clinical
neurologist. J Neurol Neurosurg Psychiatry 2001; 71: 56974.
8
Evans JG. Evidence-based and evidence-biased medicine.
Age Ageing 1995; 24: 46163.
9
Charlton BG, Miles A. The rise and fall of EBM. Q J Med 1998; 91:
37174.
10 Naylor C. Grey zones in clinical practice: some limits to evidencebased medicine. Lancet 1995; 345: 84042.
11 Swales JD. Evidence-based medicine and hypertension. J Hypertens
1999; 17: 151116.
12 Morgan WKC. On evidence, embellishment and efcacy.
J Eval Clin Pract 1997; 3: 11722.
13 Feinstein AR, Horwitz RI. Problems in the evidence of evidencebased medicine. Am J Med 1997; 103: 52935.
14 Pashos CL, Normand ST, Garnkle JB, Newhouse JP, Epstein AM,
McNeil B. Trends in use of drug therapies in patients with acute
myocardial infarction: 19881992. J Am Coll Cardiol 1994; 23:
102330.
15 Gareld FB, Gareld JM. Clinical judgement and clinical practice
guidelines. Int J Technol Assess Health Care 2000; 16: 105060.
16 Cabana MB, Rand CS, Powe NR, et al. Why dont clinicians follow
clinical practice guidelines? A framework for improvement. JAMA
1999; 282: 145865.
17 Davis DA, Taylor-Vaisey A. Translating guidelines into practice, a
systematic review of theoretic concepts, practical experience and
research evidence in the adoption of clinical practice guidelines.
Can Med Assoc J 1997; 157: 40816.
18 Ford LG, Hunter CP, Diehr P, Frelick RW, Yates J. Effect of patient
management guidelines on physician practice patterns: the
community hospital oncology program experience. J Clin Oncol
1987; 5: 50411.
19 Grol R, Dalhuijsen J, Thomas S, et al. Attributes of clinical
guidelines that inuence use of guidelines in general practice:
observational study. BMJ 1998; 317: 85861.
20 Messerli FH. Antihypertensive therapy: -blockers and diuretics why do physicians not always follow guidelines? Proceedings/Baylor
University Medical Center 2000; 13: 12831.
21 Sonis J, Doukas D, Klinkman M, Reed B, Rufn MT. Applicability
of clinical trial results to primary care. J Am Med Soc 1998; 280:
1746.
22 Wilson S, Delaney BC, Roalfe A, et al. Randomized controlled trials
in primary care: case study. BMJ 2000; 321: 2427.
23 Sellors J, Cosby R, Trim K, et al, for the Seniors Medication
Assessment Research Trial (SMART) Group. Recruiting family
physicians and patients for a clinical trial: lessons learned.
Fam Pract 2002; 19: 99104.
24 Oswald N, Bateman H. Applying research evidence to individuals
in primary care: a study using non-rheumatic atrial brillation.
Fam Pract 1999; 16: 41419.
25 Fahey T. Applying the results of clinical trials to patients in general
practice: perceived problems, strengths, assumptions, and
challenges for the future. Br J Gen Pract 1998; 48: 117378.
26 Mant D. Can randomised trials inform clinical decisions about
individual patients? Lancet 1999; 353: 74346
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
For personal use. Only reproduce with permission from Elsevier Ltd
91
Series
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
92
For personal use. Only reproduce with permission from Elsevier Ltd
Series
103 Davis KL, Thal LJ, Gamzu ER, et al, and the Tacrine Collaborative
Study Group. A double-blind, placebo-controlled multicenter study
of tacrine for Alzheimers disease. N Engl J Med 1992; 327: 125359.
104 Hall JC, Mills B, Nguyen H, Hall JL. Methodologic standards in
surgical trials. Surgery 1996; 119: 46672.
105 Shapiro SH, Weijer C, Freedman B. Reporting the study
populations of clinical trials. Clear transmission or static on the
line? J Clin Epidemiol 2000; 53: 97379.
106 European Atrial Fibrillation Trial (EAFT) Study Group. Secondary
prevention in non-rheumatic atrial brillation after transient
ischaemic attack or minor stroke. Lancet 1993; 342: 125562.
107 Algra A, Francke CL, Koehler PJJ, for the Stroke Prevention in
Reversible Ischaemia Trial (SPIRIT) group. A randomized trial of
anticoagulants versus aspirin after cerebral ischaemia of presumed
arterial origin. Ann Neurol 1997; 42: 85765.
108 Gorter JW, for the Stroke Prevention in Reversible Ischaemia Trial
(SPIRIT) and European Atrial Fibrillation Trial (EAFT) groups.
Major bleeding during anticoagulation after cerebral ischaemia:
patterns and risk factors. Neurology 1999; 53: 131927.
109 Rothwell PM, Eliasziw M, Gutnikov SA, et al, for the Carotid
Endarterectomy Trialists Collaboration. Pooled analysis of
individual patient data from randomised controlled trials of
endarterectomy for symptomatic carotid stenosis. Lancet 2003; 361:
10716
110 Merlo J, Broms K, Lindblad U, et al. Association of outpatient
utilisation of non-steroidal anti-inammatory drugs and
hospitalised heart failure in the entire Swedish population.
Eur J Clin Pharmacol 2001; 57: 7175.
111 Rothwell PM, Slattery J, Warlow CP. A systematic review of the
risks of stroke and death due to carotid endarterectomy for
symptomatic stenosis. Stroke 1996; 27: 26065.
112 Riggs BL, Hodgson SF, OFallon WM, et al. Effect of uoride
treatment on fracture rate in postmenopausal women with
osteoporosis. N Engl J Med 1990; 322: 80209.
113 McAlistair FA, Teo KK. Antiarrhythmic therapies for the prevention
of sudden cardiac death. Drugs 1997; 54: 23552.
114 The Cardiac Arrhythmia Suppression Trial (CAST) Investigators.
Preliminary report: effect of encainide and ecainide on mortality
in a randomized trial of arrhythmia suppression after myocardial
infarction. N Engl J Med 1989; 321: 40612.
115 The IFNB Multiple Sclerosis Study Group. Interferon -1b is
effective in relapsing-remitting multiple sclerosis. I. Clinical results
of a multicenter, randomized, double-blind, placebo-controlled trial.
Neurology 1993; 43: 65561.
116 The IFNB Multiple Sclerosis Study Group and the University of
British Columbia MS/MRI Analysis Group. Interferon -1b in the
treatment of multiple sclerosis: nal outcome of the randomized
controlled trial. Neurology 1995; 45: 127785.
117 Jacobs LD, Cookfair DL, Rudick AR, et al. Intramuscular interferon
-1a for disease progression in relapsing multiple sclerosis.
Ann Neurol 1996; 39: 28594.
118 Ebers GC for the PRISMS Study Group. Randomised double-blind
placebo-controlled study of interferon -1a in relapsing-remitting
multiple sclerosis. Lancet 1998; 352: 1498504.
119 Di Bianco R, Shabetai R, Kostuk W, et al. A comparison of oral
milrinone, digoxin and their combination in the treatment of
patients with chronic heart failure. N Engl J Med 1989; 320: 67783.
120 Sueta CA, Gheorghiade M, Adams KF, et al, and the Epoprostenol
Multicentre Research Group. Safety and efcacy of epoprostenol in
patients with severe congestive heart failure. Am J Cardiol 1995; 75:
34A43A.
121 Packer M, Carver JR, Rodeheffer RJ, et al, for the Promise Study
Research Group. Effect of oral milrinone on mortality in severe
chronic heart failure. N Engl J Med 1991; 325: 146875.
122 Califf RM, Adams KF, McKenna WJ, et al. A randomized controlled
trial of epoprostenol therapy for severe congestive heart failure.
Am Heart J 1997; 134: 4454.
123 Yee KM, Struthers AD. Can drug effects on mortality in heart
failure be predicted by any surrogate outcome measure?
Eur Heart J 1997; 18: 186064.
124 Hampton JR, van Veldhuisen DJ, Kleber FX, et al. Randomised
study of ibopamine on survival in patients with advanced severe
heart failure. Lancet 1997; 349: 97177.
125 Coste J, Fermanian J, Venot A. Methodological and statistical
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
For personal use. Only reproduce with permission from Elsevier Ltd
93