GUEST EDITORIAL
Ten rules for reading clinical research reports
John M. Yancey, PhD
Louisville, Ky.
The Editor, along with most surgical scholars, has been
concerned about excesses in statistical support of submitted
and published surgical articles; problems range from a total
absence of obviously needed statistical assessment to point-
lessly overdone analysis of small numbers of experimental
observations. Somewhere in between is the use of a blatantly
wrong statistical test for analysis of data. Dr. Yancey is a
frequent consultant to the Journal and has prepared this
work at the Editor's request.
Prior to the recruitment of a statistical referee, The
British Journal of Surgery commissioned an audit of the
statistical standard of 28 research papers published by
that journal in the January to March 1987 issues. The
results of that audit were published in the July 1988 issue
of The British Journal of Surgery.1 The audit identified
only 39% of the articles as containing statistical errors so
serious as to have warranted r e j e c t i o n - h a d the evalua-
tion been made prior to publication. This rate of serious
error is below that reported by many other audits of the
clinical literature. 2-12
THE PROBLEM
One of the more comprehensive reviews of the meth-
odologic quality of the medical literature was published
by New England Journal of Medicine Books in 1986. 5
Those authors reviewed the reviews of more than 4,200
research reports published in prestigious medical jour-
nals. Among their findings were the following: "The 16
assessment articles published before 1970 described a
median of 23% of 2,063 research reports as meeting the
assessor's criteria for validity; the 12 published during or
after 1970 reported that a median of 6% of 2,172
publications met the assessors' criteria. ''S
It should be noted that the criteria for validity used
were not esoteric. Most of the reviews were asking only
if the authors' conclusions were valid extrapolations from
the data presented. Included among the many shocking
conclusions in that review of the reviews was the follow-
ing: "The findings reported in the 28 assessment articles
suggest that the average practitioner will find relatively
few journal articles that are scientifically sound in terms
of reporting usable data and providing even moderately
strong support for their inferences . . . . The mere fact
that research reports are published, even in the most
From the Department of Diagnosis and General Dentistry, School of
Dentistry, University of Louisville, Louisville, Kentucky.
Reprinted with permission from Am J Surg 1990,159:533-9.
8/8/60349
558
prestigious journals, is no guarantee of their quality. ''5
When the Editor-in-Chief asked me to conduct an
audit of the research papers published by The American
Journal of Surgery in 1987 and 1988, I envisioned an
article similar in format to the article published by The
British Journal of Surgery.1 After a cursory review of those
publications, however, it was clear that my article would
be quite similar to that by Dr. Murray. The proportion,
severity, and types of errors detected by my preliminary
audit were consistent with those reported by The British
Journal of Surgery audit.
Although a data analysis error rate substantially
below that reported by similar audits of the clinical
literature might be considered good news by some, the
bad news is that I found many 1987-1988 Journal articles
with methodologic errors so serious as to render invalid
the conclusions of the authors. So, rather than devoting
my time to a precise quantification of the errors made by
recent contributors to the Journal, I decided to consider
some possible solutions to this widespread defect in the
clinical research literature.
THE SOLUTION?
Numerous books and articles have already been
published with a view toward quantifying and correcting
this serious problem. 13 There is little indication the
problem is going away; rather, the data suggest the
problem is growing worse?
Perhaps the time has come for Journal readers to
deal with authors who misinterpret their own data the
way they deal with patients who misinterpret their own
symptoms: reinterpret the misinterpretations and then
draw conclusions based upon one's own analysis of the
hard data.
CAVEATS
Before I commence my list of caveat emptor rules for
reading the data analysis section of clinical research
articles, I feel a need to make several points. First, these
are my personal opinions. Second, I found little evidence
that any of the 1987-1988 Journal authors were intention-
ally trying to mislead their readers. Third, to say that the
conclusions of an article are invalid is not to say they are
wrong. A surgeon may decide whether to operate upon a
particular patient by consulting a Ouija board and still
come to a correct conclusion.
It is important to understand that the data analysis
errors identified by just about every review of the medical
American Journal of Orthodontics and Dentofacial Orthopedics
Volume 109, No. 5
literature are not trivial misunderstandings important
only to those of us who teach statistics to Ph.D. students
and clinical residents. Most of the errors are of the
magnitude of claiming to have scientific evidence that
one surgical procedure is superior to another when no
such valid scientific data are presented.
The fact ~:hat most authors sincerely believe that their
statistics do provide scientific support for their conclu-
sions does not render the problem any less serious. If
a surgeon based a clinical decision upon the output from
a Ouija board, because he sincerely thought it to be a
scientific instrument, his conclusion would still be sci-
entifically imTalid, professionally incompetent, and ethi-
cally wrong--even if it did turn out to be clinically
correct.
TEN RULES FOR THE JOURNAL READER
Since it is reasonably clear that all clinical researchers
are about as ~ikely to start analyzing their data according
to the rigorous rules of science as it is that all advertise-
ments in clinical journals are going to do likewise in pre-
senting their wares, the typical consumer of both the ad-
vertisements and the articles might benefit from the fol-
lowing rules for the proper understanding of research data
published in contemporary clinical journals.
Rule I: Be skeptical. As noted earlier, the odds are
good that the authors have arrived at invalid conclusions.
This does not mean their conclusions are wrong, but it
does mean that their conclusions may not have the
scientific validity attributed to them by their authors.
Many 1987-1988 Journal articles were not exceptions to
this rule.
Rule II: Look for the data. Once upon a time, clinical
researchers said, "Here is a detailed description of what
we did and here is what happened and this is what we
think it all means; but since we have reported our data,
you are free to draw your own conclusions." Contempo-
rary journal authors are more likely to say, "Here is a
description of what we did, here are our statistical
analyses, and this is what all that mumbo jumbo means;
trust us." But the readers' only consult with nature is via
the data reported in the article. Everything else is a
consult with authority. That is not science.
Most 1987-1988 Journal articles did a good job of
presenting the data, but many gave the impression that
the raw data had been cooked by the only legal recipe.
Generally, there are several legitimate recipes for cook-
ing any given data set. Some recipes, however, not only
are illegitimate but they present the data in a form
incapable of reconstitution to the state required for
appropriate cooking. In many instances, both the authors
and the consumers of 1987-1988 Journal, reports would
have been better served by following the rule for raw
vegetables: ceok only if otherwise indigestible.
In any event, authors should present their data in a
form that allows consumers to apply their own favorite
recipe. Otherwise, the reader is faced with what is, in
Yancey 559
effect, a proprietary data base. This is precisely the
opposite of the scientific canon that all data must be
public. When science stops being public, it stops being
science.
Rule III: Differentiate between descliptive and inferen-
tial statistics. I believe the single most serious error made
by clinical researchers is the failure to understand the
fundamental difference between statistical indices that
summarize the data collected by the study and those that
attempt to answer questions about data that have not
been c o l l e c t e d - a n d likely never will be collected.
Statistics such as means and standard deviations
describe the data in hand; hence, they are called descrip-
tive statistics. Tools that generate p values allow one to
make inferences about the population of data from which
the present study is but one random sample; hence, they
are called inferential statistics.
That is all simple enough if one is drawing random
samples from existing populations, as in epidemiologic
research, but the vast majority of clinical studies produce
data that are not random samples from existing popula-
tions. Indeed, one often has data from patients treated as
no other patients in the world have ever been treated. So,
it is only natural that many 1987-1988 Journal authors
thought their standard errors of the mean, p val-
ues, and regression lines, to name just a few items, were
indexing something important about the data they had
just collected. They were wrong!
If one sample mean is 10 and the other 5, the
difference is precisely 5, whether the associated p value
is >0.95 or <0.0001. The question of whether that
difference is of any clinical note is a clinical judgment,
regardless of the associated p values. Authors who use p
values to support their conclusions about the magnitude
or importance of differences in group data are making
the most serious and the most common error to be found
in the clinical literature.
Most p values mean the same thing, p < 0.05, for
example, generally means that there are fewer than 5
chances in 100 of getting the statistic in question (or one
even more extreme) if the null hypothesis being tested
were precisely true in the population from which one has
a random sample. Most authors clearly do not under-
stand this simple fact. If they did, they would clearly
state: (1) What the statistic in question indexes about the
data; (2) The null hypothesis; (3) The population one is
talking about; and (4) What all of this has to do with the
basic point of the study. In a great many cases, the
statistic does not index what the authors think it indexes,
the null hypothesis is of no clinical note, the population
is not the one referenced in the author's discussion, and
the statistical analysis is irrelevant to the basic question
of the study.
One way to keep this absolutely crucial distinction in
mind is to ask oneself the following question: If the study
data were based upon all such patients in the w o r l d -
rather than a very small nonrandom sample of such
patients --would the results be of any clinical note? If the
560 Yancey
answer is no, then no statistical analysis will make the
results of the study clinically important, even if all the p
values are "highly significant." If the answer is yes, the
study is of clinical importance, even if all the p values are
"not significant."
Rule IV: Question the validity of all descriptive statis-
tics. Clinicians should be interested in individuals, but the
majority of articles in the 1987-1988 Journal used means
and standard errors of the mean to index the data. It has
been said that a fellow with one leg frozen in ice and the
other leg in boiling water is c o m f o r t a b l e - o n average.
Although it is clear that the mean temperature of the
water covering this poor fellow's lower half is a dismal
index of the degree of his discomfort, equally inappro-
priate uses of the arithmetic mean are to be found in
some of the 1987-1988 Journal articles.
An example of this type of error is found in an
October 1987 Journal article." I n Figure 5, the concen-
tration of ornithine decarboxylase in tumor sites is re-
ported as 3.5 (nmol CO2/mg protein/hour) _+3 (SEM). A
careful reader would interpret the carefully composed
bar graph as follows: With 15 subjects, a standard error
of the mean of 3 becomes a standard deviation of 11.6
(SD = SEM × x/n); however, since the concentration
cannot drop below 0, this must have been one of the most
positively skewed distributions ever recorded; and in a
severely skewed distribution, the mean is pulled in the
direction of the skew; hence, almost all of the 15 subjects
must have shown a concentration very close to 0, with 1
or 2 patients having large concentrations.
Therefore, 3.5 is a poor index of the concentration at
the tumor site; hence, it is a poor index of how the tumor
site compared with other sites. The standard error of the
mean is an even poorer index of the variability in the
data. The bar graph with its error bar does not resemble
the actual distribution of data points. After interpreting
this graph, it is clear that the difference between the
tumor site and the other sites is not in terms of the mean,
it is in terms of the number of tumor sites having
concentrations very much larger than anything observed
at the other sites. Hence, the subsequent Student's t tests
had no sensible interpretation, because the t tests were
about population means, which had no clinical meaning.
It is clear that had many of the 1987-1988 Journal
authors looked carefully at their raw data, i~t would have
become obvious that the use of the mean ± 1 standard
error of the mean gave a most misleading picture of what
h a p p e n e d - i t almost always does. When that was true,
subsequent statistical analyses seldom made any sense
because the majority of inferential statistical tools used
in the 1987-1988 Journal were about population means.
Population means, however, seldom present for treat-
ment. Even if one knew the precise value of many of the
population means being estimated by Journal authors,
that would be about as valuable to practicing physicians
as knowing the mean heart rate of all their patients
currently in the hospital.
American Journal of Orthodon~cs and Dentofacial Orthopedics
May 1996
The mere fact that so many 1987-1988 Journal au-
thors erroneously believed the mean to be a good index
of nonsymmetrical and/or highly variable data, and the
standard error of the mean to be an accurate index of
that variability, is sufficient reason for all Journal readers
to attempt to transform all data back to its raw form. The
standard error of the mean, in fact, is neither an index of
variability in the sample data nor an estimate of variabil-
ity in the population. And the larger the sample size, the
less well it indexes what should be of chief concern to all
clinicians: variability among patients.
Rule V: Question the validity of all inferential statistics.
Almost every review of the medical literature has found
that many researchers are wrong about what inferential
statistics actually index. 1-13This was clearly in evidence in
the 1987-1988 Journal articles. Many authors interpreted
"p < 0.05" to mean that an important treatment effect
had been demonstrated and "p > 0.05" to mean that no
important treatment effect existed. In truth, neither the
presence of statistical significance (p < 0.05) nor the
absence of statistical significance (p > 0.05) has any
reliable relationship either to the magnitude of the
treatment effect nor to the extent of clinical importance.
The October 1987 Journal article cited earlier to
illustrate the inappropriate use of the mean and standard
error of the mean also serves as an example of the
problem with p v a l u e s . 14 Figure 5 in that article reports a
single value of p < 0.375 for three Student!s t tests used
to compare ornithine decarboxylase concentrations at
the tumor site and three other sites. As already noted,
the t test is about population means and the mean is not
a legitimate index of tumor site concentrations. Even if
this were not the case, the Student's t test is not legiti-
mate here because of the grossly unequal sample sizes,
the grossly unequal standard deviations (which are not
presented but which can be calculated), the grossly
skewed distribution of tumor site concentrations, the fact
that most of the data are paired rather than independent,
and the fact that one mean is used in three different post
hoc comparisons. Even if one ignored all of these prob-
lems and accepted this "not significant" p value as saying
something important about these data, what is it saying?
It clearly is not saying that there is no difference
between the sample means; the tumor site mean is five
times as great as the concentration at the other three
sites. This difference in means is greater than any of the
other comparisons presented in the article, all of which
were "significant." It clearly is not saying that this dif-
ference is of no clinical note; the authors state otherwise.
It is, in fact, just saying that in the population from which
this is a random sample, we do not know which mean is
larger, p > 0.05 is a statement of ignorance, not a state-
ment of no difference.
In Figures 2 through 4 in this same article, ~4the same
statistical procedure is employed to compare mean
polyamine concentrations at the tumor site and the three
other sites. Five different p values (p < 0.05 to
American Journal of Orthodontics and Dentofacial Orthopedics
Volume 109, No. 5
p < 0.0005) are used to index nine comparisons. Does
this mean that the p < 0.0005 differences are larger or
more important than the p < 0.05 differences? No. As
previously noted, none of these differences is even close
to the magnitude of the "not significant" difference in
ornithine decarboxylase concentrations. So what do they
mean?
If the correct statistical procedures had been em-
ployed, a p < 0.05 would mean that one may be about
95% confident that precise replications of this study
would yield population means with the same directional
difference in magnitude as was observed in the sample. A
p < 0.0005 means that one may be about 99.95% confi-
dent of precisely the same thing. Neither p value gives the
slightest hint of what a clinician would need to know: (1)
What is the probable magnitude of each population
mean? (2) What is the probable magnitude of the differ-
ence between these population means? (3) What is the
probable distribution of individual values around these
means? (4) What proportion of the population has a
concentration in these two sites anywhere near that of
the sample means? (5) What is the clinical importance of
this difference? (6) Is this of the slightest value in
treating individual patients?
Still, p values have become a firmly fixed fetish in the
clinical literature, so you must learn to live with them. A
good place to start is by reading Medical Uses of Statistics,
especially Chapter 8.13'~s When statisticians talk to each
other, however, they find the use of p values by medical
researchers a subject for much raucous ridiculeJ 6
Rule VI: Be wary of correlation and regression analy-
ses. The great majority of clinical studies I have examined
over the pas! 30 years involving correlation and/or re-
gression analysis have been seriously flawed. The worst
mistake is accepting the authors' implicit suggestion that
an individual clinician may safely use the regression
equation to make predictions for individual patients.
Sometimes equations or graphs are given for 95% con-
fidence intervals, but these usually are misinterpreted to
be confidence intervals for individual predictions rather
than correctly understood to be estimates of population
means, which, more often than not, are of no use to
clinicians.
Equations for making individual predictions are pre-
sented in any worthwhile statistics book, ~7yet they almost
are never included in clinical journal articles. I was
pleasantly surprised to find a couple of instances in the
1987-1988 Journal where these correct equations were
used. It also was good to find that most authors pre-
sented scattergraphs. It is a shame that some of those
authors did not examine their own graphs well enough to
note that their significant p values constituted a poor
index of the ability to predict y from x, even in the sample
data.
For example, an article in the December 1988 Journal
used a regression equation, a correlation coefficient, a p
value, and a scattergraph where only the scattergraph
Yancey 561
should have been used. 18 In Figure 2, the authors say,
"The regression line shows that the length of the preop-
erative period was correlated with the amount of blood
required. The two variables were related with a high
degree of predictability (r = 0.66, p < 0.001). ''18 The
authors also provide the regression equation ( y -
1.1393 + 1.6376x), suggesting that in the future one
might use this equation to predict the amount of blood
required. But if one plugs 10 days into the equation, one
gets the prediction of 17.5 units of blood. In the sample
of 38 patients, the four patients with a 10-day preopera-
tive period required 6, 9, 21, and 34 units of blood,
respectively. The five patients with a 12-day delay re-
quired 2, 3, 15, 40, and 58 units, respectively, rather than
the 20.8 units predicted by the formula.
It is clear from an examination of this scattcrgraph
that there is not "a high degree of predictability," even in
the sample data. But a regression line and a regression
equation imply that one can use this equation to predict
what will happen in the future. (We already know pre-
cisely how much blood each of the sample patients
required.) The authors do not provide the data to calcu-
late precisely the 95% confidence intervals for the re-
gression, but a rough estimate yields the following limits
for a value of 12 days: 15 and 24. This means that one
may be 95% confident that in the population from which
these 38 patients are a random sample, the mean number
of units of blood required for all patients with a 12-day
preoperative period is a single value somewhere between
15 and 24.
It is quite clear from the great variability in the
sample data that even if we knew precisely the mean
value for all the 12-day patients in the world, we would
have very little notion of what the fate of any particular
patient might be. Such individual prediction limits may
be calculated. Using the data presented in the article, we
may be 95% confident that a single 12-day patient will
require somewhere between minus 6 and plus 45 units of
blood. This information is as useless as it is ridiculous.
Even if this sample size had been 38,000 (instead of only
38), a correlation coefficient of only 0.66 would not have
produced narrow individual prediction l i m i t s - b u t the p
value would have been <0.0000000H
Since this was not a prediction problem in the first
place, how did the authors come to the "high degree of
predictability-" conclusion? It is likely they did so because
of the highly significant p value. All that p value means is
that in the population from which this is a true random
sample, we may be 99.9% confident that the correlation
coefficient between days and blood is greater than abso-
lute zero. A more useless piece of clinical information is
hard to imagine. And keep in mind that if the p value
had been <0.00000001, we just would be even more
confident of precisely the same useless fact!
The sadness of it all is that the simple scattergraph
contains all the support the authors needed for their
observation that in this sample of 38 patients there was a
562 Yancey
general tendency for more blood to be required by those
patients spending the most time in the preoperative
period. The regression equation, the regression line, and
the p value stimulated the authors to claim "a high
degree of predictability" in a situation where that is
neither true nor relevant to the point being made. If they
had just presented the scattergraph, the authors might
have thought to show where the seven duplicate data
points are located, because only 31 of the 38 patients are
identifiable in their Figure 2.
Rule VII: Identify the population sampled. In the "old
days," clinicians intuitively understood that what was
true of Dr. Brown's patients in London may not be true
of Dr. Smith's patients in Biloxi, Mississippi. Today,
many researchers seem to believe that statistics can solve
that problem. They can not. In almost all instances, the
statistics used are about what would happen if the
present study were to be repeated over and over. They
say nothing about what would happen if even slight
changes were to be made in either treatments or patient
populations.
Such extrapolations surely can be made, but it is
scientific thinking and clinical judgment that justify the
leap, not the statistical analyses generally cited. Still,
most of the 1987-1988 Journal articles strongly implied
that their statistics certified the extrapolation of the study
results to patient populations much different from those
actually sampled by the study. I found few articles in
which the authors stated very clearly that their inferential
statistics simply suggested directional differences in pa-
rameters of populations that would exist only if the
present study were to be replicated precisely-warts and
all.
Rule VIII: Identify the type of study. Only truly ran-
domized tightly controlled, prospective studies provide
even an opportunity for strong cause-and-effect state-
ments. Often there are very good reasons why such a
study cannot be done, but that has no effect upon the
logic that says that in the absence of true random
assignment and rigid control of the treatment groups, any
differences in outcome might be a function of differences
in the groups other than the treatment difference. Too
few authors of nonrandomized and loosely controlled
studies in the 1987-1988 Journal articles drew sufficient
attention to the fact that no statistical manipulation can
compensate for the absence of true random assignment
of patients to treatments and strong assurance that the
only differences between the treatment groups come
from the treatment itself.
Nonrandomized, retrospective, and loosely con-
trolled studies may be suggestive of true cause-and-effect
relationships, but many replications of such studies (pref-
erably in randomly selected samples of the target popu-
lation) are required before alternative explanations of
the apparent treatment effect may reasonably be ig-
nored.
American Journal of Orthodontics and Dentofacial Orthopedics
May 1996
Rule IX: Look for indices of probable magnitude-of-
treatment effects. The great majority of the inferential
statistical tools employed by 1987-1988 Journal authors
were estimates of directional differences in population
means or population proportions. Few articles provided
estimates of the probable magnitude of these differences.
Many authors used their p values to index the magni-
tude-of-treatment effects. They were wrong, p values are
affected by the magnitude-of-treatment effects in the
sample data, but they are affected much more by sample
size, variability, and the particular recipe used to cook the
data, none of which has any relationship to either the
magnitude or clinical importance of the treatment ef-
fects.
The use of confidence intervals for population pa-
rameters and prediction limits for individual observations
actually would have changed fundamentally many of the
conclusions that were based upon p values. The regres-
sion example cited earlier is but one of many cases. 18 An
example of where confidence limits do not reverse the
conclusions reached via p values, but still add a great
deal to the analysis, may be found in a review I wrote to
accompany a recent research report in the Journal.19'2°
Most statistics texts devote almost all of their atten-
tion to hypothesis testing, but even those that give equal
treatment to parameter estimation 17make the distinction
more difficult to understand than is the case. The simple
facts are these: (1) The larger the sample size, the less
the variability and the more powerful the statistical tool,
the more likely one is to find trivial differences to be
"highly significant," and the more precisely confidence
intervals and prediction limits will tell one what one
needs to know; (2) The smaller the sample size, the
greater the variability and the less powerful the statistical
tool, the more likely one is to find profoundly important
differences to be "not significant," and the more clearly
confidence intervals and prediction limits will tell one
that one does not yet know what one needs to know.
Rule X: Draw your own conclusions. The primary
difference between science and other methods of finding
the truth is that science consults nature while other
methods consult authority. Your only consult with nature
in a research article is the section that presents the raw
data. Everything else is some authority telling you what
the truth of the matter is.
If you had good reason to believe that almost all
journal authors already had applied correctly the rules of
scientific data analysis to the process of drawing valid
scientific conclusions from their data, you could safely
skip to the conclusions section of each article. Indeed,
there would be little need for the Journal to publish any
material other than the authors' conclusions. That is not
the case. Sadly, in many cases, you will need to undo
much of the data analysis that has been done before you
can draw your own conclusions; but that, more often than
not, will be worth the effort.
American Journal of Orthodontics and Dentofacial Orthopedics
Volume t09, No. 5
If you are unable to reconstruct even a rough esti-
mate of the distribution of original data points, ask
yourself if the report is still of any use to clinicians who
remain responsible for treating patients one by one.
AN APOLOGY TO JOURNAL AUTHORS
I hereby sincerely apologize for having so harshly
criticized the majority of 1987-1988 Journal authors,
whose only error was performing perfectly usual and
customary analyses of their research data and then com-
ing to perfectly usual and customary conclusions. I am
even more apologetic for the role those of us who
provide biostatistical services to the biomedical commu-
nity have played in creating the sad state of affairs
discussed in this essay. We provided quick answers to
your quick questions just because you didn't want a
lecture on statistics. We served you as technicians just
because that is all you wanted. Worst of all, we took no
responsibility for your gross misunderstanding of our
tools. Of course, we knew we would never be called to
account because you never replicate your studies often
enough to see whether our predictions are correct or not.
Indeed, most of you never understood that our magic p
values are juat trivial predictions, rather than the rabbi's
kosher stamp we let you believe them to be.
Those of us who have spent a large part of our lives
working in industrial and military settings are more to
blame than anyone. We know that when one is serious
about predicting what will happen if a particular proce-
dure is repeated over and o v e r - b e c a u s e it will be
repeated ow~r and o v e r - o n e never uses p values or
regression equations or standard errors of the mean. We
know quite well that one uses confidence intervals and
prediction limits and estimates of probable magnitude of
effect. We also know that the statistical tools most
commonly used in the biomedical literature seldom do
nearly as good a job of predicting highly variable out-
comes as do more sophisticated integrations of math-
ematics, data, judgment, logic, experience, and guts.
You wanted simple magic, and you got simple magic.
But this thing has gotten out of hand! It is time for those
of us in my business to admit that there are no quick
answers to ycur quick questions and that good statistical
consultations require a great deal m o r e time than good
medical consultations, because the former involve far
more abstract concepts and far fewer pre-programmed
skills than the latter.
It is time for you to admit that you do not understand
the indices we have given you. For those of you who do
think you understand these tools, I suggest the following
exercise. Take your last half-dozen research reports to
your favorite statistical guru. Ask him/her to do a com-
puter simulation of 10,000 replications of each study.
Look at the results. Ask yourself if that is what the
conclusions section of each article predicted. If the
Yancey 563
answer is yes, you should be helping your less fortunate
colleagues come to the same state of understanding. If
the answer is no, ask your guru why he or she never
explained all of this to you in the first place.
THE REAL SOLUTION
Someday, the consumers of the clinical literature will
be justified in reading the conclusions of a research
article with the same confidence they now read individual
patient reports written by respected colleagues. That day
will arrive, however, only after at least three events occur:
(1) widespread recognition of the prevalence and severity
of data analysis errors in the contemporary literature; (2)
widespread acceptance of more sensible data analysis
procedures; and (3) pronouncements by editors of the
best journals that manuscripts that claim a level of
scientific validity not supported by the design employed
and data analyses performed will not be published.
Already, there is massive documentation of the prob-
lem. The references cited in this paper constitute only an
introduction to this dismal literature. Already, there are
very sensible alternatives to the inappropriate statistical
tools most often misused in the clinical literature. But
history suggests that neither science nor medicine changes
usual and customary practice just because a terrible prob-
lem exists and a dearly superior approach is available.21'22
Perhaps the Uniform Requirements for Manuscripts
Submitted to Biomedical Journals 23 will someday be as
concerned with the scientific validity of the authors'
conclusions as it is with the size of the margins and
format of the reference section. Perhaps the Guidelines
for Statistical Reporting in Articles for Medical Journals 24
will someday be as strictly enforced as the rules for the
appearance of the manuscript.
Perhaps the journals that have been courageous
enough to commission audits of their own methodologic
quality will be the first to change the real standards for
usual and customary data analysis. Perhaps they will
include in their "Information for Authors" section the
absolute requirement that all statistical analyses of re-
search data must contribute t o - r a t h e r than substitute
f o r - r i g o r o u s scientific thinking and independent clinical
judgment.
SUMMARY
This was not a scientific assessment of the scientific
quality of the papers published by The American Journal
of Surgery. It was an informal audit of the adequacy of the
data analysis in the clinical research reports appearing in
the 1987-1988 issues.
As one who has devoted more than three decades to
helping a great variety of people make sense of scientific
data, I found the overall quality of data analysis in these
papers to be above average for the medical literature; and
yet, I found many instances of errors so serious as to
564 Yancey
render invalid the conclusions of the authors. My 10 pro-
posed rules for reading clinical research reports constitute
only an interim solution to a very worrisome problem. The
real solution must c o m e from the producers of and the
gatekeepers for the medical literature.
Interested readers should examine "An exploratory study
of statistical assessment of papers published in the British
Medical Journal," especially Tables I and 2. ( G a r d n e r MJ,
B o n d J. J A M A 1990;263:1355-7.)
REFERENCES
1. Murray GD. The task of a statistical referee. Br J Surg
1988;75:664-7.
2. Bailar JC. Science, statistics, and deception. Ann Intern
Med 1986;104:259-60.
3. DerSimonian R, Charette LJ, McPeek B, Mosteller F.
Reporting on methods in clinical trials. In: Bailar JC,
Mosteller F, eds. Medical uses of statistics. Waltham, MA:
New England Journal of Medicine Books, 1986:272-88.
4. Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The
importance of beta, the type II error, and sample size in the
design and interpretation of the randomized controlled
trial: survey of 71 "negative" trials. In: Bailar JC, Mosteller
F, eds. Medical uses of statistics. Waltham, MA: New
England Journal of Medicine Books, 1986:289-304.
5. Goldschmidt PG, Colton T. The quality of medical litera-
ture: an analysis of validation assessments. In: Bailar JC,
Mosteller F, eds. Medical uses of statistics. Waltham, MA:
New England Journal of Medicine Books, 1986:370-91.
6. Gore SM, Jones IG, Rytter EC. Misuse of statistical meth-
ods: critical assessment of articles in BMJ from January to
March 1976. Br Med J 1977;1:85-7.
7. Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the
reporting of clinical trials: a survey of three medical jour-
nals. N Engl J Med 1987;317:426-32.
8. Reed JF, Slaichert W. Statistical proof in inconclusive
"negative" trials. Arch Intern Med 1981;141:1307-10.
AmericanJournalof Orthodonticsand DentofacialOrthopedics
May 1996
9. Schor S. Statistical proof in inconclusive "negative" trials:
an editorial. Arch Intern Med 1981;141:1263-4.
10. Schor S, Karten I. Statistical evaluation of medical journal
manuscripts. JAMA 1966;195:1123-8.
11. Sheehan TJ. The medical literature: let the reader beware.
Arch Intern Med 1980;140:472-4.
12. Thorn MD, Pullman CC, Symons MJ, Eckel FM. Statistical
and research quality of the medical and pharmacy literature.
Am J Hosp Pharm 1985;42:1077-82.
13. Bailar JC, Mosteller F, eds. Medical uses of statistics.
Waltham, MA: New England Journal of Medicine Books,
1986.
14. Dimery IW, Nishioka K, Grossie B Jr, et al. Polyamine
metabolism in carcinoma of the oral cavity compared with
adjacent and normal oral mucosa. Am J Surg 1987;154:429-
33.
15. Ware JH, Mosteller F, Ingelfinger JA. P values. In: Bailar
JC, Mosteller F, eds. Medical uses of statistics. Waltham,
MA: New England Journal of Medicine Books, 1986:149-69.
16. Salsburg DS. The religion of statistics as practiced in medi-
cal journals. Am Statistician 1985;39:220-3.
17. Zar JH. Biostatistical analysis, 2nd ed. Englewood Cliffs,
NJ: Prentice-Hall, 1984.
18. Grossman MD, McGreevy JM. Effect of delayed operation
for bleeding esophageal varices on Child's class and indices
of liver function. Am J Surg 1988;156:502-5.
19. Yancey JM. Editorial comment. Am J Surg 1989;158:433-4.
20. Garcia-Rodriguez JA, Puig-Lacalle J, Arnau C, Porta M,
Vallv6 C. Antibiotic prophylaxis with cefotaxime in gas-
troduodenal and biliary surgery. Am J Surg 1989;158:428-
32.
21. Kupfersmid J. Improving what is published: a model in
search of an editor. Am Psychologist 1988;43:635-42.
22. Kuhn TS. The structure of scientific revolutions, 2nd ed.
Chicago: University of Chicago Press, 1970.
23. International Committee of Medical Journal Editors. Uni-
form requirements for manuscripts submitted to biomedical
journals. Ann Intern Med 1988;108:258-65.
24. Bailar JC, Mosteller F. Guidelines for statistical reporting in
articles for medical journals. Ann Intern Med 1988;108:266-
73.