research in medical education
Quality of reporting of experimental studies in
medical education: a systematic review
David A Cook,1 Thomas J Beckman1 & Georges Bordage2
OBJECTIVE Determine the prevalence of essential
elements of reporting in experimental studies in
medical education.
DESIGN Systematic review.
DATA SOURCES Articles published in 2003 and 2004
in Academic Medicine, Advances in Health Sciences
Education, American Journal of Surgery, Journal of
General Internal Medicine, Medical Education, and
Teaching and Learning in Medicine.
REVIEW METHODS Articles describing education
experiments, including evaluation studies with
experimental designs, were identified (n = 185) by
reviewing titles and abstracts. A random sample
(n = 110) was selected for full review. The full text
of each article was evaluated for the presence of
guideline-based features of quality reporting: a
critical literature review, conceptual framework,
statement of study intent (e.g. aim, research
question, or hypothesis), statement of study design,
definition of main intervention and comparison
intervention or control group, and consideration of
human subject rights.
RESULTS Of the 105 articles suitable for review, 47
(45%) contained a critical literature review and 58
(55%) presented a conceptual framework. A state-
ment of study intent was present in 80 articles (76%),
among which the independent and dependent
variables were operationally defined in 38 (47%) and
1
Division of General Internal Medicine, Mayo Clinic College of
Medicine, Rochester, Minnesota, USA
2
Department of Medical Education, University of Illinois at Chicago,
Chicago, Illinois, USA
Correspondence: David A Cook MD, MHPE, Division of General Internal
Medicine, Mayo Clinic College of Medicine, Baldwin 4-A, 200 First
Street SW, Rochester, Minnesota 55905, USA. Tel: 00 1 507 266 4156;
Fax: 00 1 507 284 5370; E-mail: cook.david33@mayo.edu
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
26 articles (32%), respectively. A total of 17 articles
(16%) contained an explicit study design statement.
Among the 48 studies with a comparison group, 35
(73%) clearly defined the comparison intervention
or control group. Institutional review board approval
or participant consent was reported in 44 articles
(42%).
CONCLUSIONS The quality of reporting of experi-
mental studies in medical education was generally
poor. Criteria are proposed as a starting point for
establishing reporting standards for medical
education research.
KEYWORDS review [publication type]; *education,
medical; research design ⁄ *standards; professional
competence ⁄ *standards; periodicals ⁄ *standards;
guidelines; data collection ⁄ standards.
Medical Education 2007: 41: 737–745
doi:10.1111/j.1365-2923.2007.02777.x
INTRODUCTION
Medical education is a rapidly growing field of
study,1,2 as evidenced by the presence of 6 English-
language journals dedicated to reporting research
and issues in medical education, and the increasing
number of education reports published in clinical
journals.3–5 However, authors have called for greater
use of theoretical frameworks,6–8 rigorous and
creative study designs,9–11 and meaningful out-
comes12–14 in medical education research, which
together suggest the need for higher quality
research.15 Furthermore, deficiencies in reporting
quality have been identified in medical education,16–18
as in other disciplines.19–23 Although reporting
quality and methodological rigour are not synony-
mous,24,25 they are inter-related26 in that high
reporting quality is a prerequisite to understanding a
737
738 research in medical education
Overview
What is already known on this subject
Little is known about the quality of the
reporting of research studies in medical edu-
cation.
What this study adds
Many essential elements of scientific reporting
are frequently absent from articles describing
medical education experiments, including a
critical literature review, conceptual
framework, study design statement, definition
of the comparison or control group, and
acknowledgement of human subject rights.
Journal requirements (such as statements
regarding ethical approval) can significantly
improve reporting quality.
Suggestions for further research
This study might be replicated for other types
of medical education research (observational
studies, validity studies or qualitative studies)
and other elements of reporting.
Formal reporting standards might be
developed and evaluated for medical
education research.
study’s intent and methods.27 Guidelines have been
published in clinical medicine for reporting
randomised clinical trials,19 studies of diagnostic
accuracy,28 meta-analyses29,30 and non-randomised
trials.27 The goal of such guidelines is to propose
reporting criteria that improve the ethical and
scientific validity of the reports. The road to creating
such reporting criteria is long and arduous and
involves identifying a reporting problem, selecting
elements of reporting that are shown to affect the
valid interpretation of the reports, and developing
standards by the consensus of an international panel.
We found only 3 studies evaluating the quality of
reporting of medical education research.17,18,31 A
review of studies evaluating cultural competence
training18 found that important elements (namely,
setting, participant demographics, intervention,
appropriate comparison group, and reasons for non-
inclusion of data) were reported less than half the time.
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
Another study investigated the reporting of ethical
safeguards, finding that only 24% of papers reported
informed consent, and fewer than 5% reported review
by an institutional review board.17 Finally, an analysis of
27 papers presented at the annual Research in Medical
Education Conference of the Association of American
Medical Colleges31 concluded that although the
research question was explicitly stated in 96% of the
studies, there was still considerable room for improve-
ment in reporting quality. Yet these studies present an
incomplete and fragmented view of the quality of
reporting in medical education.
The purpose of the present study was to describe
more systematically the problem of reporting in
medical education research, as the first step to
improving reporting quality. Because reporting
criteria are highly specific to the type of study
reported, we focused on a single broad category of
studies: experimental studies. Experiments were
chosen because they represent the gold standard for
evaluating the outcomes of instructional interven-
tions in education, and reporting should be relatively
straightforward as the rules for experiments are well
codified in science in general. However, experiments
in education vary widely in their execution, ranging
from single-group, post-test studies to static-group
comparisons and randomised, controlled trials. Given
this heterogeneity, our study focused on how well the
intent and nature of each experiment was described,
by contrast with, for example, the Consolidated
Standards of Reporting Trials (CONSORT) state-
ment,19 which targets randomised trials and includes
specific elements of methods, results and discussion.
In addition, because of recent concerns about the
human rights of study subjects,17,32–34 we evaluated
how well human subject rights were acknowledged.
Thus, our intent was to systematically document the
quality of reporting the intent and nature of experi-
mental studies, with the realisation that reporting
may reflect the way research is conducted. The results
from this study can then be used by the medical
education community as groundwork for the next
step, namely, selecting reporting criteria that affect
ethical and scientific validity.
METHODS
Eligibility criteria
We selected a representative sample of recently
published reports of experimental studies in medical
education. We defined ÔexperimentÕ according to

Fraenkel and Wallen,35 as a study in which research-
ers manipulate a variable (also known as the
treatment, intervention or independent variable) to
assess its impact on other (dependent) variables. We
included all studies that met this definition.
Therefore, our sample was not limited to the classic
randomised trial, but embraced the entire spectrum
of weak and strong experimental designs, including
single-group, post-test only studies, static group
comparisons, and non-randomised and randomised
trials. Evaluation studies with experimental designs
were included. Abstract-length reports were
excluded.
Given the difficulties and limitations associated with
identifying medical education research using existing
databases,36,37 we hand-searched journals known for
publishing medical education experiments. We
selected the 4 journals with the highest impact
factors among journals focusing exclusively on
medical education research. After discussion with
experienced American medical educators in internal
medicine and surgery, we also selected 2 specialty
journals that frequently publish education research.
To focus on the current state of research, we limited
our review to the 2 full calendar years immediately
preceding the start of data collection. Consequently,
all articles published in 2003 and 2004 in the journals
Academic Medicine, Advances in Health Sciences Educa-
tion, American Journal of Surgery, Journal of General
Internal Medicine, Medical Education and Teaching and
Learning in Medicine were eligible for inclusion.
Study selection
The titles and abstracts for all eligible articles
(n ¼ 1459) were reviewed by the first author (DAC)
and all articles describing education experiments
were identified (n ¼ 185). Of these, we selected a
random sample of 110 articles for full review, with the
goal of including at least 100 articles in the final
analysis. This was judged an adequate sample size,
given previous studies using similar or smaller
samples.18,25,31,38–40 Sampling was stratified by jour-
nal and weighted by the number of education
experiments reported in each journal. Full text
review by pairs of authors revealed that 5 studies were
not experiments, leaving 105 articles for final analysis
(Fig. 1). A complete list of eligible and selected
articles is available as online supplementary material.
Quality criteria
Based on studies of journal reviewers’ comments and
guidelines for scientific writing in medical education,
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
739
1459 abstracts screened
Academic Medicine (210)
Advances in Health Sciences Education (33)
American Journal of Surgery (584)
Journal of General Internal Medicine (297)
Medical Education (258)
Teaching and Learning in Medicine (77)
1274 articles excluded as not
describing experimental
research or not involving
medical education
185 articles selected for further consideration
Academic Medicine (42)
Advances in Health Sciences Education (8)
American Journal of Surgery (13)
Journal of General Internal Medicine (26)
Medical Education (65)
Teaching and Learning in Medicine (31)
75 articles not in
random sample
110 articles randomly
selected for review
5 articles excluded as not describing
experimental research
Academic Medicine (2)
American Journal of Surgery (1)
Medical Education (2)
105 articles included in final analysis
Academic Medicine (23)
Advances in Health Sciences Education (5)
American Journal of Surgery (7)
Journal of General Internal Medicine (15)
Medical Education (37)
Teaching and Learning in Medicine (18)
Figure 1 Flow diagram of article selection
characteristics of good reporting for medical
education research were identified.15,16,36,41–43 For
example, Bordage16 showed that leading reasons for
the rejection of manuscripts by external reviewers in
medical education included incomplete research
questions, incomplete or outdated literature reviews,
lack of conceptual frameworks, and inadequately
described interventions. Morrison et al.41 and Reed
et al.36 each noted the research question, literature
review, study design, clear descriptions of interven-
tions and educational context, and appropriate
outcomes as key appraisal elements. We thus identi-
fied 5 elements essential to reporting the intent and
nature of education experiments: critical review of
the literature;7,44 conceptual or theoretical frame-
work;7,45 statement of study intent (e.g. aim, purpose,
research question or hypothesis);45–47 explicit
statement of study design,48 and explicit definition of
all study interventions.27 For reasons noted in the
Introduction, we added the acknowledgement of
human subject rights.17,32–34 These 6 elements
740 research in medical education

Table 1 Definitions of key elements for reporting experimental studies

1 Review of literature
The literature review cites articles relevant to the topic or study design and critically discusses these articles at the beginning of the paper44
ÔCritical discussionÕ requires thoughtful assessment of the quality of the cited literature and ⁄ or reflection on how the present study builds on
limitations or gaps in prior work
Articles citing literature without a critical discussion were counted separately
2 Conceptual framework
The conceptual framework situates the research question, intervention methods or study design within a model or theoretical framework that
facilitates meaningful interpretation of the methods and results7,45
3 Statement of study intent
Study intent can be phrased as a research question, a research hypothesis or a purpose (goal, aim)
The statement of study intent explicitly identifies the independent variable (intervention), the dependent variable (outcome), the relationship
between these variables, and the population of interest43
These elements were further rated as follows:
• independent variable: operationally defined (described in sufficient detail as to permit replication), vague or absent
• dependent variable: operationally defined, vague or absent
• relationship between variables: suggests the intent to determine a difference (e.g. ÔimproveÕ, ÔenhanceÕ, Ôbetter thanÕ, Ôsimilar toÕ)
• population of interest: includes the type of learner (e.g. medical student or internal medicine resident) and represents a population
(e.g. Ômedical studentsÕ) rather than a sample (e.g. Ôour medical studentsÕ)
4a Statement of study design
An explicit and accurate statement identifies the nature of the study design
4b Type of study design
Frameworks for experimental study design described by Campbell and Stanley48 and Fraenkel and Wallen35 were used
For each of the following, post-test only studies (assessment carried out only after the intervention) and pretest ⁄ post-test studies (assessment
carried out both before and after the intervention) were distinguished:
• randomised study: study groups were assigned randomly
• quasi-experimental non-randomised study: study groups were assigned using a non-randomised method
• matched-group study: study groups were systematically assigned by matching participant characteristics
• static-group comparison study: groups were pre-existing, formed for non-study purposes
• single-group study: a single study group
4c Additional design features
Factorial designs are those in which more than 1 independent variable is included in the study design, with combinations of the levels
of the independent variables represented in the study interventions
Crossover designs are those in which participants are systematically exposed to more than 1 intervention over the course of the study
5a Study intervention definition
The study intervention is described in sufficient detail as to permit replication27
Interventions were rated as explicit, vague, or not defined
5b Comparison intervention or control group definition
The comparison intervention or control group is described in sufficient detail as to permit replication27
Interventions were rated as explicit, vague, or not defined
This element was further classified as:
• no intervention (no educational exposure; a true control group)
• placebo intervention (exposure to an educational intervention dealing with a topic different from the study intervention)
• comparison intervention (exposure to an educational intervention different than, but on the same topic as, the study intervention), or
• not defined
6 Acknowledgements of human subject rights
Acknowledgement of human subject rights, defined as either informed consent or approval by an institutional review board, is documented33

constituted the primary outcomes for this study
(Table 1 shows definitions). We elected not to rate
outcomes because this issue in medical education has
already been addressed.12 The statement of study
intent was further evaluated for the presence of 4
basic elements:43 the independent variable; the
dependent variable; the relationship between the
variables, and the population of interest.
Data extraction
A data collection form was created to rate these
elements of reporting. The form was pilot-tested and
refined using a sample of articles published in 2002.
Each full article included in the study sample was
rated independently by pairs of authors. Disagree-
ments were resolved by discussion and consensus was
reached on final ratings.
Data analysis
Results were summarised using descriptive statistics.
Inter-rater agreement was determined for each set
of ratings using intraclass correlation coefficients
(ICC).49 Chi-square test or Fisher’s exact test was
used for subgroup comparisons. Analyses were
performed with SAS 9.1 (SAS Institute, Cary, NC,
USA).

 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745


RESULTS
A total of 105 articles were reviewed in full. Intraclass
correlation coefficients were statistically significant
for each set of ratings and ranged from 0.344 to
0.775, suggesting inter-rater agreement that was at
least fair and usually good to excellent.49
Prevalence of key elements
Almost half the articles (47 [45%]) presented a
critical review of the literature and another 52
(49%) cited literature without discussion. Slightly
over half (58 [55%]) reported a conceptual
framework. A total of 17 articles (16%) gave an
explicit statement of study design. Of the 48 studies
(46%) conducted with a comparison group, the
definition of the control group or comparison
intervention was explicit in 35 (73%). One study
reported statistical comparisons, but we were
unable to determine whether these reflected pre--
test ⁄ post-test comparisons or a separate comparison
group. Fewer than half the studies (44 [42%])
reported IRB approval or participant consent. See
online supplementary Table S1 for details.
The single-group pre-test ⁄ post-test design was the
most common study design (34 [32%]), followed
by the single-group post-test only design (27
[26%]) and the static-group comparison post-test
only design (19 [18%]). A total of 18 studies (17%)
reported randomisation. Factorial designs and
crossover designs were rare (3 [3%] and 4 [4%] of
studies, respectively). We looked at differences in
reporting quality for different study designs.
Reports of studies with stronger designs (random-
ised, quasi-experimental and matched-group
controlled trials), compared with weaker designs
(uncontrolled or static-group controlled trials),
more frequently contained critical literature reviews
(22 ⁄ 28 [79%] versus 25 ⁄ 76 [33%]; P ¼ 0.0002) and
study design statements (11 ⁄ 28 [39%] versus 6 ⁄ 76
[8%]; P ¼ 0.0001). The difference in reporting
acknowledgement of human subjects rights was not
statistically significant, albeit close (16 ⁄ 28 [57%]
versus 28 ⁄ 76 [37%]; P ¼ 0.06). Other differences
were not significant (P > 0.13).
The statement of study intent was further rated for
the presence of 4 essential elements: the adequacy of
the independent and dependent variables, respect-
ively; the relationship between the variables, and the
intended population (See online supplementary
Table S2 for details). Of the 80 articles (76%) with a
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
741
statement of study intent, the independent and
dependent variables were operationally defined in 38
(47%) and 26 (32%), respectively. All 4 elements
were fully present in 2 articles (3%) and were either
present or vaguely defined in another 34 (43%).
Several articles presented objectives for the paper
(ÔThe aim of this paper is to raise the profile of
patient safety in undergraduate curriculaÕ50) or the
study intervention (ÔCourse objectives were to …Õ),
but failed to report the intent of the study itself.
Secondary analyses
There were no significant differences across journals
among the elements rated (P > 0.08), except for
presence of a conceptual framework (P ¼ 0.015).
Conceptual frameworks were present more
frequently in Advances in Health Sciences Education
(4 ⁄ 5 articles [80%]), Medical Education (26 ⁄ 37 articles
[70%]), and Academic Medicine (15 ⁄ 23 articles [65%])
than in Journal of General Internal Medicine (5 ⁄ 15
articles [33%]), Teaching and Learning in Medicine
(6 ⁄ 18 articles [33%]), and American Journal of Surgery
(2 ⁄ 7 articles [29%]).
In 2004, Medical Education instituted a policy manda-
ting authors to report ethical approval for their study.
Of the 23 articles published before this policy went
into effect, 7 (30%) contained a statement about
human subject rights, compared with all 14 articles
(100%) published after the policy took effect
(P < 0.0001).
DISCUSSION
This systematic review of reports of experimental
studies in medical education revealed that several
essential elements of scientific reporting were
frequently missing. Although the study intent and
intervention definition were reported often, a
statement of study design was rare. The other main
elements of scientific reporting hovered at around
50%. Overall, the quality of reporting was poor,
leaving room for much improvement.
Almost all articles (94%) cited at least some litera-
ture, but fewer than half (45%) critically assessed the
literature. Many reports cited several articles in
sequence without critique, and others cited only
textbooks and monographs rather than primary
research articles. Such so-called literature reviews
generally failed to identify the current knowledge
gaps for research on the topic, and also failed to
742 research in medical education
indicate how the present study would build upon
prior work.
Barely half the studies (55%) presented a conceptual
framework. The consequence of this frequent
omission cannot be overemphasised. Conceptual
frameworks serve several purposes.45,51 Firstly, they
guide the selection of variables to be included in the
study. Secondly, they permit results to be interpreted
meaningfully and applied to other settings. Finally,
they facilitate the refinement of existing theories or
the development of new theories, and provide a basis
upon which to design and conduct further
research.7,27,52 We found several studies that were
otherwise well executed, but the absence of a formal
conceptual framework to guide interpretation
severely limited the meaning of the results.
Consistent with prior research,31 most studies (76%)
presented a statement of study intent (purpose,
research question or hypothesis). However, the great
majority of these statements were incomplete. A clear
statement of study intent sets the stage for the rest of
the study.43,53 Subsequent study design, sampling and
outcome measures all become clear once the inde-
pendent and dependent variables, the relationship
between these variables, and the intended population
have been defined explicitly.
Only a small minority of reports (16%) contained
an explicit statement of study design. Although
most reports simply omitted the design statement,
several authors used general, incorrect or ambigu-
ous terminology, employing terms such as Ônatur-
alistic studyÕ or Ôexperimental studyÕ without
further qualification, or saying that they had
conducted a Ôquasi-experimental studyÕ or Ôcohort
studyÕ when they apparently meant a single-group
pretest ⁄ post-test study. Even when accurate, such
general terms fail to convey important details
about the timing of assessments (e.g. Ôpretest ⁄ post-
testÕ) and the method of group assignments
(e.g. Ôexisting groupsÕ or ÔrandomisedÕ). Such
distinctions are essential because explicit state-
ments of study design facilitate the reader’s
understanding of study procedures.27 We also
noted the overwhelming prevalence of non-
randomised study designs, and more than half the
studies had no control or comparison group.
Whereas a previous review found a definition of the
study intervention in only 33% of articles,18 most of
the reports we reviewed clearly defined this element,
and all described the intervention at least in part.
However, over a quarter of papers reporting
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
controlled studies failed to clearly define the
comparison or control group, including 4 articles
that contained no definition at all. Without a clear
idea of what happened to the comparison group, it is
impossible to make any meaningful interpretation
regarding the study intervention.27
Consistent with previous research,17 fewer than half
of the articles we reviewed acknowledged human
subject rights. This is lower than the prevalence
observed in recent non-education studies20,54 and
may reflect researchers’ ignorance regarding the
need to consider learners as research subjects or
variations in standards across cultures and nations.
However, recent discussions emphasise that ethical
issues can no longer be ignored.32–34,55
Although inter-rater agreement was generally good,
disagreements were not uncommon. We found that
some elements, such as conceptual framework and
critical literature review, were difficult to operation-
alise and that our understanding of these concepts
matured during the pilot phase and early in the
rating of the articles. However, for these and all other
elements, disagreements most often arose when a
rater failed to locate an element that, once pointed
out by the other rater, became obvious. For example,
acknowledgement of human subject rights was
readily understood, but statements buried in the
middle of a long paragraph were easy to miss. If we
had difficulty in identifying essential elements of
research reporting despite careful scrutiny, then
casual readers will be even less likely to discern them.
Our ratings emphasised the quality of reporting
rather than the quality of study methods or the
presence or absence of errors. Our aim was to instil a
more scholarly approach to medical education
research, focusing on the fundamentals of clear goals
(statement of study intent), adequate preparation
(literature review and conceptual framework),
appropriate methods (study design), and effective
communication (reporting as a whole).56 Although
the link between reporting quality and methodolo-
gical rigour is not direct, deficiencies in reporting
may reflect a lack of awareness regarding rules for
conducting scientifically valid research, as well as the
absence of reporting standards. We call for an
international assembly of editors and authors to
define reporting standards that can affect the scien-
tific and ethical validity of medical education reports.
The criteria in Table 1 may serve as a starting point.
However, guidelines alone will probably be
insufficient. Recent research suggests that journal

instructions to authors provide little guidance in
methodology57 and infrequently cite relevant
guidelines.58 Even when instructions recommend
guidelines such as CONSORT, adherence to
the guidelines remains sub-optimal.22,23 A more
proactive approach is needed. Evidence suggests
that specific directions or checklists for authors may
have limited impact,59,60 perhaps because they are
not sufficiently enforced. However, the significant
improvement we observed when Medical Education
began to require and enforce the reporting of
ethical approval suggests that journal-initiated
interventions can impact reporting quality and
perhaps influence study procedures. Tutoring by
editors may also help,61 but studies linking this
intervention to reporting quality are still
forthcoming. At present, evidence suggests that the
peer review and editing processes have the best
chances of improving reporting quality.38,39,62–64
In this study we excluded a vast body of non-
experimental research in medical education. We do
not imply that all research should be experimental,
but do believe that all research should be rigorous
in its planning, execution and reporting. Many of
the fundamentals of research reporting that we
identified will translate to other research types,
designs and methods in medical education. We
encourage researchers to replicate this study with
other types of medical education research, such as
observational studies, validity studies or qualitative
studies. In addition, there is a need to address
other essential elements of reporting that are likely
to affect valid interpretation, such as appropriate
study design and procedures, appropriate statistical
tests, appropriate interpretation of study findings,
and consistency in reporting results (e.g. consis-
tency between text and abstract, or text and tables).
Research on methodological details will be
facilitated by establishing a consensus on reporting
standards.
In addition to limitations noted above, we acknow-
ledge that much medical education research is
published in journals not included in our sample,
and note that only 1 reviewer screened abstracts for
inclusion as experiments.
This review highlighted several major deficiencies in
the reporting of experimental studies in medical
education. The quality of reporting was generally
poor. Although our review did not directly assess the
quality of the research conducted, we are concerned
that poor reporting may reflect sub-optimal research
designs and methods, and a lack of attention to
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
743
human subject rights. Reporting guidelines are
needed (for which the criteria in Table 1 may serve as
a starting point for discussion). Greater attention to
these matters by authors, mentors, reviewers and
editors will enable researchers to further advance the
art and science of medical education.
Contributors: all authors were involved in the planning
and execution of this study and in the drafting and revising
of this manuscript.
Acknowledgements: none.
Funding: none.
Conflicts of interest: GB chairs the editorial board of
Medical Education. We are not aware of any other conflicts of
interest.
Ethical approval: not required.
REFERENCES
1 Norman GR. Research in medical education: three
decades of progress. BMJ 2002;324:1560–2.
2 Wass V, Richards T, Cantillon P. Monitoring the
medical education revolution. BMJ 2003;327:1362.
3 Petersen S. Time for evidence-based medical
education. BMJ 1999;318:1223–4.
4 Golub RM. Medical education 2005: from allegory to
bull moose. JAMA 2005;294:1108–10.
5 Branch WT Jr, Kern DE. An emerging renaissance in
medical education. J Gen Intern Med 2004;19:606–9.
6 Bligh J, Parsell G. Research in medical education:
finding its place. Med Educ 1999;33:162–3.
7 Prideaux D, Bligh J. Research in medical education:
asking the right questions. Med Educ 2002;36:1114–5.
8 Cook DA. The research we still are not doing: an
agenda for the study of computer-based learning. Acad
Med 2005;80:541–8.
9 Hutchinson L. Evaluating and researching the
effectiveness of educational interventions. BMJ
1999;318:1267–9.
10 Norman G. RCT ¼ results confounded and trivial: the
perils of grand educational experiments. Med Educ
2003;37:582–4.
11 Carney PA, Nierenberg DW, Pipas CF, Brooks WB,
Stukel TA, Keller AM. Educational epidemiology:
applying population-based design and analytic
approaches to study medical education. JAMA
2004;292:1044–50.
12 Prystowsky JB, Bordage G. An outcomes research
perspective on medical education: the predominance
of trainee assessment and satisfaction. Med Educ
2001;35:331–6.
13 Lurie SJ. Raising the passing grade for studies of
medical education. JAMA 2003;290:1210–2.
14 Chen FM, Bauchner H, Burstin H. A call for outcomes
research in medical education. Acad Med 2004;79:
955–60.
744 research in medical education
15 Education Group for Guidelines on Evaluation.
Guidelines for evaluating papers on educational
interventions. BMJ 1999;318:1265–7.
16 Bordage G. Reasons reviewers reject and accept
manuscripts: the strengths and weaknesses in medical
education reports. Acad Med 2001;76:889–96.
17 Roberts LW, Geppert C, Connor R, Nguyen K, Warner
TD. An invitation for medical educators to focus on
ethical and policy issues in research and scholarly
practice. Acad Med 2001;76:876–85.
18 Price EG, Beach MC, Gary TL et al. A systematic review
of the methodological rigor of studies evaluating
cultural competence training of health professionals.
Acad Med 2005;80:578–86.
19 Moher D, Schulz KF, Altman D, for the Consort Group.
The CONSORT statement: revised recommendations
for improving the quality of reports of parallel-group
randomised trials. JAMA 2001;285:1987–91.
20 Yank V, Rennie D. Reporting of informed consent and
ethics committee approval in clinical trials. JAMA
2002;287:2835–8.
21 Altman DG. Poor-quality medical research: what can
journals do? JAMA 2002;287:2765–7.
22 Mills EJ, Wu P, Gagnier J, Devereaux PJ. The quality of
randomised trial reporting in leading medical journals
since the revised CONSORT statement. Contemp Clin
Trials 2005;26:480–7.
23 Plint AC, Moher D, Schulz K, Altman DG, Morrison A.
Does the CONSORT checklist improve the quality of
reports of randomised controlled trials? A systematic
review. Paper presented at the Fifth International
Congress on Peer Review and Biomedical Publication,
Chicago, IL, September, 2005.
24 Jüni P, Altman DG, Egger M. Systematic reviews in
health care: assessing the quality of controlled clinical
trials. BMJ 2001;323:42–6.
25 Huwiler-Muntener K, Jüni P, Junker C, Egger M.
Quality of reporting of randomised trials as a measure
of methodological quality. JAMA 2002;287:2801–4.
26 Rennie D. CONSORT revised – improving the
reporting of randomised trials. JAMA 2001;285:2006–
7.
27 Des Jarlais DC, Lyles C, Crepaz N. Improving the
reporting quality of non-randomised evaluations of
behavioural and public health interventions. The
TREND statement. Am J Public Health 2004;94:
361–6.
28 Bossuyt PM, Reitsma JB, Bruns DE et al. Towards
complete and accurate reporting of studies of diag-
nostic accuracy: the STARD initiative. Clin Chem
2003;49 (1):1–6.
29 Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D,
Stroup DF. Improving the quality of reports of
meta-analyses of randomised controlled trials: the
QUOROM statement. Quality of reporting of
meta-analyses. Lancet 1999;354:1896–900.
30 Stroup DF, Berlin JA, Morton SC et al. Meta-analysis of
observational studies in epidemiology: a proposal for
reporting. JAMA 2000;283:2008–12.
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
31 Wolf FM. Methodological quality, evidence and
Research in Medical Education (RIME). Acad Med
2004;79 (Suppl 10):68–9.
32 Morrison J, Prideaux D. Ethics approval for research in
medical education. Med Educ 2001;35:1008.
33 Tomkowiak JM, Gunderson AJ. To IRB or not to IRB?
Acad Med 2004;79:628–32.
34 McLachlan JC, McHarg J. Ethical permission for the
publication of routinely collected data. Med Educ
2005;39:944–8.
35 Fraenkel JR, Wallen NE. How to Design and Evaluate
Research in Education. New York, NY: McGraw-Hill
2003;267–89.
36 Reed D, Price EG, Windish DM et al. Challenges in
systematic reviews of educational intervention studies.
Ann Intern Med 2005;142:1080–9.
37 Haig A, Dozier M. BEME Guide No. 3: Systematic
searching for evidence in medical education. Part 1.
Sources of information. Med Teach 2003;25:352–63.
38 Pierie JPEN, Walvoort HC, Overbeke AJPM. Readers’
evaluation of effect of peer review and editing on
quality of articles in the Nederlands Tijdschrift voor
Geneeskunde. Lancet 1996;348:1480–3.
39 Goodman SN, Berlin J, Fletcher SW, Fletcher RH.
Manuscript quality before and after peer review and
editing at Annals of Internal Medicine. Ann Intern Med
1994;121:11–21.
40 Soares HP, Daniels S, Kumar A et al. Bad reporting
does not mean bad methods for randomised trials:
observational study of randomised controlled trials
performed by the Radiation Therapy Oncology Group.
BMJ 2004;328:22–4.
41 Morrison JM, Sullivan F, Murray E, Jolly B. Evidence-
based education: development of an instrument to
critically appraise reports of educational interventions.
Med Educ 1999;33:890–3.
42 Bordage G, Caelleigh AS, Steinecke A et al. Review
criteria for research manuscripts. Acad Med
2001;76:897–978.
43 Bordage G, Dawson B. Experimental study design and
grant writing in 8 steps and 28 questions. Med Educ
2003;37:376–85.
44 Crandall SJ, Caelleigh AS, Steinecke A. Reference to
the literature and documentation. Acad Med
2001;76:925–7.
45 McGaghie WC, Bordage G, Shea JA. Problem state-
ment, conceptual framework and research question.
Acad Med 2001;76:923–4.
46 Morrison J. Developing research questions in medical
education: the science and the art. Med Educ
2002;36:596–7.
47 Shea JA, Arnold L, Mann KV. A RIME perspective on
the quality and relevance of current and future
medical education research. Acad Med 2004;79:931–8.
48 Campbell DT, Stanley JC. Experimental and Quasi-
experimental Designs for Research. Chicago, IL: Rand
McNally 1966.
49 Shrout PE, Fleiss JL. Intraclass correlations: uses in
assessing rater reliability. Psychol Bull 1979;86:420–8.

50 Flanagan B, Nestel D, Joseph M. Making patient safety
the focus: crisis resource management in the under-
graduate curriculum. Med Educ 2004;38:56–66.
51 Pangaro L, McGaghie WC. Relevance. Acad Med
2001;76:927–9.
52 Prideaux D. Researching the outcomes of educational
interventions: a matter of design. BMJ 2002;324:126–7.
53 Inouye SK, Fiellin DA. An evidence-based guide to
writing grant proposals for clinical research. Ann Intern
Med 2005;142:274–82.
54 Schroter S, Plowman R, Hutchings A, Gonzalez A.
Reporting of ethical committee approval and patient
consent by study design in 5 general medical journals.
Paper presented at the Fifth International Congress on
Peer Review and Biomedical Publication, Chicago, IL,
September, 2005.
55 Henry RC, Wright DE. When do medical students
become human subjects of research? The case of
programme evaluation. Acad Med 2001;76:871–5.
56 Glassick CE. Boyer’s expanded definitions of scholar-
ship, the standards for assessing scholarship, and the
elusiveness of the scholarship of teaching. Acad Med
2000;75:877–80.
57 Schriger DL, Arora S, Altman DG. The content of
medical journal instructions for authors. Ann Emerg
Med 2006;48:743–9.
58 Altman DG, for the Consort Group. Endorsement of
the CONSORT statement by high-impact medical
journals: survey of instructions for authors. BMJ
2005;330:1056–7.
59 Jefferson T, Smith R, Yee Y, Drummond M, Pratt M,
Gale R. Evaluating the BMJ guidelines for economic
submissions: prospective audit of economic submis-
sions to BMJ and The Lancet. JAMA 1998;280:275–7.
60 Pitkin RM, Branagan MA. Can the accuracy of abstracts
be improved by providing specific instructions? A
randomised controlled trial. JAMA 1998;280:267–9.
61 Marusic M, Markulin H, Lukic IK, Marusic A. Academic
advancement of authors receiving tutoring from a
medical journal. Teach Learn Med 2006;18:126–9.
62 Jefferson T, Alderson P, Wager E, Davidoff F. Effects of
editorial peer review: a systematic review. JAMA
2002;287:2784–6.
 Blackwell Publishing Ltd 2007. MEDICAL EDUCATION 2007; 41: 737–745
745
63 Lee KP, Boyd EA, Bero LA. Editorial changes to
manuscripts published in major biomedical journals.
Paper presented at the Fifth International Congress on
Peer Review and Biomedical Publication, Chicago, IL,
September, 2005.
64 Pitkin RM, Branagan MA, Burmeister LF. Effectiveness
of a journal intervention to improve abstract quality.
JAMA 2000;283:481.
Received 7 August 2006; editorial comments to authors 19
December 2006; accepted for publication 2 March 2007
SUPPLEMENTARY MATERIAL
The following supplementary material is available for
this article:
Table S1. Prevalence of key elements of reporting
experimental studies.
Table S2. Prevalence of the four basic elements of a
statement of study intent.
Appendix S1. Experiments in Medical Education in
Six Journals, 2003-2004.
The material is available as part of the online article
from:
http://www.blackwell-synergy.com/doi/abs/
10.1111/j.1365-2923.2007.02777.x
(This link will take you to the article abstract).
Please note: Blackwell Publishing is not responsible
for the content or functionality of any supplementary
materials supplied by the authors. Any queries (other
than missing material) should be directed to the
corresponding author for the article.