ORIGINAL REPORTS

Validation of a Web-Based Curriculum for

Resident Education in Orthopedic Surgery$

Barrett Boody, MD,* Patrick Johnson, DO, Andrew Pugely, MD, Daniel Miller, MD,
Jeffrey Geller, MD, William Payne, MD, James Boegener, MD, Michael Schafer, MD,*
and Matthew Beal, MD*

*
Department of Orthopaedic Surgery, Northwestern Memorial Hospital, Chicago, Illinois; Department of
Orthopaedic Surgery, Francisican St. James Health, Chicago Heights, Illinois; Department of Orthopaedic
Surgery, University of Iowa, Iowa City, Iowa; Department of Orthopaedic Surgery, Columbia University
Medical Center, New York, New York; and Department of Orthopaedic Surgery, University of
MissouriKansas City, Kansas City, Missouri

BACKGROUND: The Orthopedic In-Training Examina-
tion is an annual standardized examination with multiple-
choice questions focused on application of orthopedic surgery
core knowledge and principles. The outcome of this test can
be used to both predict how residents are progressing in their
orthopedic knowledge as well as assess their likelihood to pass
the Orthopedic Board Examinations, that is the American
Boards of Orthopedic Surgery Part 1, following completion
of residency. Preparation for the examination can be difcult,
as residents commonly have limited study time to review the
vast amounts of available published literature. The objective
of our study is to evaluate the effectiveness of the Orthopae-
dic In Training Examination (OITE) scores and the partic-
ipants perceived utility of the curriculum for OITE
preparation.
METHODS: Residents from 5 US Orthopedic residencies
(4 M.D. and 1 D.O.) were included in a pilot program of
the Orthobullets PASS curriculum in the academic year
2013 to 2014. Only residents enrolled in the PASS
curriculum who completed both the 2013 and 2014 OITEs
were included in the nal analysis (n 71). We used the
OITE 2013 and 2014 rank postgraduate year (RPGY)
reported scores to assess for efcacy of the PASS curriculum,
as the RPGY score provides postgraduate year-of-training
matched analysis to control for expected increased levels of
knowledge with subsequent retesting.
RESULTS: While OITE scores incrementally increased for
the group as a whole (n 71, RPGY mean improvement
2.5%, p 0.406), the junior resident subgroup (post-
graduate year 1-2) produced a statistically signicant increase
in scores (n 28, RPGY mean increase 10.1%,
p 0.0260). Nearly 90% (38/42) of curriculum participants
surveyed reported a preference to complete a similar review
curriculum for future OITE preparation. The participants
completing greater than 150 PASS questions (n 57) were
analyzed for OITE predictive capacity of the PASS curricu-
lum. Pearson analysis with PASS questions percent answered
correctly and 2014 OITE raw score (independent and
dependent variables, respectively) suggests moderate correla-
tion of the variables (R 0.682, p o 0.05).
CONCLUSION: We propose that the Orthobullets PASS
curriculum is a useful tool for OITE preparation, especially
for junior residents, with a focus on achieving competency
for a broad fund of orthopedic knowledge, whereas mastery
of these topics would be better suited through using
complementary sources. ( J Surg Ed ]:]]]-]]]. J C 2016
Association of Program Directors in Surgery. Published by
Elsevier Inc. All rights reserved.)
KEYWORDS: orthopedic surgery, web-based curriculum,
in-training examination, resident self-assessment
COMPETENCIES: Medical Knowledge

An author (D.J.M.) received a monetary reward from his residency program for
his high level of performance in the Orthobullets PASS curriculum. No authors have INTRODUCTION
nancial ties to Orthobullets or any other direct or indirect conicts of interest related
to this article. Program directors nor residency programs were compensated for their The Orthopedic In-Training Examination is an annual
utilitzation of the PASS curriculum.
standardized examination administered to all US orthopedic
Correspondence: Inquiries to Barrett Boody, MD, Department of Orthopaedic
Surgery, Northwestern Memorial Hospital, 676 N St. Clair St, Ste 1350, Chicago, residents with multiple-choice questions focused on appli-
IL 60611; fax: (312) 926-4643; e-mail: barrett.s.boody@gmail.com cation of orthopedic surgery core knowledge and principles.

Journal of Surgical Education  & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 1
Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.05.020
The purpose of the examination is to provide an objective
assessment of a residents gross fund of orthopedic knowl-
edge, reporting results regarding both the overall testing
population, and a level-of-training matched analysis (rank
postgraduate year [RPGY]). The outcome of this test can
be used to both predict how residents are progressing in
their orthopedic knowledge as well as assess their likelihood
to pass the Orthopedic Board Examinations, that is, the
American Boards of Orthopaedic Surgery Part 1, following
completion of residency.1-3
Our study evaluates the utility and efcacy of a new web-
based curriculum designed for orthopedic residentsPASS
curriculum. Preparation for the examination can be dif-
cult, as residents commonly have limited study time to
review the vast amounts of available published literature.
The Orthobullets PASS curriculum provides access to daily
comprehensive reading curriculum and optional timed
subject quizzes and mock examinations. The curriculum
used scheduled monthly quizzes in ofce of intramural
training and education (OITE) format with the intended
purpose of assisting residents identify their progression and
deciencies in orthopedic knowledge, as well as facilitate
program directors in identifying at-risk residents for poor
performance on the OITE, and thus more importantly,
failure risk for residents preparing to take American Board
of Orthopaedic Surgery (ABOS) Part 1.
The objective of our study is to evaluate the effectiveness
of the Orthobullets PASS curriculum on improving OITE
scores and the participants perceived utility of the curric-
ulum for OITE preparation. We compared residents OITE
performance from both the year before (2013) and year of
the Orthobullets PASS curriculum (2014) to assess the
comparative effectiveness of the new curriculum technique
over conventional examination preparation techniques. We
additionally compared residents PASS curriculum and
OITE performance to evaluate the PASS curriculum for
predictive capabilities regarding future OITE performance.
Additionally, we surveyed the residents to understand how
participants using the Orthobullets PASS curriculum used
the resource, perceived the efcacy of the resource, and their
likelihood to use the curriculum again.
DESIGN
Residents from 5 US orthopedic residencies (4 M.D. and 1
D.O.) were included in a pilot program of the Orthobullets
PASS curriculum in the academic year 2013 to 2014.
Residency programs integrated the PASS curriculum into
their core OITE preparation curriculae and received com-
plementary access to the PASS curriculum for their partic-
ipation in pilot testing. Residents were required to
participate in the PASS curriculums monthly multiple-
choice tests designed to reect similar material previously
presented on the OITE. Although program directors were
2 Journal of Surgical Education  Volume ]/Number ]  ] 2016
made aware of participants results, no consequences for
poor performance or noncompliance were reported. Results
of PASS curriculum monthly quizzes were provided for
analysis by the Orthobullets editorial team. Program direc-
tors from the 5 institutions submitted their programs
OITE results from the 2013 to 2014 years. Once each
residents OITE and Orthobullets PASS program data were
correlated for statistical analysis, identifying information was
deleted for condentiality. Only residents enrolled in the
PASS curriculum who completed both the 2013 and 2014
OITEs were included in the nal analysis. All residents
participating in the PASS program from the 5 institutions
were asked to complete a brief survey about their OITE
study preparation and their perceived utility of the Ortho-
bullets PASS curriculum for OITE preparation. The North-
western University Institutional Research Board approved
the study. Neither the participating programs nor the
authors were compensated for the utilization of the PASS
curriculum or publication of this article.
Statistics
Correlated OITE and PASS data for 71 residents were
available for statistical analysis. We used the OITE 2013
and 2014 RPGY reported scores to assess for efcacy of the
PASS curriculum, as the RPGY score provides postgraduate
year (PGY) year-of-training matched analysis to control for
expected increased levels of knowledge at subsequent
retesting. Subgroup analysis was performed to determine
the effect of the PASS curriculum for junior resident (PGY
1 and 2 at the time of the 2013 OITE) versus senior
resident (PGY 3 and 4 at the time of the 2013 OITE) and
individual PGY subgroups. Paired t-tests were performed for
overall and subgroup analyses as participants OITE scores
allow for matched pair analysis and 2-tailed p-values were
reported. Pearson correlations were performed for the PASS
curriculum performance to determine predictive relation-
ships with performance on the 2014 OITE.
RESULTS
An overall (n 71) RPGY score increase of 2.5% (p
0.406) was seen after participation in the PASS curriculum,
with raw scores on average increasing from 163 to 176
(maximum score of 275). On subgroup analysis, the junior
resident subgroup (n 28, PGY 1 and 2 at the time of the
2013 OITE) demonstrated signicant improvements in
OITE performance, with raw score increases from 139 to
164 and an average 10.1% improvement in RPGY score
(p 0.0259). The senior resident subgroup (n 43, PGY
3 and 4 at the time of the 2013 OITE) demonstrated an
average raw score improvement from 178 to 183an average
of 2.4% decrease of their RPGY score on the 2014 OITE,
which lacked statistical signicance (p 0.5445). The
 Overall Analyses Including Junior
Versus Senior Resident
Subgroups: Basic Stascs
RPGY 2013 RPGY 2014
PGY 2013 Mean Std Mean Std
PGY 1&2
(n=28) 48.18% 29.89% 58.25% 28.41%

PGY 3&4
61.07% 27.32% 58.65% 27.57%
(n=43)
Overall
55.99% 28.86% 58.49% 27.70%
(n=71)

Overall Analyses Including Junior

Versus Senior Resident
Subgroups: Paired t-tests with p-
values
Group DF t Value Pr > |t|
PGY 1&2 27 2.356 0.026
PGY 3&4 42 0.611 0.545
Overall 70 0.835 0.406

FIGURE 1. Overall and junior versus senior resident subgroup performances are shown. Basic statistics of mean scores and standard deviation are
provided for the 2013 and 2014 OITE administrations. As the overall and subgroup analyses are comprised of subsequent retesting of the same
participants, a paired t-test with p-values are provided.

decrease in mean RPGY score for the senior residents
appeared to be largely from the 2013 PGY 4 subgroup
(mean decrease of 9.8%, p 0.067). When excluding
the 2013 PGY 4 subgroup from the overall analysis, the
2013 PGY 1 to 3 subgroup improved an average 7.3%
(p 0.041). Score analyses that could not be meaningfully
correlated to previous OITE performances without account-
ing for the interval improvement in OITE scores
as expected with increased experience (i.e., raw scores)
were provided for context, but not statistically analyzed
(Figs. 1 and 2).
Participants completed an average of 194 PASS questions
(standard deviation [SD] 75.4) of a possible 275 PASS
milestones mock examination questions before the 2014
OITE with an average of 69% answered correctly (SD
12%). The results of 70 participants who completed at least
1 PASS test (25 questions per test) were subsequently
analyzed for OITE predictive capacity of the PASS curric-
ulum. Pearson analysis with PASS questions percent
answered correctly and 2014 OITE raw score (independent
and dependent variables, respectively) suggests moderate
correlation of the variables (R 0.649, p o 0.05) (Fig. 3).
Totally, 42 residents from the 5 participating residency
program responded to surveys regarding their OITE study
habits and utility of the PASS program. Residents reported
high perceived utility of the PASS program for OITE
preparation. When asked to rate the utility of the PASS
curriculum using a global rating scale from 1 (no utility) to

Subgroups by PGY level:

Basic Stascs
RPGY 2013 RPGY 2014
PGY 2013 Mean Std Mean Std
1 49.36% 30.44% 55.91% 26.37%
2 47.41% 30.45% 59.77% 30.35%
3 60.74% 29.86% 64.70% 32.20%
4 61.45% 25.86% 51.70% 19.62%

Subgroups by PGY level:

Paired t-tests with p-
values
PGY2013 DF t Value Pr > |t|
1 10 0.722 0.487
2 16 3.02 0.008
3 22 0.689 0.499
4 19 1.944 0.067

FIGURE 2. Subgroup analysis by PGY level. Basic statistics of mean scores and standard deviation are provided for the 2013 and 2014 OITE
administrations. As the subgroup analyses are comprised of subsequent retesting of the same participants, a paired t-test with p-values is provided.

Journal of Surgical Education  Volume ]/Number ]  ] 2016 3

 Correllaon of PASS quesons
answered correctly versus 2014 OITE
RAW score

240

220

200
2014 OITE RAW Score

180

160

140

120

100
0.4 0.5 0.6 0.7 0.8 0.9 1
PASS % Correct

Pearson Correlation Coefficient 0.649

R Standard Error 0.009
t 6.98
p-value <0.05
H0 (5%) rejected

FIGURE 3. Linear regression model demonstrating correlation between percentage of PASS milestone questions answered correctly and 2014 OITE
raw scores. Participants who had completed at least 1 PASS test (Z25 questions answered) were included in the analysis (n 70).

10 (signicantly useful) for OITE preparation, respondents
reported on average 6.9 (SD 2.24). Furthermore, nearly
90% (38/42) noted they would prefer to complete a similar
structured review curriculum before future OITEs. When
asked specically about study habits, practice questions
appeared to be the most common study technique, with
35.7% of respondents (15/42) using practice questions for
51% to 75% of their OITE preparation time and 50.0% of
respondents (21/42) using practice questions for 76% to
100% of their OITE preparation time (Figs. 4-6).
States Medical Licensing Examination steps 1 and 2 scores
with OITE and ABOS performance.1,2,4
Although success on the ABOS Part 1 may be predicted
by residents raw test-taking abilities, a residents orthopedic
fund of knowledge and problem-solving capabilities may
demonstrate the most robust predictive capacity for board
examinations. Multiple authors have correlated OITE
performance with success on board examinations such as
the ABOS Part 1 and the Royal College of Physicians and
Surgeons of Canada (a Canadian orthopedic board

DISCUSSION
Various publications have sought to establish correlations
and predictive capacity of standardized tests taken by
orthopedic residents to the ABOS Part 1 licensing exami-
nation. Although orthopedic residents historically have
performed well on standardized examinations, the interest
in predicting ABOS outcomes is because of the high stakes
nature of board examinations and the desire to predict
FIGURE 4. Selected responses from the resident participation survey.
likelihood of success on the boards. For example, several Responses for Would you prefer a similarly structured curriculum before
authors have demonstrated predictive correlations of United future OITEs? (n 42 responses).

4 Journal of Surgical Education  Volume ]/Number ]  ] 2016


FIGURE 5. Selected responses from the resident participation survey.
Responses for Please rate the utility of the PASS curriculum for OITE
preparation (1no utility, 10signicant utility), results stratied by
PGY level. (n 42 responses).
certication examinations).2,3,5,6 However, as the ABOS
Part 1 is a licensing examination with a pass/fail format,
more interest has been focused on identifying predictive
tools for residents at risk of failing. Dougherty et al.1
reported a 57% chance of failing the ABOS Part 1 for
residents with mean OITE scores below the 27th percentile.
Similarly, Swanson et al.6 noted PGY 5 residents with
OITE scores below the 10th percentile were more likely to
fail the ABOS Part 1 than those who scored above the 50th
percentile (18% versus 3%, respectively). Accordingly,
residency programs have placed signicant emphasis on
OITE preparation, as programs with an increased emphasis
on OITE performance perform better on the OITE.7
Multiple review materials are available for OITE prepa-
ration, ranging from review books (e.g., American Academy
of Orthopaedic Surgeons [AAOS] Comprehensive Review
and Millers Review of Orthopaedics), practice questions
from Orthobullets and AAOS, and review websites such as
Orthobullets.com and WheelessOnline.com. The most
commonly used and recommended sources are practice
questions modeled on previous OITE questions, such as
Orthobullets and AAOS self-assessment examinations.8,9
Our data similarly suggested a preference for practice
questions, with 85% of participants using practice questions
for over 50% of their examination preparation time.
Beyond providing practice questions, the PASS curriculum
uses a structured curriculum to maintain steady progress
with examination preparation, as daily reading and more
hours committed to studying have correlated to increased
OITE scores.10 In addition, a daily reminder with a more
proactive format suggests a more consistent approach with
more focused topical learning.
Our results demonstrate improved performance on OITE
examinations with the PASS curriculum intervention, with
the junior residents subgroup (2013 PGY 1-2) displaying
statistically signicant improvement in scores. Further
review of the data suggest a 2013 PGY 1 to 3 subgroup
Journal of Surgical Education  Volume ]/Number ]  ] 2016
(i.e., eliminating the 2013 PGY 4 group from the overall
analysis) would see statistically signicant improvements in
OITE performance; however, this was not one of our
intended subgroups for analysis. The reported utility of
the PASS curriculum was high, as 90% of participants
reviewed the curriculum positively and likely to use again
with even distributions of reported satisfaction across each
PGY class. After analyzing our results, we propose the
material available on Orthobullets PASS curriculum may be
better suited to achieving competence within the general
knowledge base of orthopedics, whereas senior residents
who have already mastered these topics may not see similar
incremental improvement with merely a review of high-
yield topics. Although review of performance on PASS
curriculum questions demonstrated good correlative
capacity with 2014 OITE raw scores, we are hesitant to
report predictive capacity of the PASS curriculum for OITE
performance from our small cohort (n 57, participants
with 150 completed PASS questions or greater).
Although multiple studies have been conducted on
review materials and curricula for OITE preparation, we
believe ours is the highest quality data to be reported.11,12
We report the rst multicenter data from East Coast and
Midwest residency programs, including both D.O. and M.
D. programs, undergoing a structured curriculum with the
purpose of improving OITE scores and core orthopedic
knowledge. Our study improves upon prior research
through increased sample sizes (n 71 residents, 5 residency
programs) and use of a well-dened, reproducible curricu-
lum, improving generalizability, and relevance of our studys
outcomes. Although comparative studies have demonstrated
improvements in resident OITE performance with imple-
mentation of a standardized curriculum, most are under 25
residents and limited to 1 residency program.11,12
The Orthobullets PASS curriculum study does demon-
strate several limitations. Our sample size (n 71) limits
reporting of subgroup analysis and predictive capacity of the
FIGURE 6. Selected responses from the resident participation survey.
Responses for Would you prefer a similarly structured curriculum before
future OITEs?, results stratied by PGY level (n 42 responses).
5
PASS curriculum. Furthermore, as the project did not
collect participant ABOS Part 1 data, we cannot directly
comment on the efcacy of the PASS curriculum for
improving ABOS Part 1 outcomes. We also were not able
to correlate survey results and OITE/PASS scores, limiting
our ability to associate subjective perceptions of utility, and
objective efcacy of the PASS curriculum. Despite these
limitations, our data suggest a valuable role for the PASS
curriculum in the preparation of residents for the OITE
with demonstrated efcacy through improved OITE scores
and high perceived utility from curriculum participants.
CONCLUSION
Although the scores incrementally increased for the group as
a whole, the junior resident subgroup (PGY 1-2) produced a
statistically signicant increase in OITE scores after com-
pletion of the PASS curriculum. Furthermore, the residents
have largely reported positive feedback regarding the per-
ceived utility of the curriculum with 90% preferring to
repeat a similar curriculum for future OITE preparation.
We propose that the Orthobullets PASS curriculum is a
useful tool for OITE preparation, especially for junior
residents, with a focus on achieving competency for a broad
fund of orthopedic knowledge, whereas mastery of these
topics would be better suited through using complementary
sources.
REFERENCES
1. Dougherty PJ, Walter N, Schilling P, Najibi S,
Herkowitz H. Do scores of the USMLE Step 1 and
OITE correlate with the ABOS Part I certifying
examination? A multicenter study. Clin Orthop Relat
Res. 2010;468(10):2797-2802.
2. Dyrstad BW, Pope D, Milbrandt JC, Beck RT,
Weinhoeft AL, Idusuyi OB. Predictive measures of a
residents performance on written Orthopaedic Board
scores. Iowa Orthop J. 2011;31:238-243.
3. Ponce B, Savage J, Momaya A, et al. Association between
orthopaedic in-training examination subsection scores
6 Journal of Surgical Education  Volume ]/Number ]  ] 2016
and ABOS Part I examination performance. South Med
J. 2014;107(12):746-750.
4. Black KP, Abzug JM, Chinchilli VM. Orthopaedic in-
training examination scores: a correlation with
USMLE results. J Bone Joint Surg Am. 2006;88(3):
671-676.
5. Yen D, Athwal GS, Cole G. The historic predictive
value of Canadian orthopedic surgery residents ortho-
pedic in-training examination scores on their success
on the RCPSC certication examination. Can J Surg.
2014;57(4):260-262.
6. Swanson D, Marsh JL, Hurwitz S, et al. Utility of
AAOS OITE scores in predicting ABOS Part I out-
comes: AAOS exhibit selection. J Bone Joint Surg Am.
2013;95(12):e84.
7. Evaniew N, Holt G, Kreuger S, et al. The orthopaedic
in-training examination: perspectives of program direc-
tors and residents from the United States and Canada.
J Surg Educ. 2013;70:528-536.
8. Krueger CA, Aden J. Is there an association between
study materials and scores on the American Board of
Orthopaedic Surgeons Part 1 examination? J Surg
Educ. 2014;71:375-384.
9. LaPorte DM, Marker DR, Seyler TM, Mont MA,
Frassica FJ. Educational resources for the Orthopedic
In-Training Examination. J Surg Educ. 2010;67(3):
135-138.
10. Miyamoto RG Jr, Klein GR, Walsh M, Zuckerman
JD. Orthopedic surgery residents study habits and
performance on the orthopedic in-training examina-
tion. Am J Orthop. 2007;36(12):E185-188.
11. Weglein DG, Gugala Z, Simpson S, Lindsey RW.
Impact of a weekly reading program on orthopedic
surgery residents in-training examination. Orthopedics.
2015;38(5):e387-393.
12. Klena JC, Graham JH, Lutton JS, Temple JL, Beck
JD. Use of an integrated, anatomic-based, orthopaedic
resident education curriculum: a 5-year retrospective
review of its impact on orthopaedic in-training exami-
nation scores. J Grad Med Educ. 2012;4:250-253.