Survey Course on Research Methods: Integrating Statistical

Analysis and Study Design

JoLaine R. Draugalis, Jean T. Carter and Marion K. Slack

College of Pharmacy, The University of Arizona, P.O. Box 210207, Tucson AZ 85721-0207

The purpose of the three-credit (45 hours) required course is to ensure PharmD student competence in
selecting, interpreting, and evaluating clinical drug literature. The course is a prerequisite for a three-credit
Drug Information/Drug Literature Evaluation course in the professional curriculum. Topics covered include
principles of research design, acquisition and interpretation of data, statistical analysis, and validity of
conclusions. The essential connection between research design and statistical analysis is emphasized
throughout the course in lectures, examples, assignments, and exams. The purpose of this article is to share
the methods used to convey statistical analysis and research design concepts as well as methods for
assessing the students. The techniques and approaches described are also applicable to other courses in
the pharmacy curriculum. Examples also show how instructors used evaluation and revision to cultivate
integration of concepts, encourage more active learning and class participation, and enhance assessment
procedures.

INTRODUCTION
In 1992, an article was published that provided a description
of a three credit (45 hours) required research evaluation
course in an entry-level PharmD program(1). The article
concluded with a list of course improvement efforts includ-
ing content changes, additions of assignments, and examina-
tion improvement. In that more institutions will be offering
a course of this nature by virtue of increased numbers of
entry-level PharmD programs, this update details our on-
going improvement efforts as well as other refinements of
the past six years. The materials and methods of this course
address a number of specific educational outcomes and
general ability-based outcomes (including thinking abili-
ties, communication abilities, responsible use of values and
ethical principles, and self-learning abilities and habits)
deemed requirements of a competent pharmacy practitio-
ner(2). As well, the course meets Standards 12 (teaching and
learning processes) and 13 (student evaluation) in the new
ACPE accreditation manual(3). Particularly relevant are
Guidelines 12.2 and 12.3 which encourage teaching ap-
proaches that require problem solving capabilities and pro-
mote life-long learning to in turn, ensure professional com-
petence. Guideline 13.1 specifically argues for an evaluation
system that examines higher levels of learning, such as
integration, application, and synthesis rather than simply
testing for rote memorization of specific facts. Finally, a
philosophy of teaching as scholarship as espoused by Boyer
guides instructors in that the goal is to transform and extend
knowledge, rather than simply transmit information(4).
Although this paper provides examples for a specific con-
tent area, the methods embraced and the techniques and
approaches used most certainly can be applied to many
other courses in the pharmacy curriculum.
COURSE DESCRIPTION
In order for students to use the skills gained in this course
throughout their study plan, a curriculum revision resulted
in moving the course from the final didactic semester (prior
to the final year of clerkship rotations) to first semester of
the second professional year. The Drug Information/Litera-
ture Evaluation (DI) course is now in the semester immedi-
ately following the Methods course, rather than in the
semester preceding it. The DI course was also increased by
one unit to three units. Some topics previously covered in
the Methods course (e.g., Meta-Analysis, Proposal Writing,
Evaluation Methods, and article critique) have been moved
to the DI course, in order to dedicate more time to statistical
techniques/data analysis as an introductory biostatistics
course is no longer a prerequisite. This decision was made
because prerequisite course content and student retention
levels were so varied. However, as the prepharmacy curricu-
lum is evaluated over the next year, an entry level biostatis-
tics course requirement will be revisited. The pharmaco-
economic evaluation component was deleted several years
ago as this topic is now covered in much more detail in the
Professional Practice Management course.
The overall course purpose is to provide the student
with the skills and principles of clinical research design and
biostatistics needed for interpretation and evaluation of the
medical literature and assessment of research reports and
proposals. The overview of statistics module begins with a
quotation from H.G. Wells, “Statistical thinking will one
day be as necessary for efficient citizenship as the ability to
read and write.” Our quest for conceptual understanding
over mathematical “plug-and-chug” begins in earnest on
day one and continues throughout the course. Teaching
students to be healthy skeptics, arming them with tools to be
proficient professional decision makers, and challenging
them to accept responsibility for their actions are necessary.
Encouraging open-mindedness and discouraging the urge
to over-generalize based on an N of 1 are crucial. The
overriding objective is to ensure competence in selecting,
interpreting, and evaluating clinical drug literature, includ-
ing principles of research design, acquisition and interpreta-
tion of data, statistical analysis, and validity of conclusions.

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 17


Fig. 1. Course outline. Each topic is covered in one 75-minute
session unless otherwise noted.
The specific objectives addressing this charge have not been
changed (other than deleting those related to deleted/reas-
signed material) since the original report was presented(1).
The original report also details the specific learning objec-
tives. The course is now offered in two 75-minute class
periods per week (see Figure 1).
COURSE METHODS
The first day of class we disseminate a 360-page course
notebook. This notebook contains the syllabus, lecture notes,
reprints, objectives, and references to be used during the
semester. Class materials, including in-class examples, are
chosen to make concepts more meaningful for students be it
clinical or practice relevance. Articles and reports from the
medical literature as well as the lay press provide an endless
source of new material. The course coordinator oversees all
aspects of course preparation and planning and provides 80
percent of the lectures. The other instructor provides 20
percent of the lectures. Occasionally, a guest lecturer may
provide one or two lectures (e.g., epidemiology topics). A
graduate teaching assistant is available to prepare and grade
assignments.
The course is still primarily lecture-based, however we
have sought to provide increased opportunities for interac-
tion as well as in-class activities for certain topics. In fact, on
the first day of class, each of the instructors provides about
25 minutes of lecture on terminology and the scientific
method and the remaining 25 minutes is spent having the
students (either in groups or the class as a whole) generate a
18 American Journal of Pharmaceutical Education Vol. 62, Spring 98
scientific prediction, describe an example of a cause-and-
effect relationship in health care, and identify the indepen-
dent (including levels) and dependent variables. Some re-
cent examples include: ill-fitting shoes cause blisters, sun
exposure causes skin cancer, drinking Tucson water causes
pregnancy, and completing lecture early results in class
dismissal. The class found out early-on that they would be
unable to test the last hypothesis. From day one, we begin
the discussion of plausible alternative hypotheses and the
role of research design in controlling these threats.
In Part 2, Statistical Techniques/Data Analysis, the
course notes provide definitions and compare and contrast
each of the statistical techniques. See Figure 2 for an in-class
exercise for the chi square test of distribution. The format
includes a very brief scenario, data display, null and alterna-
tive hypothesis, calculation example, and finally an exercise
where students use a table of critical values to ultimately
make a statistical decision.
RESEARCH DESIGN SECTION (Part 3 of Course)
The goal of the research design section is for students to
learn basic research design concepts. Specifically, students
should be able to distinguish among different types of
research design including experimental, quasi-experimen-
tal, pretest-posttest, and case studies, and associated threats
to internal validity when they complete the section. The
section is based on the presentation of research design and
internal validity by Campbell and Stanley(5). To learn
concepts, students must see examples of each type of re-
search design and learn to identify the critical characteris-
tics that differentiates one design from another(6). The
basic instructional strategy is to present a definition and
describe the critical characteristics of each research design
then to analyze a research report using that design. Follow-
ing the discussion of research designs, related issues such as
types of randomization, how to handle compliance and
withdrawals in a clinical study, and how to interpret P-
values are presented in Part 4. Additional practice analyz-
ing research reports is provided in the following semester in
the drug information course.
Every effort is made to connect research design issues
to practice. To introduce the concept of research design, an
example study problem is presented. The current example
problem is a practitioner wanting to conduct a study to
determine if using community health workers will increase
the number of people who complete prophylactic therapy
for tuberculosis. Students are presented with the question
“How will we know if this program works?” and asked to
generate ideas about how they would answer the question.
Student responses are listed on the board and, as appropri-
ate, their ideas are translated into research design language.
For example, a student may express concern over the type
of education that the patients in the control group will
receive outside the study situation; this concern is trans-
lated into a concern with internal validity. Or a student
might suggest that a control group is needed which is
followed by a brief discussion of how to assign patients to
groups.
To further increase relevance to a practice situation,
external validity (generalizability) is presented as answer-
ing the question “Do the results of this study apply to
patients that I see in my practice?” Students are directed
toward the inclusion/exclusion criteria for the study and the
description of the study sample in the results section. They
Fig. 2. Course notes example
are shown how to use the demographic and disease informa-
tion to describe the sample and relate it to patients they
might see in their practice.
The research design concepts are presented based on a
core knowledge approach. That is, certain concepts are
considered critical to learning additional concepts and facts.
We believe that by learning to recognize basic research
designs and the characteristics of those designs that students
can refine their skills as they gain more experience reading
research reports. For example, until students learn to recog-
nize an independent versus a dependent variable or to
differentiate an experimental from a pretest-posttest de-
sign, they cannot hope to recognize threats to internal
validity or identify whether the statistical tests reported are
appropriate. Continued self-learning is very difficult if the
student cannot recognize the critical features of a research
study such as the independent and the dependent variable.
On the exam at the end of the section, students averaged 93
percent on the items testing recognition of research designs
last semester.
Active participation in learning is included during class
time to the extent possible. Students are provided with a
complete set of notes so that time is not required during class
to transmit information. Important points or suggestions
about learning the material are provided during the opening
segment of the class followed by a class discussion or small
group activities. The activities range from generating syn
American Journal of Pharmaceutical Education Vol. 62, Spring 1998
onyms for a term (e.g., research design) to completing in-
class exercises such as using a random numbers table to
assign patients to treatment groups or to completing a
worksheet identifying research design issues related to an
example research problem.
We use an integrated approach to discussing issues
associated with research design such as internal validity
issues and statistical issues. For example, threats to internal
validity and research design are integrated by asking stu-
dents to know the primary threat or threats to internal
validity associated with each type of research design. Spe-
cifically, that experimental mortality (attrition) is the pri-
mary threat to internal validity for an experimental research
design and that differential assignment to groups (selection)
is the primary threat for a quasi-experimental design. We
discuss how other threats to internal validity, for example
history or experimental mortality, may be issues with a
quasi-experimental design but students are tested only on
the primary threat. An average of 82 percent of the students
correctly identify the primary threat to internal validity for
the four research designs presented. Presentation of rel-
evant statistical tests proceeds in a similar manner. Statisti-
cal tests appropriate for use with an experimental design
include a Student’s t-test, an ANOVA, or a chi square,
depending on the type of data collected.
To facilitate student learning of the critical characteris-
tics of each research design, a cognitive map is provided. An
19
 example is shown in Figure 3. The cognitive map enables
students to focus on critical characteristics and to differen-
tiate relevant from irrelevant information. The students use
the cognitive map to analyze example research reports.
Students usually analyze six example reports. They are
provided with copies of the reports and the cognitive maps,
and asked to read the report and fill in the map before they
come to class. At the beginning of class, they turn in one copy
of the map (they keep a second copy for reference during
class) and then we discuss the characteristics of each study
and the issues associated with it. The discussion follows a
prescribed sequence in which the type of research is identi-
fied, the purpose of the study, the independent and depen-
dent variables, the statistical results, the primary threat to
internal validity and generalizability to patients seen in
practice. In general, the discussion is focused on research
design issues; strictly clinical issues may be mentioned but
are not discussed in depth.
Experience from teaching the course has shown us that
concept learning is not a linear, straight forward process.
Typically, students need to revisit basic concepts before they
can readily identify differences. For example, students can
have a great deal of difficulty learning to differentiate
random selection from random assignment. The topic is
introduced early in the course and revisited at every oppor-
tunity. In the 13th lecture (Topics in Sampling), the concepts
of random selection and random assignment are defined
and illustrated. Then as we analyze example research re-
ports, students are asked whether the researchers used
random selection or random assignment. We point out the
differences between the two techniques and the reasons
random selection is not used with clinical studies. When
students analyze example research reports, most students
can correctly differentiate random selection from random
Fig. 3. Cognitive map example. assignment. Last year, 52 of 53 students correctly identified
random assignment as the technique used to assign subjects
to treatment groups in an experimental study. Another

Fig. 4. Judgment steps for assessing a research report.

20 American Journal of Pharmaceutical Education Vol. 62, Spring 1998

difficult objective is learning to differentiate independent
from dependent variables; identification of variables is re-
visited during most research design lectures. On the relevant
exam questions, about 92 percent of the students answer the
questions correctly.
Experience also has shown us that students have a naive
misconception concerning the relative importance of inter-
nal validity versus external validity. Students typically judge
the quality of a research study by perceived external validity
and they tend to treat studies conducted in major research
and teaching hospitals as having little external validity. To
facilitate learning to assess internal validity first and exter-
nal validity only if the study is judged internally valid, a
sequential process is followed when discussing the report in
class. A supplemental handout (see Figure 4) also is used
that clearly delineates the sequence of steps in assessing a
research report: assessing the statistical conclusion, then
internal validity, and finally, external validity(7).
During an informal in-class evaluation of the research
design section, students were asked what they knew now
about research design that they didn’t know before. The
response of several students was “everything”, that they did
not know anything about research design before taking the
class. Several students commented that they liked the frame-
work (cognitive map) for analyzing the research reports and
one student stated they wished other professors would teach
this way. They also said the homework was very helpful and
they seemed to see value in the class. One student remarked
that the research design analysis was very basic, an observa-
tion that accurately reflects the level of the class.
Human subject considerations is the first topic in Part 4
(The Fundamentals of Clinical Trials). Real life examples of
unethical research and violation of informed consent are
used to underscore the need for institutional review boards’
critical analysis of research protocols. In addition, students
are shown official written informed consent and project
approval forms. Ensuing discussion invariably raises differ-
ing opinions leading to teachable moments about the role of
institutional review boards.
The final part of the class contains two sessions on each
of the topics, survey research and epidemiologic methods.
In survey research we concentrate on comparing and con- Fig. 5. Examples of assignment items over time.
trasting the four sources of error (coverage, sampling,
nonresponse, and measurement) as defined by Salant and ASSIGNMENTS
Dillman(8). Because surveys are so prevalent, it is a widely Assignments were only 10 percent of the semester grade
held belief that surveys are easy to conduct. Instructors when originally introduced to the course, now twenty-five
stress that survey research requires significant amounts of percent of the course grade is based on assignments. Five 20-
time, energy, money, and expertise; particularly when rela- point homework assignments are interspersed throughout
tionships are examined and concepts generated. Therefore, the semester serving as study guides for examinations. The
sampling, questionnaire construction, data analysis, and assignments are intended to provoke discussion and we
other technical aspects are covered. A journal reprint detail- encourage students to work together in completing course
ing the specifics of evaluating the results of mail survey objectives and homework.
research is provided to students(9). Homework assignments have evolved in several ways.
The epidemiology objectives were added to the course Items are based on scenarios from research reported in the
five years ago due to the increased numbers of literature literature. The format of items on the assignments was
reports using these methods. The primary objectives are: changed to include a mixture of multiple choice questions,
describe and evaluate the measures of disease in a popula- simple calculations, completion of tables, (e.g., source table
tion and compare and contrast the study designs used in for an analysis of variance or blocked random sampling
epidemiologic research with respect to their design and scheme). The question portion of the multiple choice items
analyses. Interpretation of relative risk ratios and odds- was converted from one-sentence statements about a con-
ratios and associated confidence intervals along with evalu- cept to simple scenarios that required the student to identify
ation criteria for cohort and case-control studies round out the concept based on the description. For example, rather
the materials. than ask the student which potential threat to internal
validity is most likely representing an experimental design,

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 21

the student received a three or four sentence description of
a study with an experimental design (i.e., prospective, con-
current comparison groups with random assignment) and
was asked which potential threat to internal validity was
most likely with this study design. The identification re-
quirement was used to further discourage rote memoriza-
tion and provide experience in determining study designs
based on descriptions of research (see Figure 5).
The multiple choice items were further revised to in-
clude several questions with each scenario. This approach
has several advantages. Well-written scenarios and sets of
responses require a substantial amount of time and numer-
ous iterations. Using the same set of responses for a concept,
such as potential threats to internal validity for different
scenarios that may or may not have the same threats re-
quires students to compare and contrast the research de-
scribed. The assignment contains examples of experimental,
quasi-experimental, and crossover designs to provide the
student with convenient comparisons.
Another advantage of this assignment preparation strat-
egy is the ability to change scenarios from year to year
without having to completely re-write response items. Stu-
dents with access to homework assignments from previous
years should not have an advantage over students without
such access. The effect of access to the additional examples
may help the student learn the concepts.
The third advantage of this approach is the ability to use
reported research as the basis for the scenarios. On-line
searches provide quick and easy access to material for
scenarios that are based on actual research. The reports are
still cleaned up somewhat to help the student focus on the
statistical and design concerns, but some “red herrings” are
occasionally left in to maintain some sense of reality. The
use of reported studies begins to bring the students closer to
working with the literature. The scenarios may also include
tables of data or results, a correlation matrix, or a source Fig. 6. Assignment example of tests for categorical frequency data.
table to further simulate the literature. The final assignment
asks a series of questions about a recently published full
research article. change numbers in the small data set or use an entirely new
The response items used in the multiple choice items are scenario each year without changing the questions that
chosen carefully and the importance of knowing why each provide a model of an approach to statistical analysis. Figure
response is or is not correct is emphasized throughout the 6 shows the use of the template with the chi-square statistic.
semester. The information in the course builds on itself, so This commonly used statistic appears to be a source of
concepts covered early in the semester are woven into the confusion among practitioners and PharmD candidates who
assignments. For example, a scenario in the last assignment call seeking advice. It is hoped that this increased experience
may ask the student to determine the level of measurement with tests for categorical frequency data will improve under-
or identify the dependent variable. standing in the future.
After observing several episodes of a student copying Assignments have been based on practitioner inquiries
the answers to the multiple choice format, assignments were as well as the medical and pharmacy literature. Another
developed with approximately half multiple choice ques- successful approach has been to base an assignment on a full
tions and half simple calculation or other hands-on tasks. article (particularly appealing if research was conducted at
This was done to encourage students to spend more time the home institution). It has been very gratifying hearing
with the material and hopefully discourage last minute “hall” talk and seeing debates rage in the student lounge
copying. over these assignments. Students rated assignments very
The calculations are integrated into an overall schema favorably on course evaluations.
for conducting a statistical analysis. A template has been
developed that may be adapted to a variety of parametric EXAMINATIONS
and nonparametric procedures. A series of questions leads We assess application and other higher levels of learning, by
the student from the research question to the statistical constructing items using examples that require students to
hypothesis to be tested, selection of an appropriate test, apply concept skills rather than simply recognize or recall
calculation of the statistic, selection of a critical value, a information. The multiple choice format has been retained
judgment of whether the null hypothesis is retained or for several reasons: item analysis may be performed and,
rejected, and how this all relates to the original research tests may be machine graded, while retaining the ability of
question. This approach allows the course instructor to assessing higher levels of cognition (1,6). Examinations are

22 American Journal of Pharmaceutical Education Vol. 62, Spring 1998

evaluated and revised based on item analysis; such analysis
differentiates a poorly written item missed by all levels of
performers from a well written item about a difficult concept
missed by all but the top performers. There are also “core”
or very basic items that everyone is expected to get right.
Each examination (three total) contains fifty multiple choice
items and constitutes 25 percent of the course grade.
For the past five years, reliability coefficients have
typically ranged from 0.65 to 0.85 using the KR-20 estima-
tion method. In that several assumptions (e.g., normal distri-
bution of both item and test scores, some amount of varia-
tion in the item scores) are violated, this is a conservative
reliability estimate(10). However, reliability estimates for
classroom examinations are typically much lower. A table of
specifications is constructed for each exam in order to assess
content validity(11).
FUTURE PLANS
We continue to evaluate and refine the course based on
instructor, teaching assistant, and student input. Summative
instructor and course evaluations are conducted each time
the course is offered using standardized forms. Students
report that they use the course notes as a reference through-
out the remainder of the program and in some cases, at their
residency or practice site. We continuously gather and inter-
pret quantitative data on student, instructor, and course
performance. As suggested by Miles and Huberman, we
find qualitative information serves as an additional rich
source to validate and reinterpret the quantitative find-
ings (12). In the future we hope to have the students manipu-
late data sets using statistical software programs.
American Journal of Pharmaceutical Education Vol. 62, Spring 1998
Am. J. Pharm. Educ., 62, 17-23(1998);received 9/16/97, accepted 1/15/98.
References
(1) Draugalis, J.R. and Slack, M.K., “A course model for teaching re-
search evaluation in colleges of pharmacy,” Am. J. Pharm. Educ., 56,
48-52(1992).
(2) Educational Outcomes, Center for the Advancement of Pharmaceu-
tical Education Advisory Panel, American Association of Colleges of
Pharmacy, Alexandria VA (1994).
(3) Accreditation Standards and Guidelines for the Professional Pro-
gram in Pharmacy Leading to the Doctor of Pharmacy Degree,
Adopted June 14,1997, American Council on Pharmaceutical Educa-
tion, Chicago IL (1997).
(4) Boyer, E.L., Scholarship Reconsidered - Priorities of the Professoriate.
The Carnegie Foundation for the Advancement of Teaching, Princeton
NJ (1990).
(5) Campbell, D.T. and Stanley, J.C., Experimental and Quasi-Experi-
mental Designs for Research. Houghton Mifflin Company, Boston
MA (1963).
(6) Merrill, M.D. and Tennyson, R.D., Teaching Concepts: An Instruc-
tional Design Guide. Educational Technology Publications, Englewood
Cliffs NJ (1977).
(7) Cook, T.D. and Campbell, D.T., Quasi-Experimentation-Design &
Analysis Issues for Field Settings, Houghton Mifflin Company, Boston
MA (1979), pp. 37-39.
(8) Salant P. and Dillman D.A., How to Conduct Your Own Survey, John
Wiley and Sons, New York NY (1994).
(9) Harrison, D.L. and Draugalis, J.R., “Evaluating the results of mail
survey research,” J. Am. Pharm. Assoc., 37, 662-666(1997).
(10) Speedie, S.M., “Reliability: The accuracy of a test,” Am. J. Pharm.
Educ., 49,76-79(1985).
(11) Kimberlin, C.L. “Characteristics desired in tests: Validity,” ibid., 49,
73-76(1985).
(12) Miles, M.B. and Huberman, A.M., Qualitative Data Analysis: An
Expanded Sourcebook, 2nd Ed., Sage Publications, Inc., Thousand
Oaks CA (1994) p.10.
23