Research Designs

Gerardo L. Munck
University of Southern California, Los Angeles, California, USA

Jay Verkuilen
University of Illinois, Champaign-Urbana, Champaign,
Illinois, USA

Glossary
case study A study in which one unit is analyzed, typically in
an intensive manner that is attentive to time and process.
Although, strictly speaking, in a case study the N 1,
frequently the effective number of observations is considerably higher.
cross-sectional design A study in which observations on
a variable or multiple variables are collected across units at
the same point in time.
experimental design A study in which the treatment is
consciously manipulated by the researcher and in which
units are randomly assigned to treatment and control groups.
To distinguish experimental from quasi-experimental
designs, the former are sometimes called randomized
experiments.
external validity Concept originally introduced by Donald T.
Campbell to refer to the generalizability of the finding of
a causal relationship between variables beyond the domain
of the actual units, spatial and temporal setting, and specific
treatments that are examined.
internal validity Concept originally introduced by Donald T.
Campbell to refer to a causal relationship between variables
in the actual units, spatial and temporal setting, and specific
treatments that are examined.
large N study A study in which observations are made across
a large number of units. Such studies, however, vary
significantly in terms of their N, with typical cross-national
studies in the field of comparative politics and international
relations oscillating between 30 and 100 and those using
opinion surveys reaching into the thousands.
longitudinal design A study in which multiple observations
on variables are collected across time for the same unit.
Also known as time series design.

observational studies Nonexperimental studies, also called

correlational studies, in which the treatment is not
consciously manipulated by the researcher. Rather,
researchers simply record the values of variables as they
naturally occur. These studies include natural experiments
in which particularly sharp and obvious changes in the
value of a variable are held to offer an analogy to the
introduction of a treatment.
pooled time series, cross-sectional design A design that
combines a cross-sectional design and a longitudinal design;
includes panel studies and repeated measures design.
quasi-experimental design Design in which the treatment is
consciously manipulated by the researcher, as in an
experiment, but in which, unlike in experiments, units are
not randomly assigned to treatment and control groups.
research design A key aspect of the research process that
revolves around the direct impact on the prospects of causal
inference of four core questions: How is the value of the
independent variable(s) assigned? How are units selected?
How many units are selected? and How are comparisons
organized (i.e., whether temporally and/or spatially)? In
addition, research designs can be evaluated in terms of
their indirect impact on causal inference in light of their
requirements and contributions vis-a`-vis theory and data.
small N study A study in which observations are made across
a small number of units. Typically, each unit is treated
as a case study so that multiple observations on each unit
are made.

The methodology of research design hinges on the choices

made with regard to four core questions: How is the value
of the independent variable(s) assigned? How are units

Encyclopedia of Social Measurement, Volume 3 2005, Elsevier Inc. All Rights Reserved.

385

386

Research Designs

selected? How many units are selected? and How are

comparisons organized (i.e., whether temporally and/or
spatially)? These choices can be assessed in terms of their
direct impact but also their indirect impactdue their
requirements and contributions vis-a`-vis theory and
dataon the prospects of making causal inferences.
Three research traditionsexperimental and quasi-experimental, quantitative, and qualitativerepresent
distinct responses to these methodological choices and
each has important strengths but also significant
weaknesses. Thus, the need for choices about research
design to be explicitly addressed and justified, and the
need to actively construct bridges across research traditions, is emphasized.

Introduction: Goals, Problems,

and Options
The pioneering work on research design by Donald T.
Campbell and associates has made such a major contribution that it is difficult to think about research design
without, in one way or another, drawing on their insightful
discussions. They provide a valuable template for thinking
about research design, helpfully framing the discussion in
terms of the basic goal of validity, a range of problems or
threats to validity, and a set of options or design choices
that can be pursued as a way to guard against these threats.
At the same time, their work displays some limitations and
biases and ultimately fails to offer a clear, encompassing,
and balanced understanding of the challenges involved
in research design. This assessment and comparison of
research designs thus adopts Campbell et al.s basic
template and some of their key ideas about the goals,
problems, and options of research design but also parts
company with them in some significant ways.
First, Campbell et al.s discussion of the goals of research design in terms of the concept of validity is both
somewhat confusing and biased. Initially, Campbell introduced the concepts of internal and external validity, which
disaggregated the ultimate goal of research designto
increase the prospects of making causal inferences
and aptly distinguished different problems of causal
inference that are affected and potentially solved by different design choices. Over time, however, these two
concepts have been awkwardly relabeled and, more important, defined in different ways in different texts. In
addition, two other types of validitystatistical conclusion
validity and construct validitythat pertain in part to research design but spill over into questions of data
analysis and measurement, respectively, have been introduced into the discussion, further complicating matters.
Another problem with this typology of validity is that it is
somewhat biased. Thus, Lee Cronbach has argued that

Campbell et al. gave undue primacy to internal over

external validity and did not recognize the importance
of generalizability. This is a critique that they have acknowledged and sought to address in their latest statement.
However, they still do not fully incorporate the difficulties
of generalizing on the basis of experimental and quasiexperimental designs in their overall assessment of the
strengths and weaknesses of design options.
To avoid these problems, we only consider internal and
external validity to be the core goals directly relevant to
a discussion of research design, and both retain the classic
labels of internal and external validity and follow the original definitions offered by Campbell and Julian Stanley in
1966. As they argue, the establishment of the internal
validity of a causal proposition involves showing that
a factor is the cause of an effect or, more modestly, probing alternative hypotheses and opting for those that stand
up best to attempts at disconfirmation. In contrast, the
verification of the external validity of a causal proposition
entails demonstrating that a causal proposition can be
generalized beyond the domain of the actual units, spatial
and temporal setting, and specific treatments that are
actually studied. In turn, both internal and external
validity can be viewed as distinct and equally essential
goals of research design.
Second, Campbell et al.s discussion of the problems of
causal inference is organized around a fairly ad hoc and
cumbersome list of threats to validity. To be sure, the
listed threats to validity are all important, and their analysis of the way different designs get around or fail to get
around these threats to validity is exemplary. Indeed, they
offer many specific recommendations that creatively respond to thorny and complex problems routinely encountered in the conduct of substantive research. However,
they do little to present their list of threats in a logically
explicit manner, to distinguish threats that are relevant
to experimental designs from those that pertain to nonexperimental designs, and to offer a sense of the extent
to which the threats to validity they discuss constitute a
complete list. In place of their list, we propose a scheme
whereby research design choices are evaluated in light
of a set of problems of causal inference that are directly
and indirectly affected by design choices.
Figures 1 and 2 provide a graphic representation of
the assumptions and potential problems of causal inference that design options impact in a direct manner. With
regard to the analysis of single units, the core concerns
are the need to ascertain that the posited direction of
causality is correctly described and that the proposed
model of the link among causal variables is fully and
correctly specified. With regard to the analysis of multiple
units, the key issues are the need to validate the assumptions of unit independence and unit homogeneity. In turn,
Fig. 3 draws attention to the need to place the discussion
of research design in the broader context of the research

387

Research Designs
Assumption

Potential problem

Assumption

Potential problem

Direction of causality

Reverse causation

Independence of units, i.e. units

are not affected by each other

Diffusion across units

(Galtons problem)

Unit 1 at time 1 Unit 1 at time 2

Unit 1 at time 1

IV

DV

IV

DV

Endogeneity (e.g., reciprocal,

nonrecursive causation)
IV/DV

DV/IV

DV

DV
IV

IV

ii

DV

iii

iv

All variables affecting the DV

are included in the model and
are correctly specified as being:
i: an antecedent variable,
ii: an interacting variable,
iii: an intervening variable, or
iv: another independent
variables

Omitted variable: failure to

identify any variable that
affects the DV
Misspecified variable: failure to
correctly specify the link
between all identified
variables (e.g., an interacting
variable is modeled as another
independent variable)
DV, dependent variable; IV,
independent variable.

DV

Units 2, 3, etc.
at time 1
IV

Spuriousness

Fully and correctly specified

model of the link among causal
variables

IV

DV

IV

DV

Units 2, 3, etc.
at time 2
IV

IV

DV

Units 2, 3, etc. at time 1

IV

DV

DV

DV, dependent variable; IV,

independent variable.

Diffusion within units across

time (the problem of history)
Unit 1 at time 1 Unit 1 at time 2
IV

DV

IV

DV

Unit homogeneity, i.e. changes

in the dependent variable are
produced by same causal model

Heterogeneity of units within a

sample or a population

Unit 1

Unit 1

IV1

DV

IV2

IV1

DV

IV2

Units 2, 3, etc.
IV1

Units 2, 3, etc.
DV

IV2

IV3

DV

IV4

Figure 2 Problems of causal inference: Issues in the analysis

of multiple units and singular units across time.

Figure 1 Problems of causal inference: Issues in the analysis

of single units.

process and to consider how design options have an indirect but important impact on causal inference as a result
of their requirements and contributions vis-a`-vis theory
and data.
Third, Campbell et al.s discussion of design options is
fairly limited and their assessment biased toward experimental designs. Indeed, they place such strong emphasis
on certain design optionsthe ability of researcher to
consciously manipulate independent variables and randomly assign units to treatment and control groupsthat
they offer a narrow and unbalanced optic on questions of
research design. They do highlight the difficulties of
conducting experiments and emphasize the ways in
which experiments are likely to never guarantee that
all threats to validity are eliminated. Moreover, occasionally they offer an exemplary display of pluralism.
However, they tend to overlook some significant shortcomings associated with experimental data and downplay
the significant potential virtues of nonexperimental data,
and they ignore both the role played by design choices
other than those that are defining elements of experiments and the way in which design choices have an indirect impact on causal inference. Thus, to assess the

Theory

Research
design

Causal inference

Data

Figure 3 Research design in context.

Although research is frequently portrayed as proceeding in
a linear fashion from theory to data collection and causal inference, as a matter of practice research is an interactive and iterative
process. Thus, it is important to recognize that design options are
(i) affected by the state of available theory and can also affect
theory building and (ii) limited by the availability of data and also
have an impact on the quality of data that are generated.

strengths and weaknesses of different research designs

in a systematic and balanced manner, it is important to
recognize that design options revolve around at least four
core questions: How is the value of the independent
variable(s) assigned? How are units selected? How

388
Table I

Research Designs
Research Designs: Classification Criteria and Options

Classification criteria
How is the value of the independent
variable(s) assigned?

Main options

Disaggregate designs

Manipulation, random assignation of cases to

treatment and control groups
Manipulation, nonrandom assignation of cases
to treatment and control groups
Nature

Experimental (randomized experiment)

Quasi-experimental
Observational

How are units selected?

Random sample
Purposive (deliberate or intentional) sample
Entire population

Representative
Typical (mode, mean, or median)
Heterogeneous (extreme and typical)
Census

How many units are selected?

Many
Few
One

Large-N
Small-N
Case study

How are comparisons organized?

Across units
Across time

Cross-sectional
Longitudinal

many units are selected? and How are comparisons organized (i.e., whether temporally and/or spatially) (Table I)?
Moreover, it is necessary to address the manner in which
design choices have both a direct and an indirect impact
on causal inference.
Recognizing the seminal nature of the work by
Campbell et al. but also seeking to overcome its limitations,
this article provides an overview of the current state of
knowledge on research design. The number of alternative
research designs is, in principle, very large. Indeed, focusing only on the main options related to the four criteria
presented in Table I, it is clear that the range of possible
designs is much greater than usually believed, although not
all will be sensible. However, the following discussion is
organized in terms of research traditions, given that this is
how much research and discussion about design options
is carried out, focusing on experimental and quasiexperimental designs first, turning next to large N or
quantitative designs, and completing the discussion with
case study and small N or qualitative designs. In each case,
the focus is on the prototypical designs associated with
each tradition and an assessment and comparison of the
strengths and weaknesses of these designs. The need for
choices about research design to be explicitly addressed
and justified, and the need to actively construct bridges
across research traditions, is emphasized.

Experimental and QuasiExperimental Designs

Experimental designs, also called randomized experiments, are characterized by two distinguishing features:
(i) the conscious manipulation by the researcher of
a treatment or, more generically, an independent variable
of interest, and (ii) the random assignment of units to

treatment and control groups (Fig. 4). Experiments

also draw upon other design elements, and thus it is possible to distinguish a variety of experimental designs.
However, these two features are the defining features
of the prototypical experimental design.
The strengths of this design are undeniable. Given that
the treatment is administered before a posttest measure
on the dependent variable is registered, the direction of
causality is clearly established. In addition, the random
assignment of units to treatment and control groups
ensures that these groups are equivalent; that is, they
do not vary systematically on any variable except the manipulated variable. In this manner, multiple unexamined
variables are held to not vary in any patterned manner
and, hence, control over these alternative hypotheses is
ensured by turning other variation into noise. Thus, the
beauty of experimental designs is that the data generated
lend themselves to a particularly simple form of analysis,
in which the differences in posttest measures of the treatment and control groups can be interpreted as a measure
of the causal effect of the treatment. Indeed, experiments
are the most powerful means of gaining control and establishing the internal validity of a causal claim (Table II).
A quasi-experimental design differs from an experimental design in that in the former the treatment is consciously manipulated by the researcher but the units are
not randomly assigned to treatment and control groups.
Thus, this design offers less of a basis for assuming the
initial equivalence of treatment and control groups and
requires researchers to consider how third extraneous
variables might confound efforts at ascertaining causal
effects and to be explicit about alternative hypotheses.
The difficulties this difference between experiments
and quasi-experiments introduces are central to the
work of Campbell et al. Indeed, their work can be understood essentially as an effort to alert researchers to

Research Designs

389

The units are randomly divided into two groupsthe treatment group and the control groupand a treatment is applied to the
treatment group and not to the control group.
Measures are taken of the values of the dependent or outcome variable for the treatment and control groups. The generated data are
organized as follows:
Treatment group
Control group
Units 1, 2, 3, . . . , n
Units 1, 2, 3, . . . , n
Independent
Yes
No
variable
(treatment)
Dependent
Value
Value
variable
____________________________
Alternatively, after the units have been randomly divided into two groups, a pretest measure of the value of the dependent variable
for both the treatment group and the control group is made. The generated data are organized as follows:
Treatment group
Control group
Units 1, 2, 3, . . . , n
Units 1, 2, 3, . . . , n
T1
T2
T1
T2
Independent
No
Yes
No
No
variable
(treatment)
Dependent
Value
Value
Value
Value
variable
____________________________
In both instances, the analysis of the data focuses on the difference in values on the dependent variable between treatment and
control groups (the italicized values), a difference that can be interpreted as a measure of the causal effect of the treatment.
When a pretest measure on the dependent variable is obtained in addition to a posttest measure, a comparison of the values of units
of the treatment and control groups at T1 (the underlined values) serves to double-check that all units are equivalent, given that
this goal is already ensured by the random assignment of units.
Figure 4 Experimental designs: Defining features.
Table II Experimental Designs: An Assessment
Design elements

Strengths

Conscious manipulation of
independent variables

Establishes causal direction

(internal validity)

Random assignment of units to

treatment and control groups

Establishes equivalence of
units, helps guard against
third, extraneous variables
(internal validity)

Nonrandom selection of units,

setting, and treatment

the various ways in which factors other than the treatment

may be responsible for the observed posttest variance
between treatment and control groups. Thus, in large
part due to the significant effort by Campbell et al.
to anticipate potential threats to the validity of quasiexperimentsand to offer a range of designs that minimize

Weaknesses
Lack of viability, due to practical and/or ethical reasons, to
study many important questions
Unsuitable for the study of action and lack of attention to
causal mechanisms (internal validity)
Unsuitable for an assessment of complex causes
Requires a priori knowledge of plausible independent
variables and well-specified causal model; not useful in
theory generation
Tends to generate obtrusive, reactive measurements
Lack of viability, due to practical and/or ethical reasons, to
study many important questions

Difficult to generalize from sample to population (external

validity)

and help guard against such threatsresearchers have

a sophisticated road map for consciously taking into consideration how third extraneous variables may exercise
a confounding effect. In summary, even though quasiexperiments are more complicated than experiments,
both designs offer a powerful basis for making claims

390

Research Designs

about causality, and it is thus hardly surprising that these

designs have been used often in psychology and economics and are increasingly being adopted by political
scientists.
The considerable strengths of experimental and quasiexperimental designs notwithstanding, it is important to
recognize the serious weaknesses associated with these
designs. One standard limitation concerns the viability
of conducting experiments, whether for practical and/or
ethical reasons, to study the sort of questions that are of
interest to many social scientists. The fact that we cannot
or would not want to manipulate some variables and/or
randomly assign subjects to control and treatment groups
has vast implications. Indeed, as Hubert Blalock (1991)
argues, If we were to confine our analyses to experimental
and quasi-experimental designs, virtually all of sociology
and political science would have to go to the wayside.
(p. 332) However, the problem is deeper than the standard
concern with viability would indicate and actually derives
from the very substance of the subject matter of the social
sciences.
The core difference between the social and natural
sciences is that the former are first and foremost about
agents and actions. However, a basic feature of experimental and quasi-experimental designsthe conscious
manipulation of the treatmentis founded on a notable
asymmetry, which places the experimenter in an active
role and relegates the experimentee to a passive, reactive
role. In effect, experiments and quasi-experiments treat
subjects as objects and embody an implicit behaviorist
perspective that renders them ineffectual instruments
for the study of the causal significance of action and, relatedly, forces them to be silent on the critical question of
causal mechanisms. Indeed, as William Shadish et al.
recognize, although experimental and quasi-experimental
designs can be used to predict what the effect of a factor is,
they are less useful for explaining why and how such
effects are generated. This is a significant limitation. Indeed, it is not far-fetched to argue that the internal validity
of a causal argument is not fully established until the
causal mechanisms that generate the causal effect are
properly identified and tested.
Experiments are also of limited use with regard to the
study of complex causal relationships. This failure is in
part due to the inability to deal squarely with agency
through experiments and quasi-experiments. Consequently, these designs are not suitable means for getting
at a core theoretical concern in social theory: the interaction between structures and agents, or macro and micro
causal factors. In addition, because experiments and
quasi-experiments are in essence instruments geared to
the study of short-term effects, they are not useful for
studying causes that work themselves out over an extended period of time or for assessing the interaction
between long- and short-term causal factors. Moreover,

because experimenters, as a way to manipulate the

treatment, must rigidly assign variables to the status of
independent and dependent variables, they do not
constitute a means of assessing feedback effects or reciprocal and nonrecursive causation. In short, a range of
theories simply cannot be tested through experimental
and quasi-experimental designs.
Another significant limitation concerns the generalizability of results derived from experiments. The reason for
this is that although a defining feature of experiments is
that units are randomly assigned to treatment and control
groups; the units, settings, and treatments are usually not
randomly selected. Indeed, inasmuch as control is gained
through the manipulation of the treatment and the random assignment of units to treatment and control groups,
it is practically inevitable that the ability to randomly select these units will decline. The purposive or intentional
selection of units that researchers have to use is not without merits, and it can certainly be carried out with an eye
to the relationship between the studied sample and the
universe. However, even when carefully practiced, purposive selection tends to lead to biased results. Indeed, in
many instances the gap between the conditions of experimental research (whether in the laboratory or in field
settings) and the phenomenon being studied can be substantial; hence, the ability to generalize beyond the domain of the actual units, spatial and temporal setting, and
specific treatments that are examined is compromised.
Thus, it is important to recognize that both internal
and external validity are critical aspects of knowledge,
and that there are good reasons for the standard view
that the gains made by experiments in terms of internal
validity tend to come at the cost of a loss in external
validity.
Finally, two other limitations are indirect consequences of the manipulation of treatments that characterizes experimental and quasi-experimental designs.
First, because of this design element, experiments and
quasi-experiments approach the two-sided issue of causal
theorizing from one side: They focus on the effect of
causes rather than on the causes of effects. Thus, they
require a priori knowledge about plausible independent
variables and presume that all that needs to be determined is the effect of preselected causes. Hence, experiments and quasi-experiments are of less use during the
early, exploratory stage in the research process, when
a typical challenge is to uncover potential independent
variables by working backward from a dependent variable. Furthermore, experiments and quasi-experiments
assume a well-specified causal model and thus require
that the state of theory building already be fairly advanced. Second, the manipulation of treatments makes
experiments and quasi-experiments a particularly obtrusive and reactive form of generating data. The gain made
by manipulating treatments comes at the cost of the

Research Designs

quality of the data generated for analysis. In summary, as

shown in Table II, although experiments especially, but
also quasi-experiments, have some important strengths,
they are also associated with a number of significant
weaknesses.

Observational Designs
The distinction between experimental and observational
studies is a fundamental and deep one. The key difference
is that in observational studies, control of possible third
variables is not attained automatically through random
assignment. Rather, in observational studies third
variables have to be formulated and measured explicitly,
and control is sought through the analysis of the data.
However, the distinction between large N studies, on
the one hand, and case and small N studies, on the
other hand, is equally profound and probably more pervasive. This second distinction is not unique to observational studies. Indeed, the quantitative vs qualitative
distinction runs through both the experimental and the
observational research communities. However, the discussion of this distinction is developed only in the context
of observational studies and focuses primarily on the prototypical quantitative and qualitative studies: the large N,
cross-sectional study and the small N study based on the
longitudinal case study, respectively.

Large N Studies
A large N study has some considerable strengths that
make it, in some regards, superior to an experimental
study. First, because it uses data generated through the
natural course of events, it is a viable design for studying

Table III

391

important questions that involve nonmanipulable

variables that cannot be addressed with an experimental
method. Second, because a large N study is not constrained by the requirement to randomly assign units to
treatment and control groups, it is more likely to entail
a randomly selected sample. This is a major benefit that
gives large N researchers the ability to generalize beyond
the domain of the actual units, spatial and temporal setting, and specific treatments that are actually studied and
to establish the generalizability of their findings (external
validity), a core weakness of experimental methods.
However, much as is the case with regard to random
assignment in the context of experimental designs, the
beauty of random selection is tarnished by the difficulty
of applying this design element to many units, settings,
and variables (Table III).
Another important strength of large N studies, due to
their tendency to study quite large samples, is their ability
to use statistical analysis to establish patterns of association with a high degree of precision and confidence. Such
results, however, differ significantly from those that can
be obtained using experimental data. Indeed, although
experimental data offer strong grounds for making claims
about causality, because large N studies are observational
it is crucial to remember the simple but profound point
that association is not causation and, moreover, that
even a lack of correlation does not disprove causation
(Bollen, 1989: p. 52). This does not mean that claims about
causality cannot be made on the basis of large N studies.
Indeed, such claims can be made legitimately if researchers verify the causal assumptions of their statistical
models, including nonspuriousness, the lack of omitted
variables, independence of cases, and unit homogeneity.
However, it is extremely difficult to substantiate that patterns of association establish causality (internal validity).

Observational Designs: Large-N Studies

Design elements
Assignment of value of the
independent variable(s) by nature
Random selection of units or selection
of an entire population
Selection of many units that are
compared cross-sectionally and/or
longitudinally

Strengths
Viability of studying important questions
that involve nonmanipulable variables
Establishes generalizability (external
validity)
Establishes patterns of association with
a high degree of precision and
confidence
Constitutes a tool for theory generation

Weaknesses

Lack of viability for many units, settings, and

variables
Association does not establish causation
(internal validity) and it is difficult to verify
the causal assumptions in statistical models
Associations are more interpretable when
guided by a strong theory, i.e., a theory
with few variables and detailed predictions
Measurement validity is harder to establish
the larger the N and the larger the number
of variables
Lack of attention to causal mechanisms
(internal validity)

392

Research Designs

In short, claims about causality derived from large N

studies should be treated with great caution.
Regarding the indirect consequences of design
choices, it bears emphasizing that large N studies are
quite useful for exploratory work. In an experiment, treatments need to be planned in advance and need to be
relatively few in number, a particularly difficult and restrictive requirement for many areas of the social science.
In contrast, at least with regard to the kinds of questions
that are common to macro-oriented inquiries in sociology,
political science, and economics, it is possible to obtain
more data after examining some relationships. This practice runs the risk of capitalizing on chance if not rigorously
tested on new data, but it is potentially an important part
of any study.
This benefit notwithstanding, a core problem regarding the demands that large N research puts on theory and
data should be highlighted. On the one hand, associations
are more interpretable when guided by a strong theory
that is, one with few variables and detailed predictions. In
other words, good large N research not only requires
a causal model specified prior to testing but also puts
a heavy burden on the ability of theory builders to reduce
the number of potential explanatory factors. On the other
hand, to ensure that the causal model is fully specified, it is
important that potential independent variables are not
omitted. This fact complicates the interpretation of
results. Moreover, it makes a heavy, sometimes practically
impossible, demand concerning data. Indeed, because
potentially confounding factors are not controlled automatically through random assignment in large N studies
but rather must be modeled and measured explicitly, the
need to collect data on numerous variables across a large
number of cases and/or time entails some serious costs.
This demand opens the door to well-founded charges
concerning the validity of the data. In addition, the
data requirements of large N designs make it extremely
difficult to offer a quantitative study of causal mechanisms, a crucial shortcoming that further weakens the
claims about causality (internal validity) that can be
made on the basis of a large N study. Gains in terms of
generalizability and findings about patterns of association
are heavily dependent on good theory and good data
valuable resources that are not always available.
Some advances that go beyond the cross-sectional design typically used in large N studies help get around some
of these limitations. Especially noteworthy are time series
and panel studies, event history analysis, and hierarchical
modeling. These methods offer fruitful ways of addressing
the problems associated with the assumptions of unit independence and unit homogeneity. However, these potential gains tend to make even more imposing demands in the
area of data collection than cross-sectional designs and thus
are achieved at a cost. In summary, as Table III highlights,
much as in the case of experimental and quasi-experimental

designs, it is only fair to note the strengths of quantitative

or large N designs but also to recognize their weaknesses.

Case and Small N Studies

The status of the case and small N studies tradition in the
social sciences has been marked by a notable disjuncture
between the high number of its practitioners and its low
standing in the broader methodological community. This
odd situation, however, has been changing over time.
After famously stating that one-shot case studies [are]
of almost no scientific value, Campbell retracted this
harsh critique of a staple of qualitative research. In
turn, statistician David Freedman has argued that case
studies, when well designed, can establish causality in
a more powerful manner than is standard in most quantitative research. Finally, a flourishing debate on case and
small N research in recent years has increasingly made
explicit the rationale for choosing a qualitative research
design and the methodological foundations of rigorous
qualitative research. As a result, the reconstructed logic
of qualitative methods is catching up with the logic of
qualitative research, and a clear sense of the strengths
and weaknesses of this tradition is emerging.
One well-established and important strength of case
and small N studies is that they represent a viable design to
address important questions that involve nonmanipulable
variables. This is a virtue shared by small and large N
studies, but in this regard small N researchers have an
advantage, particularly in light of their ability to gain access to various types of data and form a complex picture of
their cases, including a keen sense of developments over
time. For this reason, much of the analysis in the social
sciences on a range of critical questions, especially during
the initial stages in the research process, is done by qualitative researchers (Table IV).
A feature of case and small N studies setting them apart
from both large N and experimental studies and accounting for a key comparative advantage is the manner in
which these studies can be used to analyze the role of
agency and hence to establish causal mechanisms. This
is a critical point. Indeed, as philosophers of science and
methodologists have increasingly insisted, it is not enough
to focus on causal effects. Rather, it is necessary to go
beyond statements about what the effect of a factor is
and to consider how and why the effect operates. This
calls for the specification of causal mechanisms, which
requires, in most areas of the social sciences, considering
agency. In this regard, it is probably not an overstatement
to suggest that a qualitative design is the method par excellence to study agency and hence to empirically ground
the analysis of causal mechanisms.
A related strength of case and small N studies is that
they offer a basis for assessing causality (internal validity).

Research Designs
Table IV

393

Observational Designs: Case and Small-N Studies

Design elements
Assignment of value of the
independent variable(s) by nature
Selection of one or a few units that are
compared longitudinally and/or
cross-sectionally

Strengths

Weaknesses

Viability of studying important

questions that involve
nonmanipulable variables
Addresses agency and establishes
causal mechanisms

In the absence of strong theory, it is difficult

to establish control and eliminate potential
variables (internal validity)

The study of causal mechanisms, and

within- and cross-case analysis,
offers a basis for assessing causality
(internal validity)
Constitutes a powerful tool for theory
generation
Measurement validity is easier to
establish the smaller the N
Purposive selection of a few units

This is done, most fundamentally, through a within-case

form of analysis that uses empirical evidence about causal
mechanisms as a way to check expectations concerning
the direction of causal processes, to eliminate potential
third variables, and to verify the assumption of unit independence and unit homogeneity. Moreover, this goal is
also frequently advanced by combining the prototypical
longitudinal within-case design with a cross-sectional design, such as a traditional cross-case study or a cross-sectional within-case study either focused on different
implications of a theory or cast at a different level of
aggregation. Through these means, which capitalize on
knowledge about process and the ways in which case
studies lend themselves to more observations than are
suggested by the strict definition of a case study as an
N 1 study, small N researchers can make valuable contributions that are important to highlight.
However, the weaknesses of qualitative methods as
a tool for establishing causality should also be duly recognized. To use case studies to assess causality, it is necessary
to be explicit about the posited causal modelincluding all
its variables, the relationship among the variables, and the
form of the effect of each variable or combination of
variablesas well as about the causal mechanisms that
are considered to be in play. This is a demanding task,
but failing to specify these things in advance makes it
easy for researchers to simply focus on confirming evidence and to disregard alternative interpretations. Indeed,
a drawback of most analyses of causal mechanisms is that
they fail to specify formally what mechanisms are posited
and what plausible alternative mechanisms should be considered and also to set up the study of mechanisms as
a standard test among competing hypotheses.
Moreover, even if such steps are taken, it is necessary to
recognize the limits of small N analysis, in light of the
number of observations they entail, as a tool for causal

Difficult to generalize from sample to

population (external validity)

assessment. Some important exceptions to this general

principle exist. First, occasionally qualitative researchers
may be able to design their research in a way that resembles a natural experiment, in which the hypothesized
causal factor changes markedly while other factors remain
the same. Second, it is possible to establish patterns of
association with precision even with a relatively small
sample using techniques of analysis such as exact tests,
permutation tests, resampling, or Bayesian methods that
do not rely on asymptotics.
Third, presuming a powerful theory is being tested,
a few observations could very well serve as the basis for
clear results. However, frequently qualitative researchers
seek to assess complex causal relationships and, when this
is the case, small N designs tend to constitute weak means
of controlling for alternative hypotheses and for ruling out
chance. Indeed, claims to test complex causes using small
N designs rely on the quite unreasonable assumptions that
(i) the world operates in a deterministic fashion, (ii) the
proposed causal model is complete, and (iii) there is no
measurement error. In short, the use of case studies and
small N studies to assess causality tends to rely on some
very stringent assumptions concerning the state of theory
and data.
Another significant limitation concerns the difficulty of
using case studies to make generalizations. When studying a small number of cases, random selection is not
advisable. Indeed, random selection offers a basis for
generalization only when a large number of cases are
selected and hence the law of large numbers takes
force. Thus, qualitative researchers have to resort to
the purposive selection of their cases. When this is
done, it is important to avoid drawing a sample of convenience that bears an unclear relationship to the broader
population. Moreover, it is important that researchers be
aware of how their choice of cases might introduce bias.

394

Research Designs

Such efforts to select cases in a conscious and careful

manner, however, should not be mistaken as steps providing a sufficient basis for making claims about generalizations (external validity).
Finally, with regard to the indirect consequences of
design choices, two strengths of case and small N studies
deserve mention. One is that this type of design constitutes a powerful tool for theory generation. Indeed, one of
the clear benefits of qualitative research is its fruitfulness
at a tool for generating ideas about causal variables and
mechanisms and for theorizing that is closely informed by
knowledge of the substantive problem of interest. Another related benefit is that the detailed knowledge of
context that is associated with case studies plays
a critical role in helping researchers establish measurement validity. In summary, as is the case with the experimental and quantitative traditions, the qualitative
tradition is characterized by a mix of strengths and
weaknesses (Table IV) that must be considered in any
balanced assessment of the potentials of different
research designs.

Conclusion: Choices and Bridges

The broad message concerning research design we have
sought to convey is that making causal inferences about
the complex realities of interest to social scientists is probably more difficult than is generally believed and that
questions of research design play a key role in determining
whether researchers have a solid basis for making claims
about causal relationships. Technical fixes cannot, in general, get around design problems, but more attention goes
to the former than the latter. Indeed, given the centrality
of research design to the research process as a whole, it is
probably fair to say that research design is a relatively
unappreciated aspect of methodology and that it deserves
more attention from methodologists and practicing researchers alike. Beyond this generic admonition, two further points that build on but go beyond the previous
discussion offer material for further consideration.
One point is the need for choices about research design
to be explicitly addressed and justified. There is a tendency
for researchers to work within distinct research traditions
and to simply opt for certain designs as a matter of default.
Such a tendency is understandable in that different designs require different skills and training, and in a practical
sense researchers are thus not free to choose among research designs. Nonetheless, short of justifying their design choices in light of the range of possible options, at the
very least researchers should address the impact of their
choices on the certainty of their conclusions. As this article
shows, this entails a consideration of the direct impact on
the prospects of causal inference of the four core choices
involved in research design and of the indirect impact,

due to their requirements and contributions vis-a`-vis theory and data, of these choices.
A second point is the need to creatively construct
bridges across research traditions. Although it is common
for certain traditions to be presented as inherently superior to others and the standard against which other traditions should be measured, this article has shown that all
traditions are characterized by certain strengths and
weaknesses and that it is thus more accurate and useful
to think in terms of the tradeoffs involved in working
within different traditions. An implication of this assessment, then, is that greater effort should be made to capitalize on what are clearly the complementary strengths of
different traditions (compare Tables II
IV).
Efforts at bridging, whether carried out through multiple studies on the same question or mixed designs that
combine multiple designs within a single study, are very
demanding. Thus, although it is common to point out that
multiple studies in the context of a shared research program offer a way of combining different designs, such
combinations are only effective inasmuch as research
programs are organized around clearly specified concepts
and questions and are advanced, at least to a certain extent, through explicitly coordinated teamwork. In turn,
the effective use of mixed designs requires a level of
methodological sophistication, as well as theoretical and
substantive knowledge, that is rare. Nonetheless, the high
payoffs associated with the use of mixed methods make
these options strongly recommendable.

See Also the Following Articles

Experiments, Overview  Explore, Explain, Design 
Longitudinal Cohort Designs  Observational Studies 
Quasi-Experiment  Sample Design  Survey Design 
Time-Series
Cross-Section Data  Validity, Data Sources

Further Reading
Blalock, H. (1991). Are there really any constructive alternatives to causal modeling? Sociol. Methodol. 21, 325
335.
Bollen, K. A. (1989). Structural Equations with Latent
Variables. Wiley, New York.
Brady, H. E., and Collier, D. (eds.) (2004). Rethinking Social
Inquiry: Diverse Tools, Shared Standards. Rowman &
Littlefield/Berkeley Public Policy Press, Lanham, MD.
Campbell, D. T. (1988a). Factors relevant to the validity of
experiments in social settings. In Methodology and
Epistemology for Social Science: Selected Papers. Donald
T. Campbell (E. Samuel Overman, ed.), pp. 151
166.
University of Chicago Press, Chicago [Original work
published 1957].
Campbell, D. T. (1988b). Degrees of freedom and the case
study. In Methodology and Epistemology for Social Science:
Selected Papers. Donald T. Campbell (E. Samuel Overman,

Research Designs
ed.), pp. 377
388. University of Chicago Press, Chicago
[Original work published 1975].
Campbell, D. T. (1999). Relabeling internal and external validity for applied social scientists. In Social Experimentation
(D. T. Campbell and M. J. Russo, eds.), pp. 111
122. Sage,
Thousand Oaks, CA [Original work published 1986].
Collier, D. (1993). The comparative method. In Political
Science: The State of the Discipline II (A. W. Finifter, ed.),
pp. 105
119. American Political Science Association,
Washington, DC.
Cronbach, L. J. (1982). Designing Evaluations of Educational
and Social Programs. Jossey-Bass, San Francisco.
Freedman, D. A. (1991). Statistical analysis and shoe leather.
Sociol. Methodol. 21, 291
313.
Goldthorpe, J. H. (2000). On Sociology: Numbers, Narratives,
and the Integration of Research and Theory. Oxford
University Press, Oxford, UK.
Good, P. I. (2000). Permutation Tests: A Practical Guide to
Resampling Methods for Testing Hypotheses. Springer,
Berlin.
Kagel, J. H., and Roth, A. E. (eds.) (1995). The Handbook
of Experimental Economics. Princeton University Press,
Princeton, NJ.

395

King, G., Keohane, R. O., and Verba, S. (1994). Designing Social Inquiry. Scientific Inference in Qualitative
Research. Princeton University Press, Princeton, NJ.
Mahoney, J. (2000). Strategies of causal inference in small-N
research. Sociol. Methods Res. 28(4), 387
424.
McDermott, R. (2002). Experimental methods in political
science. Annu. Rev. Political Sci. 5, 31
61.
Oehlert, G. W. (2000). A First Course in Design and Analysis
of Experiments. Freeman, New York.
Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002).
Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, Boston.
Smith, H. L. (1990). Specification problems in experimental and
nonexperimental social research. Sociol. Methodol. 20, 59
91.
Tashakkori, A., and Teddlie, C. (1998). Mixed Methodology:
Combining Qualitative and Quantitative Approaches. Sage,
Thousand Oaks, CA.
Webb, E. J., Campbell, D. T., Schwartz R. D., and Sechrest, L.
(2000). Unobtrusive Measures. Revised Edition. Sage,
Thousand Oaks, CA.
Western, B., and Jackman, S. (1994). Bayesian inference
for comparative research. Am. Political Sci. Rev. 88(2),
412
423.