Alternative Research Design

1
Alternative Research Designs
Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
2
(VI) Alternative Research Designs
3
Alternative Research Designs &

Threats to Validity
In their 1963 work, and in extensions of the original

paradigm, Campbell, Stanley, Cook and others
present various research designs as exemplars of
effective and ineffective ways to deal with various
threats that jeopardized the validity of a research
study
In this section we will focus on various of these

designs indicating by way of example how each
either exposes or protects the study from to internal
and external validity.
Threats to statistical conclusion validity and

construct validity will not be indicated since the
threats go beyond the mere architecture of the
research design.
The designs that will be presented in this section are

as follows.
Not all the designs
4
Pre-experimental designs
One-shot case study
One-group pretest-posttest design
Static-group design
True experimental designs
Pretest-posttest control group design
Solomon four-group design
Posttest-only control group design
Quasi-experimental designs
Time series design
Equivalent time-samples design
Equivalent materials design
Nonequivalent control group design
Counterbalanced design
5
Separate-sample pretest-posttest control group

design
Multiple time-series design
Recurrent institutional cycle design
Regression-discontinuity analysis
The following symbols will be used in presenting the

architecture of each design.
Symbol Interpretation
X The treatment or independent variable.
O Observation or measurement of the

outcome or dependent variable.
R Signifies that the sample subjects have

been randomly assigned to comparison
groups, experimental and control
groups.
(X) ???????
Mk ???????
6
------- This line separating comparison groups

indicates that there is no evidence that
the groups were equivalent in the first
instance.
One-Shot Case Study:

A Pre-Experimental Design
The chief of a 48-officer police department arranges

with a local health club to provide discounted
membership to all officers. He also convinces the
city manager to pay for the membership to improve
the general health and morale of the department.
42 of the officers take advantage of their health

club membership (X), and …
Six months later the chief concludes that both

the general morale (O) of the department is
better than he remembers it being six months
ago.
The Research Design
7
One-Shot Case Study Design
X O
Validity Problems
This study suffers from several threats to validity.
There is no pre-measure of morale other than

the chief’s memory which is non-empirical and
subject to bias.
Events external to the study may have

influenced the morale of the officer independent
of their health club participation, e.g. a pay raise,
promotions, changes in personnel (History).
The morale of the officers might have been

changing over time anyway (maturation).
The morale of the department may have

improved by the mere act of the chief doing
something about it independent of the
8
arrangement for free health club membership

(interaction of selection & X).
The table below summarizes the potential threats to

the validity of a one-shot case study. Consistent with
Campbell and Stanley’s 1963 notation, …
A minus sign (-) indicates the potential presence

of the threat, …
A plus sign (+) indicates that the design protects

against the threat, …
NA indicates that the threat is irrelevant to the

design, and …
A question mark (?) indicates a potential for the

threat dependent upon other considerations.
9
Potential Threats to Validity in a One-Shot Case Study
Potential Threats to Validity Effect On

The Design
Internal Validity
History 
Maturation 
Testing NA
Instrumentation NA
Regression NA
Selection 
Mortality 
Interaction: selection & maturation NA
External Validity
Interaction: testing & X NA

Interaction: selection & X 
Reactive arrangements NA
Multiple –X interference NA
10
One-Group Pretest-Postest Design:

Working with the police department’s crime

prevention unit, …
The city manager installs high intensity street

lighting in a downtown area with an increasing
incidence of street crime.
When the instillation is completed, the cumulative

incidence of crime in the three months proceeding
and the three months following are compared.
Crime fell by 14%.
The Research Design
One-Group Pretest-Posttest Design
O X O
11
Validity Problems
There are several problems with this study.
Any number of extraneous factors external to

the study might have been operant that
depressed the incidence of crime other than the
lighting program, e.g. changes in seasons, fewer
people frequenting the area due to increasing
crime (History).
Crime might have been declining anyway

(maturation).
The table below summarizes the potential threats

inherent in a one-group pretest-posttest design.
12
Potential Threats to Validity in a One-Group

Pretest-Posttest Design

The Design
Internal Validity
History 
Maturation 
Testing 
Instrumentation 
Regression ?
Selection +
Mortality +
Interaction: selection & maturation 
External Validity
13
Interaction: testing & X 

Reactive arrangements ?
Static-Group Comparison Design:

A civil judge, tired of granting continuances to

unprepared lawyers and seeing her docket growing
out of control, …
Adopts a near zero-tolerance policy – “If you

want a continuance you better have a pretty
good reason.”
After six months of holding to this policy, she

compares the number of cases on her docket with
the number on the dockets of the other civil judge in
the courthouse.
Her docket is 12% smaller.
14
The Research Design
The architecture of this static-group comparison is

shown below.
Static-Group Comparison Design
X O
--------------
O
Validity Problems
The essential problem with this study is that there is

no empirical evidence that the dockets of the two
judges were the same at the time the zero-tolerance
policy was initiated.
If two groups are to compared to determine the
15
effect of the independent variable on the dependent

variable, …
Some strategy must be employed to assure that

the groups are comparable on all variables that
affect the dependent variable other than the
independent variable.
This can be accomplished in several ways

including …
Random assignment of cases to groups,
Matching the groups on extraneous

variables,
Using the extraneous variables as blocking

factors or covariates.
The table below identifies the potential threats to the

validity of static-group comparison designs.
16
Potential Threats to Validity in a Static-Group

Comparison Design

The Design
Internal Validity
History +
Maturation ?
Testing +
Instrumentation +
Regression +
Selection 
Mortality 
Interaction: selection & maturation 
External Validity
Interaction: testing & X NA

Reactive arrangements NA
17
Pretest-Posttest Control Group Design:

A True Experimental Design
The state’s narcotic control agency initiated a test to

determine the utility of drug-dogs in increasing both
the number of drug-busts and the quantity of drugs
sized.
The state has 32 local interagency drug task forces,

which were randomly divided into two groups.
A coin flip was used to determine which of the

two groups would participate in the experiment.
The 16 task forces selected for the study were

then randomly divided into two more groups and
…
Another flip of the coin was used to determine

which 8 task forces group would receive grants
to buy drug-dogs.
18
When the results were analyzed a year later, …
The differences in both the mean number of

arrests and the mean amount of drugs seized
were significant, indicating the effectiveness of
the canine program
The Research Design
Pretest-Posttest Control Group Design
R O X O
R O O
Validity Problems
This is a powerful design in [protecting from threats

to validity as indicated in the table below.
If the subjects in such a design know that they are

being measured pre, however, …
The knowledge may interact with the treatment
19
variable (X) limiting the generalizations from the

study to a pretest sensitized population
(Interaction of testing & X).
This may be different from the population the

study was designed to test.
Potential Threats to Validity in a Pretest-Posttest

Control-Group Design

The Design
Internal Validity
History +
Maturation +
Testing +
Instrumentation +
Regression +
Selection +
Mortality +
Interaction: selection & maturation +
External Validity
20
Interaction: testing & X 

Interaction: selection & X ?
Solomon Four-Group Design:

The state’s prison was found libel in five sexual

harassment suits in 16-months, and as a result, has
been directed by the governor to initiate a training
program for all personnel.
Based upon a call to the National Institute of

Corrections, the director finds a curriculum (X) of
what is reported to be an exemplary program, but …
Rather than implementing it system-wide, he

wants to test it efficacy in changing attitudes
about gender issues in the workplace (O).
The test was conducted by taking a random sample

of 200 employees and randomly dividing them into
21
four groups of 50 each.
Two of the groups would take a gender issues

attitudinal survey both one week before and one
week after the implementation of the training
program, but …
Bnly one group would actually participate on the

program
R O X O
R O O
One of the other two groups would participate in the

program but would only take the survey one week
after completing the program, while the other group
would not participate in the program but take the
survey one week later
R X O
R O
The Research Design
22
Solomon Four-Group Design
R O X O
R O O
R X O
R O
As indicated in the table below, this is a powerful

design in controlling for the various threats to
internal and external validity.
Notice that this design cobbles together both a …
Pretest-posttest control-group design (blue), and

a…
Posttest-only control group design (yellow), …
Which taken together greatly enhance its power

to control for threats to validity.
R O X O
R O O
23
R X O
R O
The threats to validity controlled by this design are

presented below.

Potential Threats to Validity in a

Solomon Four-Group Design

The Design
Internal Validity
History +
Maturation +
Testing +
Instrumentation +
24
Regression +
Selection +
Mortality +
External Validity
Interaction: testing & X +

Posttest-Only Control Group Design:

In the previous example on gender issues in the

workplace, the Solomon four-group design included
design included a posttest-only control group design.
By itself, it is a powerful design for controlling threats

to validity, and is more efficient than the Solomon
design since it involves …
Half the number of comparison groups, and only
25
half the number of observations on the

dependent variable.
The Research Design
Posttest-Only Control Group Design
R X O
R O
Validity Problems
The table below indicates the threats that this design

controls.
Potential Threats to Validity in a

Posttest-Only Control Group Design

The Design
Internal Validity
26
History +
Maturation +
Testing +
Instrumentation +
Regression +
Selection +
Mortality +
External Validity
Interaction: testing & X +

Time Series Design:

A Quasi-Experimental Design
The southeastern ward of the city is a transitional area
notorious for crack houses, open prostitution, gambling,
vandalism, burglary and more than its fair share of
robberies and barroom assaults.
After considerable pressure from the councilman who

represents the ward, the police organize a task force to
27
clean the area out.
Over a four moth period, undercover officers work the

area gathering intelligence and …
Over a five day period sweep through the ward

making dozens of arrests and bulldozing crack
houses.
The graph below shows the variation in reported calls for

service in the area four months before and four months
after the task force swept through the area.
CALLS Trend plot

900
850
800
750
700
650
600
Task force sweep
550
500
0 1 2 3 4 5 6 7 8 9
MONTH
Calls for service over an eight month period.
The Research Design
28
Time Series Design
O1 v O1 O1 X O1 O1 O1 O1
Validity Problems
There are two major problems with this design.
History – We have no control group to indicate

whether some external extraneous factor could
account for the down turn in the number of calls
for service.
The interaction of testing with X – In this type of

design there is always the risk that the pre-
measure of the dependent variable might
interact with the treatment variable and …
Make the subjects more or less receptive to

the treatment.
This is not a problem in this task force

example since the pre-measures are
29
unobtrusive.
The table below indicates how this design controls

for various threats to validity.

Potential Threats to Validity in a Time Series Design

The Design
Internal Validity
History —
Maturation +
30
Testing +
Instrumentation ?
Regression +
Selection +
Mortality +
External Validity
Interaction: testing & X —

Equivalent Time Series Design:

Jails are noisy places with the clanging of steel

doors, the shuffling sound of inmates being brought
in and out of cell black, officers calling out inmates,
and of course no soft sound deadening materials like
carpet.
All this noise creates a tense environment for
31
both inmates and staff.
A jail administrator decided to test the effect of easy

listening music on the incidence of inmate
infractions, assaults and the like.
Over a one-month period on random days varying

morning and afternoons, he had music piped into the
cells blocks and barracks.
Analysis of the data indicated that there was a

small yet significant decrease in the incidence of
incidents on the days that music was played (X1
O) compared with days when it was not played
(X0 O).
The Research Design
Equivalent Time Series Design
X1 O X0 O X1 O X0 O X1 O … X0 O
32
X1 = days music played
X0 = days music not played
Validity Problems
Two potential problems with this design are:
Interaction of testing with X, and …
Multiple treatment interference
Testing & X – If the subjects know that they are

being observed, tested, then the pretest sensitization
may affect or interact with the independent variable
(X). This of course is not a problem in the jailhouse
music example.
Multiple treatment interference – The repeated

treatments in this design could have an additive or
even multiplicative effect on the outcome measure
and not be generalizable to applications where there
will be one treatment and one measure of outcome.
The potential threat to validity in this design are

presented in the table below.
33
34
Potential Threats to Validity in an

Equivalent Time Series Design

The Design
Internal Validity
History —
Maturation +
Testing +
Instrumentation ?
Regression +
Selection +
Mortality +
External Validity
Interaction: testing & X —

Multiple –X interference —
35
Equivalent Materials Samples Design:

36
Nonequivalent Control Group Design:

Two juvenile probation officers decide to experiment

with the effect of a “scared straight” program (X) on
the reduction of recidivism (O) among first time
offenders.
One decides to serve as the control group, …
The other agrees to take all of her first-time

probationers to both the state prison and the
state’s juvenile reformatory within the first two
months on probation, there to be “scared
straight.”
The caseloads of the two officers are not exactly the

same since …
Cases are not assigned randomly, rather …
They are assigned on an as-needed basis to

assure that all officers are carrying the same
number of cases.
37
Needless to say that the two officers are not

equivalent either, one is male, the other is female,
and the former has six more years of experience.
The Research Design
O X O
-----------------
O O
Relative to internal validity, …
The primary limitation of this design is that there

is no empirical evidence that the two groups are
equivalent to start with, …
As would be the case if the subjects were

randomly assigned to comparison groups.
None the less, it can be a useful design since

frequently in applied settings, the comparison groups
are pre-existing and randomization to groups is not
possible.
38
Validity Problems
As with other designs that do not have equivalent

comparison groups, this design is potentially liable
for an interaction between selection and maturation,
although this is not likely in the “scared straight”
example.
In addition,the pretests on the outcome variable may

sensitize the
The assets and liabilities of this design are shown in
the table below.

Alternative Research Design

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Alternative Research Design

Uploaded by

Copyright:

Available Formats

1

Alternative Research Designs

(VI) Alternative Research Designs

Alternative Research Designs &

In their 1963 work, and in extensions of the original

In this section we will focus on various of these

Threats to statistical conclusion validity and

The designs that will be presented in this section are

Not all the designs

One-shot case study

One-group pretest-posttest design

True experimental designs

Pretest-posttest control group design

Solomon four-group design

Posttest-only control group design

Time series design

Equivalent time-samples design

Equivalent materials design

Nonequivalent control group design

Separate-sample pretest-posttest control group

Multiple time-series design

Recurrent institutional cycle design

The following symbols will be used in presenting the

X The treatment or independent variable.

O Observation or measurement of the

R Signifies that the sample subjects have

------- This line separating comparison groups

One-Shot Case Study:

The chief of a 48-officer police department arranges

42 of the officers take advantage of their health

Six months later the chief concludes that both

The Research Design

One-Shot Case Study Design

This study suffers from several threats to validity.

There is no pre-measure of morale other than

Events external to the study may have

The morale of the officers might have been

The morale of the department may have

arrangement for free health club membership

The table below summarizes the potential threats to

A minus sign (-) indicates the potential presence

A plus sign (+) indicates that the design protects

NA indicates that the threat is irrelevant to the

A question mark (?) indicates a potential for the

Potential Threats to Validity in a One-Shot Case Study

Potential Threats to Validity Effect On

Interaction: testing & X NA

One-Group Pretest-Postest Design:

Working with the police department’s crime

The city manager installs high intensity street

When the instillation is completed, the cumulative

Crime fell by 14%.

The Research Design

One-Group Pretest-Posttest Design

There are several problems with this study.

Any number of extraneous factors external to

Crime might have been declining anyway

The table below summarizes the potential threats

Potential Threats to Validity in a One-Group

Potential Threats to Validity Effect On

Interaction: testing & X 

Static-Group Comparison Design: