CHANCE

ISSN: 0933-2480 (Print) 1867-2280 (Online) Journal homepage: http://amstat.tandfonline.com/loi/ucha20

The Odds of Justice: Actuarial Risk Prediction and

the Criminal Justice System
Sonja Starr
To cite this article: Sonja Starr (2016) The Odds of Justice: Actuarial Risk Prediction and the
Criminal Justice System, CHANCE, 29:1, 49-51, DOI: 10.1080/09332480.2016.1156368
To link to this article: http://dx.doi.org/10.1080/09332480.2016.1156368

Published online: 24 Feb 2016.

Submit your article to this journal

Article views: 70

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://amstat.tandfonline.com/action/journalInformation?journalCode=ucha20
Download by: [Florida State University]

Date: 24 September 2016, At: 08:00

[The Odds of Justice]

Mary W. Gray
Column Editor

Actuarial Risk Prediction and

the Criminal Justice System
Sonja Starr

onsider the following scenario: Two defendants,

Robert and William, are
convicted of the same crimeEach
stole $10,000 worth of jewelry from
a jewelry store. Neither has any
prior convictions. In other ways,
their situations differ.
Robert is a regional manager of
a jewelry store chain; he is 35, married, college-educated, grew up in
a stable middle-class family, and
now lives in a suburban house with
a comfortable income. Over the
course of several months, he stole a
number of diamonds and replaced
them with fakes, selling the proceeds and using them to pay for the
lease on a luxury car.
Meanwhile, William is 18 and
unemployed. He grew up in a poor
family that has relied on welfare
and been periodically homeless; his
mother is an occasionally verbally
abusive drug user; his brother and
father are incarcerated. William and
his mother were recently evicted
again. To try to earn money to rent
a new apartment, William dropped
out of high school and started
looking for jobs at local businesses.
However, after weeks of searching, he had no success. Frustrated,
he saw an unlocked display case
while passing a jewelry store and
grabbed a handful of necklaces. He
was caught moments later by a mall
security guard.
Should Robert and William be
sentenced identically? If not, who
should get the more serious sentence?

Your answers might depend on

what you think the main purpose
of sentencing is. If your goal is to
assign a morally just punishment in
proportion to what the defendant
deserves, you might think that, if
anything, William deserves a lesser
sentence: His crime was the economically desperate act of a kid
who has faced every disadvantage
in life, trying to save his family from
homelessness. Perhaps all of this
mitigates his culpability.
But theres a counterargument.
Williams poverty, homelessness,
joblessness, young age, dropout status, and difficult family background
all suggest that he poses a much
higher risk of being arrested or convicted again in the future than Robert does. If our goal is to protect the
public from the defendants future
crimes, maybe William should be
put in prison, while Robert, who
poses little risk, should stay free.
Judges have always had to balance
a number of competing sentencing
considerations and make difficult
judgment calls. Today, though, that
process is increasingly often shaped
by a new factor: the use of actuarial methods to assess defendants
future crime risk.The risk assessment
trend is often referred to as actuarial sentencing or evidence-based
sentencing. For example, before
sentencing a defendant, judges in
most jurisdictions consider a presentence report prepared by some
official such as a probation officer.

In many jurisdictions, these presentence reports now include an

actuarial prediction of the probability
that the defendant will reoffend in
the future. Similar predictions have
been provided to parole boards for
decades, and they are increasingly
also being used to inform other
criminal justice processes, such as
bail decisions. I describe and critique
this trend in much greater detail in a
recent law review article (Starr 2014).
In a risk-score calculation,
both Robert and William would
benefit from having no criminal
records and, in many jurisdictions,
both would be penalized for being
male, but the other factors affecting
their scores would be very different.
The most popular risk assessment
instruments, such as the LSI-R and
COMPAS, include demographic
factors (especially age, marital status, and sometimes gender), as well
as a variety of socioeconomic and
family background factors.
For example, LSI-R would
penalize William for unemployment, severe financial difficulties,
family members criminal records,
unsatisfactory housing, reliance on
welfare, troubled parental relationships, and failure to complete high
school (Department of Corrective
Services 2002).
I dont know exactly how
Williams score would come out
like other actuarial instruments
produced by private corporations,
the LSI-R is proprietary, and the
weights attached to risk factors are
CHANCE

49

neither made public nor provided

to defendants. But it is certain
that his risk score would be higher
than Roberts.
Statistically, the methods used
to produce the scores are generally uncomplicated. The weights
assigned to particular risk factors
are typically based on those factors regression coefficients in studies of past offenders outcomes. The
actuarial instruments usually do a
pretty good job of predicting group
average rates of recidivism, provided
they are applied to a population that
is similar to the one used to produce
the instrument. Individual criminal
conduct is obviously much harder
to predict than group averages; few
criminal defendants have recidivism
probabilities close to 0 or 1.
Individual predictive accuracy is
often evaluated using the area under
the curve (AUC) method. AUC
scores can be interpreted as representing the probability that, when an
eventual recidivist and eventual nonrecidivist are paired randomly, the
scoring system would have ranked
the recidivist as higher risk. By this
method, the best instruments tend to
score around 0.75; a random instrument would be expected to score 0.5
(see Starr 2014 for a review).
There is a rich literature on
scientific questions surrounding
this risk assessment processfor
example, Berk (2012) and Rhodes
(2013) looked at whether machinelearning methods could produce
better predictions.
The trend toward actuarial
prediction, however, raises serious
questions that go beyond the methodological. Is it morally right for
the state to increase a defendants
punishment because of his various socioeconomic disadvantages,
or because of his familys misdeeds?
What about taking gender into
account, or age, or marital status?
To be sure, all of these factors are
indeed statistical predictors of crime.
Adding them to a predictive model
VOL. 29.1, 2016

50

that already includes criminal history

probably doesnt buy much additional
predictive value, although it buys
somebut at what cost, in terms of
our legal systems values?
One of the core principles of
our criminal justice system is that
all people stand equally before it.
Indeed, in the case of Griffin v.
Illinois, in 1956, Supreme Court
Justice Hugo Black described the
goal of providing equal justice to
poor and rich, weak and powerful alike as the central aim of our
entire judicial system. It is true that
our system has perpetually failed
to satisfy that aim; many obstacles
stand in the way of equal justice
for the indigent, for example. The
current move toward actuarial risk
assessment, however, still represents
something extraordinary.
When the state puts markers
of poverty, family situation, and
demographics in a risk score and
tells judges to use it to decide a
defendants sentence (or to shape
other high-stakes decisions), it is
embracing explicit discrimination
on the basis of those factors. It is
telling the Williams of the world
that the state thinks theyre dangerousand plans to punish them
more harshly; not just because of
what they have done, but because of
who they are, who their families are,
and how much money they have.
This, in my view, is a noxious message, and a noxious practice. It can
only contribute to a perception that
the criminal justice system is rigged
against the poorbecause it is rigging the system against the poor.
Moreover, it is unconstitutional.
The Supreme Court has consistently held that there are some kinds
of statistical generalizations that it is
not acceptable for the government
to rely on, even if they are true.
In 1983, the court decided
the case of Bearden v. Georgia,
holding that it was unconstitutional wealth discrimination to
revoke a defendants probation

(incarcerating him) when he lost

his job. The state introduced a
raft of empirical studies showing
that unemployment and poverty
increased the risk of recidivism, and
argued that his job loss justified
treating him as a higher crime risk.
The court firmly and unanimously rejected that argument. It
held that a sentence increase could
not be based on lumping [the
defendant] together with other
poor persons and thereby classifying him as dangerous.This would be
little more than punishing a person
for his poverty.That is exactly what
todays risk assessment instruments
do; their designers seem to have
forgotten about Bearden, but it is
still a binding legal precedent.
What about race? The prediction
instruments do not include race itself
as a risk factoreven their advocates almost universally agree that
that would be unconstitutionalbut
virtually every risk factor the instruments do include is race-correlated,
so they are bound to produce higher
risk scores for people of color. They
thus risk exacerbating the serious
racial disparities that pervade our
criminal justice system.
The inclusion of age and gender might seem more innocuous
(although the use of gender is
unconstitutional, too, for similar
reasons), but will further concentrate the demographic impact. It is
young men of color who bear the
brunt of mass incarceration: One
black man out of nine under age
35 is currently behind bars (Pew
Center on the States 2008).
Of course, judges already consider risk; they are usually forbidden from using race, gender, or
socioeconomic status to do so, but
could do so covertly. Still, while
judges will remain free to balance risk
reduction with other concerns (such
as moral culpability), risk assessment
scores places a heavy thumb on the
risk part of the scale, and are likely

to amplify the influence of the factors found in the instruments.

For example, I did an experiment
providing law students with Roberts and Williams stories (quite
similar to those above), and provided half of them with risk scores
reporting that Robert was low risk
and William was moderately high
risk. The students without the risk
scores gave Robert longer sentences
(about 17% longer on average); this
pattern reversed when the risk score
was provided (Starr 2014).
Although one cannot make
much of this small experiment (n=83), real-world research
(reviewed in Starr 2014) also shows
that judges, and people generally,
tend to defer to quantitative, scientific assessments. A high-risk
assessment may also create political
pressure on judges, especially elected
ones, to give a tough sentence.
Despite these problems, the risk
assessment movement shows no
sign of slowing down. Risk assessment has been embraced as part of
the modern criminal justice reform
movementit plays a key role in a
big reform bill now pending in the
U.S. Senate, for example. Why are so
many smart, well-meaning people on
board? Part of it is that they see it as a
way to buy support for much-needed
efforts to reduce incarceration, by
convincing people that only low-risk
people will be let out of prison.
In my view, though, it is not obvious that providing judges with risk
assessments will reduce sentences
on balance, rather than increasing
them. Moreover, access to programs
that reduce sentences or allow
early release shouldnt depend on
wealth or identity, either. The U.S.
needs to break its addiction to
incarceration, but risk assessment
is not an essential part of a commitment to doing so. At least, risk
assessment could be based on criminal conduct, rather than on factors
outside the defendants control.

Another factor is the buzzy 21stcentury appeal of evidence-based

or data-driven anything. Advocates
see, in risk assessment, the promise
of bringing scientific methods to an
inherently difficult, and inherently
qualitative, process. Note, though,
that risk assessments do not get rid of
difficult qualitative decisions. Again,
judges still have to decide what
weight to give the assessments, and
different judges will do so differently.
Finally, I believe that one big
reason risk assessment instruments
have received so much support is
that politicians dont understand
themnor, unfortunately, do most
judges and legal practitioners. Virtually no politician would stand up
in public and say, I believe that
people should be punished more for
being poor, for example. I suspect
that few people understand that,
when judges are told to sentence
based on a risk score that includes
measures of poverty, that is exactly
the policy the state is endorsing.
These features of the instruments
do not tend to be broadly advertised.
Instead, all most people hear is that
they are scientific, evidence-based
risk predictionsand when you put
it that way, who could oppose it?
Statisticians can help advance the
level of knowledge in the public
debate by explaining how actuarial
predictions work.
I am an empirical researcher, and
I believe strongly in the use of data
to improve criminal justice policy,
but data-driven improvements do
not have to entail assigning risk
scores to individualsespecially
not risk scores based on morally and
legally problematic factors. Much
of the research underlying the
prediction instruments could have
other, benign applications.
For example, take the finding that unemployment predicts crime (a relationship that is
almost certainly at least partially
causal). You could take that finding and conclude: We should lock

unemployed people up for longer.

Or you could take it and conclude:
We should create effective job
training and placement programs
for prisoners. Likewise, empirical researchers could focus their
efforts not just on predicting crime,
but on figuring out how to reduce
itimproving causal assessment of
the effects of criminal justice
reforms, including alternatives
to incarceration.
There is much we can learn from
data, but we should not let the desire
to quantify and predict cause us to
forget our core values.

Further Reading
Berk, Richard. 2012. Criminal Justice Forecasts of Risk: A Machine
Learning Approach. SpringerBriefs in Computer Science.
Department of Corrective Services.
2002. LSI-R Training Manual.
Pew Center on the States. 2008.
One in 100: Behind Bars in
America.
Rhodes, William. 2013. Machine
Learning Approaches as a Tool
for Effective Offender Risk
Management. Criminal & Public Policy, 12:507510.
Starr, Sonja B. Evidence-Based
Sentencing and the Scientific
Rationalization of Discrimination, Stanford Law Review, 66:
803872.

About the Author

Sonja B. Starr is a member of the faculty of the
University of Michigan School of Law. Her research includes
quantitative empirical assessment of the effects of criminal
justice policies, as well as analysis of legal theory and
doctrine. She has clerked for the Hon. Merrick Garland of
the U.S. Court of Appeals for the D.C. Circuit and the Hon.
Mohamed Shahabuddeen of the shared Appeals Chamber of
the International Criminal Tribunals for Rwanda and the former
Yugoslavia in The Hague. Her JD is from Yale Law School,
where she served as senior editor of the Yale Law Journal. She
received her AB from Harvard, summa cum laude, and is also
an alumna of the Inter-University Consortium for Political and
Social Research Summer Program in Quantitative Methods.
CHANCE

51