Statistical Models and Causal Inference: A Dialogue with the Social Sciences by David Freedman; David Collier; Jasjeet

Sekhon; Philip Stark Review by: Jacob Felson Contemporary Sociology, Vol. 40, No. 1 (January 2011), pp. 30-31 Published by: American Sociological Association Stable URL: http://www.jstor.org/stable/25762936 . Accessed: 22/01/2014 11:52
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Sociological Association is collaborating with JSTOR to digitize, preserve and extend access to Contemporary Sociology.

http://www.jstor.org

This content downloaded from 157.253.50.10 on Wed, 22 Jan 2014 11:52:56 AM All use subject to JSTOR Terms and Conditions

30

Reviews traditional regression and weighted regression in terms of bias and variance of the causal effect estimates. Weighting regres sions by propensity scores can reduce bias, but the variability inweights increases ran dom error and the estimated standard errors produced by standard software will be too small. Bias is particularly severe in logistic regression models weighted by propensity PS
scores. These are

and Causal Inference: A Statistical Models Dialogue with the Social Sciences, by David Freedman. Edited by David Collier, Jasjeet Sekhon, and Philip Stark. Cambridge, UK: Cambridge University Press, 2010. 399pp. $29.99 paper. ISBN: 9780521123907. William Paterson University felsonj@wpunj.edu In this posthumously published collection of essays and articles, statistician David Freed man evaluates the work of quantitative social scientists, and gives them a failing grade. According to Freedman, quantitative social scientists routinely estimate models that rely on tenuous assumptions and that manner in bear littleresemblance to the which their data were generated in the real world. Testing model assumptions with diagnostic tests is generally unhelpful, since the diagnos tic tests are not powerful enough to detect consequential violations. In short, Freedman argues that statistical models have become Procrustean beds intowhich social data are routinely and ritualistically rammed. Some methodologists may agree and argue that a solution to the problem can be found through less reliance on canned approaches and more customized models. But Freed
man's

Jacob Felson

he is sympathetic tomodel customization as a solution in some cases, Freedman believes that statistical ingenuity has quickly dimin ishing returns when applied to social data. Social data rarely behave in a way that is con genial for statisticalmodeling. In each chapter, Freedman provides an example to illustrate the fragilityof a variety of widely used methods and models. Many of these examples are less closely related to the social sciences than one might expect in a book subtitled "a dialogue with the social sciences." For example, chapters are devoted to earthquake forecasts and to the swine flu vaccine. However, there should stillbe plenty of material of interest to quantitative social
scientists.

critique

is more

far-reaching.

While

Freedman's general argument that too much research is based on tenuous assump tions will not be unfamiliar to many. His main point is more likely to be met with rather than begrudged acknowledgment Freedman's surprise. approach harks back to a frequently cited paper written twenty eight years ago: "Let's Take the Con out of Econometrics" (Learner, 1983). But as social movement theorists have long known, is of a problem widespread knowledge insufficient to lead to change, especially when incentives are aligned against it. As Upton Sinclair said, "It is difficult to get a man to understand something, when his salary depends upon his not under standing it." In terms of solutions, Freedman offers only a broad outline. He says quantitative researchers should spend less time coming up with new models and more time attempt ing to replicate existing models on new data. We should forego the quest to identify causal effectswith statistical techniques except in the rare instances when there is strong evi dence backing up the model assumptions. Researchers should seek answers to causal questions through a combination of logic and deep knowledge of the subject matter. Statistics will no doubt play a role as well, but to paraphrase Glenn Firebaugh (2008), method should be the servant, not the master. a series of Freedman Through examples, seeks to demonstrate that successful process es of discovery have relied as much on infor mal reasoning and qualitative insights as on statistical analysis. Discoveries also occurred through theprocess of investigating apparent
anomalies.

important

lessons.

For example, Freedman and his coauthor Richard Berk in their chapter find serious problems with weighting regression esti mates by propensity score matching. Using simulated data, Freedman and Berk compare Contemporary Sociology40,1

All of the exemplars of research presented by Freedman are frommedicine, which is odd, given that the book is pitched to social scientists. Although of social examples research do not resound with success like

This content downloaded from 157.253.50.10 on Wed, 22 Jan 2014 11:52:56 AM All use subject to JSTOR Terms and Conditions

Reviews lifesavingmedical discoveries, social science examples would resonate more with social scientist readers. Medical examples are also not particularly helpful for social scientists because causes in the social world do not usually operate like viruses and vitamin defi ciencies in thebody. Diseases and syndromes are much more predictable and less contin
gent than social causes. Freedman's discus

31

Glenn. Seven Rules for Social 2008. Firebaugh, Research. Princeton, NJ: Princeton University Press. Freese, Jeremy. 2007. "Reproducibility in Quantitative Social Science: Why Standards Not Soci

ology?"
Learner,

36:153-172.

Sociological Methods
1983. "Let's American Take

and Research
of

Edward.

the Con Out

Econometrics." 73:31-43.

Economic

Review.

sion ofmedical discoveries is interesting for in Cristobal. 2009. "Model Uncertainty Young, itsown sake, but not as useful as a demonstra to Reli An Research: Sociological Application tion of how social scientific research could or American Socio Growth." gion and Economic should be done. It would have been more 74:380-97. logical Review. useful to see examples of compelling social research, and to see the role (if any) played by statisticalmodeling. Codes of theUnderworld: How Criminals Com Quantitative researchers who are sympa Princeton, municate, by Diego Gambetta. are to and Freedman's thetic perspective Princeton NJ: University Press, 2009. 336pp. about how for detailed guidebooks looking $35.00 cloth. ISBN: 9780691119373. to proceed should look elsewhere. Those interested in a more circumspect approach Jana Arsovska to quantitative analysis might profit more JohnJayCollege ofCriminal Justice from reading Richard Berk's (2003) book jarsovska@jjay.cuny.edu about regression modeling, Freese's (2006) for replication standards, and Diego Gambetta's innovative book Codes proposal Young's (2009) paper about how to avoid of theUnderworld: How Criminals Communi cate has a lot to say about signaling, trust underestimating the uncertainty in model and semiotics. By combining theory with specifications. But many of theways inwhich causes are empirical findings, Gambetta illustrates the found cannot be distilled to theiressence, sys dilemmas criminals face as they struggle tematized, and categorized in a separate field with the problems of whom to trust, how called "methods." Instead, themeans of dis to make themselves trusted, and how to covery are often unique to the problem at maintain secrecy. Criminals want to signal hand. Freedman, the statistician, is abdicating untrustworthiness, namely that they are the throne that quantitative researchers may crooks. For thepurpose ofworking with oth him. As Learner built have for er criminals, they also wish to signal that unwittingly (1983)wrote, "methodology, like sex, is better they are reliable. This need for dual signal communication demonstrated than discussed, though often underworld ing makes a remarkable case study. As Gambetta better anticipated than experienced." Can one thinkof a major social scientific question rightly claims: "[0]ne wonders how crimi thathas been resolved primarily through the nals ever manage to do anything together" use of statisticalmodels? This is the question (p. 32). Freedman's book implicitly asks. For his sup Gambetta's task in Codes of theUnderworld his For is rhetorical. the to decode how criminals signal to each is porters, question other.When identifying partners, criminals detractors, the answer to the question would can miss be their first line of defense. failing to see opportunities, that the potential partners disguises through adopt. They may also mistakenly approach can undercover agents. Miscommunication References therefore be costly. Codes of theUnderworld the techniques of dealing with A Con 2003. Regression Analysis: analyzes Beck, Richard. under conditions of uncertainty. information CA: Thousand structive Critique. Oaks, Sage It captures the reader's attention by raising Publications. Sociology40,1 Contemporary

This content downloaded from 157.253.50.10 on Wed, 22 Jan 2014 11:52:56 AM All use subject to JSTOR Terms and Conditions