Reply: Market Premium, Social Process, and Statisticism

Author(s): Yu Xie and Xiaogang Wu

Source: American Sociological Review, Vol. 70, No. 5 (Oct., 2005), pp. 865-870
Published by: American Sociological Association
Stable URL: http://www.jstor.org/stable/4145364
Accessed: 01/02/2010 03:49

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/action/showPublisher?publisherCode=asa.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Sociological Association is collaborating with JSTOR to digitize, preserve and extend access to
American Sociological Review.

http://www.jstor.org
Replyto Jann

MarketPremium,SocialProcess,
and Statisticism
Yu Xie Xiaogang Wu
Universityof Michigan Hong Kong Universityof Science & Technology

is a powerful, yet potentially dan- The thrustof Jann's(2005) comment is that

Statistics
gerous,tool. Morethantwo decadesago, the
late Otis Dudley Duncan (1984:226) warned
us concerningthe dangerof"statisticism":"the
notion that computing is synonymous with
doing research,the naive faith that statistics is
a complete or sufficient basis for scientific
methodology, the superstition that statistical
formulasexist for evaluatingsuch things as the
relative merits of different substantive theo-
ries."Duncan's warning has long been under-
stood as applying to sociologists who do
quantitativeresearch.His concern, however,is
equally applicableto readers,even some care-
ful readers,of quantitativeresearch.
In our earlierwork (Wu and Xie 2003), we
askedthe question,"Does the marketpay off?"
Ouremphasiswas on the potentialheterogene-
ity of workersin the marketsector.Using work
history data, we distinguished between two
types of workers in the market sector: early
birds and later entrants.We were concerned
with the possibility thatpooling earlybirdsand
later entrants,even if they each have the same
educationreturnsas stayersin the state sector,
may makethe returnsto educationappearhigh-
er in the marketsector than in the state sector.
Our main empirical results, which also were
confirmed by Jann,showed that later entrants,
but not early entrants,have significantly high-
er returnsto educationthan stayers.
Directall correspondenceto Yu Xie, P.O.Box
1248, 426 ThompsonStreet,PopulationStudies
Center,
University AnnArbor,Michigan
of Michigan,
(yuxie@umich.edu) orXiaogangWu,SocialScience
Division, Hong KongUniversityof Science and
Technology,ClearWaterBay,Kowloon,HongKong 1 Itwasthefirstauthorwhosuggestedto Jannthat
(sowu@ust.hk).TheauthorsthankEmilyGreenman, his problemwith the Wu and Xie (2003) article
JerryA. Jacobs,andTonyPerezfor commentsand shouldbe rephrased asa problemof insufficientsta-
suggestions. tisticalpower.
thereis insufficient statisticalpowerin the data
for differentiatingthe educationreturnsof early
birds from those of later entrantsand those of
stayers.'This point is technicallyvalid,although
only within the narrowstatisticalparadigmof
"null hypothesis significance testing," which
has been severely criticized in recent decades
(e.g., Cohen 1994). A basic problemis thatany
differenceis boundto become "statisticallysig-
nificant" with sufficient data. The late John
Tukey(1991:100)hadthe followingto say about
this paradigm:"Statisticiansclassically asked
the wrongquestion-and werewilling to answer
with a lie, one that was often a downrightlie.
They asked 'Are the effects of A and B differ-
ent' and they were willing to answer 'no'....
[Weknow]thatthe effects ofA andB arealways
different-in some decimal place-for any A
and B."
In the final analysis, our disagreementwith
Jannis not aboutthe technicalcorrectnessof sta-
tisticalmethods,but abouthow statisticalmeth-
ods shouldbe used in sociological research.We
are strong believers in the viewpoint that sta-
tistical methodology should not be separated
from substantiveconcerns in guiding research.
Jann's"methodological"critiqueof ourworkis
misdirected precisely because it is narrowly
methodological, lacking an understandingof
both the substantiveresearchquestion and the
underlyingsocial processes. As we show later,
the substantiveresearchquestion rendersirrel-
evant his test for the difference between early
birds and later entrants.

REVIEW,2oo5, VOL. 70 (October.865-87o)

AMERICANSOCIOLOGICAL
866 AMERICAN REVIEW
SOCIOLOGICAL
To be fair,Jannshould not be singled out for
falling into the trap of "statisticism"because
such practice is so widespreadin currentsoci-
ology that it often makes quantitativeresearch
unappealing.Jann'scommentillustratesa com-
mon temptation among quantitativesociolo-
gists: reliance on canned statisticaltests rather
than substantiveknowledge.Thus, we take this
opportunityto drawa general lesson for all of
us: only when combined with a substantive
understandingof the social processes involved
can statistical methods result in fruitful
research.
At a fundamentallevel, Wu andXie's (2003)
studywas descriptive.We emphasizedthis point
throughoutthe articleby alertingreadersto the
dangerof aggregationwhen workersin a single
group are in fact heterogeneous.Although we
questioned the prevailingwisdom that marke-
tizationper se "caused"the educationreturnsto
be higher, we never intended our statistical
analysis to be more than a descriptiveexercise.
In such an exercise, formal statisticaltests can
be informativewhen there is sufficient statisti-
cal power. In the absence of strong statistical
information,substantiveknowledgeshouldpre-
vail.
In his comment,Janntreatsthe three groups
underdiscussion-stayers, earlybirds,and later
entrants-as thoughthey were symmetric,like
those of an experimentaldesign. In doing so, he
borrows the language of multiple-groupcom-
parisons commonly used in analysis of vari-
ance (ANOVA) associated with experimental
designs. However, because we were dealing
with observationaldata,ourconcernswere with
the between-group and within-group hetero-
geneity generatedby social processes.The earn-
ings regimes for the threegroupsresultedfrom
a cumulative historical process that clearly is
asymmetrical (Figure 1) and thus should be
treated as such in an analysis. Comparingthe
three groupsas if they were threeexperimental
conditionsis bothinappropriate andmisleading.
Figure 1 presents a schematic flow chart of
the respondenttypes in the 1996 survey of Life
Histories and Social Change in Contemporary
China,the dataused by WuandXie (2003). The
y-axis representsthe employmentsector (state
vs. market),andthe x-axis representshistorical
time. We make the convenientassumptionthat
the marketsector is an absorbingstate so that
there is no reverse transitionfrom the market
sectorto the statesector.2In 1978, at thebegin-
ning of the Chinese economic reform, 1,197
respondentsworkedin the statesector.By 1987,
11 percenthadmadethe transitionintothemar-
ket sector (d = 1) and are called "earlybirds."
Among the remaining1,068 workersin the state
sector and the 522 new entrantswho startedin
the sector between 1978 and 1987, 16 percent
made the transitioninto the marketsector(d =
2) and are called "laterentrants."3 The remain-
ing 1,337 respondentsare called "stayers."
The Wu and Xie (2003) article was partly
responsible for causing Jann's confusion,
because his reanalysisis modeled afterWuand
Xie's (2003) regressionsof logged earningson
education, sector, and their interactions.
Extensive discussion in Wu and Xie (2003) on
the differentialreturnsto education by sector
makesit appearas if educationis the causalfac-
tor, with sector as a covariate. However, our
real researchquestion concernedearningsdif-
ferences by sector, with education as a con-
founder.Let us revisit a passage (Wu and Xie
2003:430) cited and emphasizedby Jann.
Thecrucialdifferencebetweenthetwohypothe-
of earlybirds.In Hypothesis
ses is the treatment
1, earlybirdsare groupedwithinlaterentrants
becausetheysharethe commonfeatureof being
in the marketsector.... In Hypothesis2, early
birdsare groupedwith stayersbecausethe two
typesof workers wereapproaching a convergence,
against which later entrantswere selectively
recruitedintothemarketsector.
Jannassumesthatthephrase"aregroupedwith"
here means "sharethe same educationcoeffi-
cient with."This is incorrect,as shownin Figure
1 of Wu and Xie (2003:431). It is possible, for
example, thatearlybirds and laterentrantsdif-
fer in earningsand educationdistributionsbut
have the same educationreturnsas stayers,yet
pooling early birds and later entrantstogether
still could yield higher returns to education
2Thatis, we excludeda smallnumberof"market
losers"fromFigure1 becauseof the group'ssmall
size (WuandXie 2003).Webasedtheclassification
onWuandXie'scomprehensive measureof themar-
ketsector.
3 Here275 workers,including82 "laterentrants"
in WuandXie's(2003)originalarticlewhoentered
the laborforceafter1987 weredroppedfromthe
analysis.
 AND REPLY 867
COMMENT

NewEntrants to the
StateSector(522)

o
Experienced Stayers Stayers Stayers
a Workers(1,197) (1,068) (1,590) (1,337)

d= P= 0.11 d=2 P2= 0.16

LaterEntrants
S(253)

EarlyBirds EarlyBirds
(129) (129)

1978 1987 1996

Year

Figure 1. Flow Chartof LaborMarketTransitionsin China,1978-1996.

amongworkersin the marketsectorthanamong
stayersin the state sector.
To advancethis inquiry,let us reconceptual-
ize the substantiveproblemwith explicit coun-
terfactualsin the language of causal inference
(Heckman2005; Holland 1986; Manski 1995;
Winship and Morgan 1999). Suppose we are
interestedin the causalimpactof the entryto the
marketsector on (potential) futureearningsin
1996. Conceptually,there are two causal ques-
tions in this setup: (1) what is the effect of an
earlytransition?(i.e., d = 1) and (2) what is the
effect of a late transition?(i.e., d= 2). Of course,
these two questions are inherentlyasymmetric.
The second is sensible only for those workers
who did not experience an early transition,
whereas the first involves the counterfactual
comparisonbetweenthose who experiencedan
early transitionand those who did not, regard-
less whathappenedto them later.To borrowthe
notationfor causal inferencewith time-varying
treatments(BrandandXie 2005), let Yi denote
the ith person'spotentialoutcome if the person
has madea transitionat time d (d = 1, 2, oo),with
d = oo denoting thatthe person has not made a
transitionby the end of the study (i.e., a stay-
er).Note thatfor a personwho has madean early
transition(d = 1), the counterfactualoutcome
shouldfollow the principleof "forward-looking
sequential expectation"(Brandand Xie 2005),
a combination of a late transition(d = 2) and
staying (d = oo).We thus define the average
causal effect for the first question as
- = E(Yd=) -
E(ydl) E(yd>l)
+ E(yd=m)(1- P2)].
[E(Yd=2)P2
Note thatthe transitionprobabilitiesare condi-
tional so that P2 = P(d = 2 Id > 1).
For the second question, the comparison is
simpler,involving two regime-specific means:
E(Yd=2)- E(Yd>2)= E(Yd=2)- E(Yd=o) (2)
It is neverpossibleto computequantitiesdefined
by equations 1 and 2 because we observe only
one of the three potential outcomes for each
worker. To infer causality, it is necessary to
introduce the ignorability assumption, which
mustbe takenas provisionalbecauseit is unlike-
ly to hold in reality.The ignorabilityassumption
868 AMERICAN
SOCIOLOGICAL
REVIEW
statesthatall systematicdifferencesassociated
with the transitions can be summarizedby a
set of observedcovariates(X) (Rosenbaumand
Rubin 1984).
Given this assumption, the expected earn-
ings can be estimated on the basis of the
observed covariates, including education. As
shown in equations 1 and 2, we need four con-
ditional expectations for the causal
analyses:E(Ydl IX) andE(YP'lIX) for the first
question,andE(yd=2IX) andE(yd>2IX) for the
second question. The ignorabilityassumption
means thatE(Yd I X) can be estimatedamong
earlybirds,E(yd=2X) amonglaterentrants,and
E(yd>2IX) amongstayers.However,E(Y>2 IX),
as a weighted sum of two conditionalexpecta-
tions, should be estimated from both later
entrantsand stayers.Given thatP2 is small (at
.16), a crude approximationof E(Y"> IX) can
be estimated from stayers (i.e., giving a full
weight to stayers). This approximationis an
interpretationof Wu and Xie's (2003) analysis
strategy.Because laterentrantsconstitutedonly
a small proportion of the appropriategroup
againstwhich earlybirdsare to be compared,it
makes little sense to compare,as Jann recom-
mends, later entrantsdirectlywith early birds.
To illustratethe utility of this reconceptual-
ization, we performpropensityscore analyses.
Because of space limitation,we presentonly the
most importantfindings in this discussion.The
full results fromthis exercise are reportedelse-
where (Xie and Wu 2005). To borrow the jar-
gon from the causal inference literature, we
considertwo "treatments" in ourstudy:an early
entryto the marketsectoranda late entryto the
marketsector.Forthe first treatment,the "con-
trol" group consists of workers who did not
make an early entry,and thus includes stayers
as well as later entrants.For the second treat-
ment, the "control"group consists of stayers
only.The propensityscore methodallows us to
summarizeall the differencesbetweenthe treat-
ment and control groups with a single dimen-
sion: the probabilityof receiving a particular
treatment.We then compute the averagetreat-
ment effect on earnings within each propensi-
ty score stratum.
There are two main findings from this
propensityscore analysis. First,the propensity
model for a late transitiondiffers from that for
an earlytransitionbecause the mechanismsfor
making transitions changed. Whereas human
capital and political capital measures such as
education, party membership, seniority, and
cadreconnectionnegativelypredictedthe prob-
ability of an early transitionto the marketsec-
tor, this patternwas much less pronouncedfor
a late transition.
Second, we find the treatmenteffect of mar-
ket entry to be very differentfor a late transi-
tion than for an early transition.For an early
transition, we find no effects on earnings in
any of the propensity score strata.For a late
transition,the estimatedtreatmenteffect is rel-
atively large and significantly different from
zero for the four lowest propensityscore stra-
ta. We presentthe resultsin Figure2. If we pool
the differentstratatogetherfor an overalltreat-
ment effect under the homogeneous effect
assumption, the estimate is 236 RMB yuan
(Chinese currency), with a standarderror of
54, resulting in a highly significant t value of
4.36. However, the assumption of a homoge-
neous treatmenteffect is clearlyviolatedby the
downwardtrendobserved in Figure2. Using a
hierarchicallinearmodel, we find thatthe size
of the treatmenteffect stronglyand negatively
depends on the propensity score, with a unit
change in stratumrank(i.e., crossinga propen-
sity score stratum)associated with a reduction
of 94 RMB yuan in the treatmenteffect (a sig-
nificant relationship, with t = -3.6). That is,
the benefit of a late transitioninto the market
sector is greatest among those who were least
likely to make the transition and diminishes
with the propensityfor making the transition.
To the question "does the marketpay off?"
these new resultsyield no simpleanswer.We do
not find a generic market effect on earnings.
Rather,the effects vary acrosstwo dimensions.
First, confirming Wu and Xie's (2003) earlier
results, we again find no evidence of a premi-
um for an earlymarkettransition,whereasa late
transition into the market sector is associated
with higher earnings. Furthermore,we show
that even among later entrants,the benefit of
working in the market sector sharplydecreas-
es with the propensityof havingmadethe tran-
sition. Hence, the summary finding of our
reanalysisis that the marketpremiumis limit-
ed to only laterentrantswho otherwisehada low
likelihood of making a transitionto the market
sector.Who are they?
In all likelihood, these low-propensitylater
entrantsareworkersdoing especiallywell in the
 AND REPLY 869
COMMENT

900 -
* 6.07 * Stratum-specific -Linear fit (HLM)

700 -

500 -
Y = 645.2 - 94.3 X STRATUM_RANK

'o 300 -
o 500
t 3.33
0.31
,

S100-.9
-100-
* -0.65

1 2 3 4 5 6 7 8
PropensityScore Stratum

Figure 2. MarketTreatmentEffecton Earningsby PropensityStratum:LaterEntrantsversusStayers.

Notes:Numbersin the scatterplotaret valuesfor an earningscomparisonbetweenlaterentrants(treatment
group)andstayers(controlgroup);t < 1.96 indicatesthatthereis no significantdifferencein earningsbetween
the treatmentandcontrolgroupswithina propensityscore stratum.The linearfit is basedon the hierarchical
linearmodelestimates(level 2 modelwith slopes fromthe level 1 model as outcomesregressedon propensity
stratumrank);effect of propensitystratumrankis statisticallysignificant(t = -3.6).

statesector.A cost-benefitanalysissuggeststhat tive statisticaltests at the expense of substantive

for a person to make a transitionfromthe state
sector to the marketsector, the benefit of vol-
untarilyenteringthe marketmust exceed thatof
stayingin the statesector.Workerswho do well
in the state sector and are unlikely to lose their
jobs havea good incentiveto stayput. Forthem,
the attractionof the marketsector must be suf-
ficiently large to morethan compensatefor the
advantagesthey alreadyenjoy in the state sec-
tor.Therefore,only those with the best market
opportunitiesactually make the transition.We
highlightedthese individualsgraphicallyin Wu
and Xie (2003:435), referringto them as vol-
untarylater entrants.These results illustratea
classic violation of the ignorabilityassumption,
the problem of endogeneity.Individualsselect
their "treatment"on the basis of the anticipat-
ed outcome, which is not homogeneous across
workers.This kindof insightinto social process-
es can never be producedby analyses such as
Jann's.His critiquefocuses on an inappropriate
comparison between early birds and later
entrants,a result of his reliance on uninforma-
knowledge. As such, Jann'sstatisticalexercise
contributes little to the understandingof the
social processes underlyingthe empiricalpat-
terns reportedby Wu and Xie (2003).
YuXie is OtisDudleyDuncanProfessorofSociology
and Statistics at the University of Michigan; a
ResearchProfessor at thePopulationStudies Center
andtheSurveyResearchCenterof theInstitute for
Social Research;and a FacultyAssociate at the
Centerfor Chinese Studies. His main interests are
social stratification,demography,statisticalmethods,
Chinesestudies,andsociologyof science.Hispub-
lished works include Statistical Methods for
CategoricalDataAnalysis(AcademicPress 2000)
with Daniel Powers,Womenin Science: Career
ProcessesandOutcomes(HarvardUniversity Press
andA Demographic
2003)withKimberlee Shauman,
PortraitofAsianAmericans(Russell SageFoundation
and Population Reference Bureau 2004) with
KimberlyGoyette.
XiaogangWuisAssistant ProfessorofSocialScience
at theHongKongUniversity ofScience&Technology
(HKUST). Hisresearchinterestsincludesocialstrat-
ificationandmobility,labormarketsandeconomic
870 AMERICAN
SOCIOLOGICAL
REVIEW
sociology, and organization. He has published in
American Sociological Review, Social Forces, and
Demography.His current research applies causal
models to examinethe relationshipbetweenstratifi-
cationprocesses andstructuralinequalitiesin China.
He recently received a 3-year grant,funded by the
Research Grants Council of Hong Kong, to study
social inequality in access to educational opportu-
nities in reform-eraChina.
REFERENCES
Brand,Jennie E. andYu Xie. 2005. "Time-Varying
Treatments,Time-VaryingEffects:CausalEffects
in a LongitudinalSetting."AnnualWinter/Spring
Meeting of Sociological Methodology Section,
American Sociological Association, April 22,
Chapel Hill, NC.
Cohen, Jacob. 1994. "TheEarthIs Round(p < .05)."
TheAmericanPsychologist 49:997-1003.
Duncan, Otis Dudley. 1984. Notes on Social
Measurement,Historical and Critical.New York:
Russell Sage Foundation.
Heckman, James J. 2005. "The Scientific Model of
Causality."Departmentof Economics, University
of Chicago,Chicago,IL. Unpublishedmanuscript.
Holland, Paul W. 1986. "Statistics and Causal
Inference"(with discussion).JournalofAmerican
StatisticalAssociation 81:945-70.
Jann,Ben. 2005. "EarningsReturnsto Educationin
UrbanChina:A Note on TestingDifferenceamong
Groups." American Sociological Review
70:860-864.
Manski,Charles.1995.IdentificationProblemsin the
Social Sciences. Boston, MA: HarvardUniversity
Press.
Rosenbaum, Paul R. and Donald B. Rubin. 1984.
"Reducing Bias in ObservationalStudies Using
Subclassificationon the PropensityScore."Journal
ofAmerican StatisticalAssociation 79:516-24.
Tukey,John W. 1991. "The Philosophy of Multiple
Comparisons."Statistical Science 6:100-16.
Winship,Christopherand StephenL. Morgan.1999.
"The Estimation of Causal Effects from
ObservationalData."AnnualReviewofSociology
25:659-707.
Wu, Xiaogang andYu Xie. 2003. "Does the Market
Pay Off? EarningsReturnsto Educationin Urban
China."AmericanSociologicalReview68:425-42.
Xie, Yu andXiaogangWu. 2005. "MarketPremium,
Social Process, and Statistical Naivety: Further
Evidence on DifferentialReturnsto Educationin
Urban China." PSC Research Report 05-578,
PopulationStudiesCenter,Universityof Michigan,
Ann Arbor,Michigan.