Resampling Methods for Computation-Intensive Data Analysis in Ecology and Evolution Author(s): Philip H.

Crowley Source: Annual Review of Ecology and Systematics, Vol. 23 (1992), pp. 405-447 Published by: Annual Reviews Stable URL: http://www.jstor.org/stable/2097295 . Accessed: 11/07/2011 18:37
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at . http://www.jstor.org/action/showPublisher?publisherCode=annrevs. . Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Annual Reviews is collaborating with JSTOR to digitize, preserve and extend access to Annual Review of Ecology and Systematics.

http://www.jstor.org

Annu.Rev. Ecol. Syst.1992. 23:405-47 ? Inc. All rights Copyright 1992 byAnnual Reviews reserved

RESAMPLING METHODS FOR COMPUTATION-INTENSIVE DATA ANALYSIS IN ECOLOGY AND EVOLUTION

PhilipH. Crowley
Center Evolutionary for T. Schoolof Biological Sciences, Ecology, H. Morgan for of 40506-0225, NERC Centre and University Kentucky, Lexington, Kentucky Park, Ascot, Berkshire 7PY Population Biology, Imperial College Silwood at SL5 United Kingdom
KEYWORDS: bootstrap, Monte Carlomethods, tests permutation randomization tests, jackknife,

INTRODUCTION
of widely available)microcomThe advent fast, relatively inexpensive (thus, is the datainecologicalandevolutionary puters transforming waywe analyze research.Even more profound, however,are the associatedchanges in and methods conducted, interpretations questions asked, empirical used,studies Now thatan array computation-intensive of statistical methods is offered. to availablefor important assesstheir newly general use, itseemsparticularly and to beingused, and advantages limitations, notehow theyare currently then consider to for implications thefuture. I focusin thisreviewon fourrelated knownin the statistical techniques andbiological literature randomization permutation) MonteCarlo as tests, (or I to as and methods, bootstrapping, thejackknife. refer themcollectively taking several-to-many samples resampling methods, becauseeach involves from original bootstrap, jackknife) froma or the data set (randomization, stochastic liketheone believed havegenerated dataset(Monte to the process is an of family techniques Carlo). Each of thesemethods actually extensive I be thoroughly examined and specific that here;instead, applications cannot in thefocalmethods then and therecent literature characterize survey briefly associatedwith to the ecologyand evolution identify issuesmostfrequently in thesetechniques. emerges resampling It that methods wellrepresented are 405 0066-4162/92/1120-0405$02.00

406

CROWLEY

issues and intense data analysesrelatedto some of the most important of in currently thesefields research. controversies of objectives thispaperare: The specific with biologists and of a 1. to acquaint widerarray ecologists evolutionary havebeen recently, which,at leastuntilvery theseusefultechniques, training; in or underemphasized ignored statistical and questions one research certain between the 2. to document association methods; or moreof theseresampling the in the 3. to emphasize role of thefocalmethods expanding rangeof basisofdata the designs inshifting conceptual and feasible experimental analysis; methodswith more standard resampling 4. to compare and contrast that key and assumptions other features bearon their noting approaches, and for applications; appropriateness particular in and that methods needclarification development, thehope 5. tohighlight and by that thesewill soonbe addressed statisticians biometricians. in studies, accordwith ecological biasedtoward is Thereview intentionally analysesare univariate Simpler, and experience interests. myown research in of and emphasized theinterest clarity also becausea reviewof computa(165). To respect in is methods ecology inprogress tion-intensive multivariate of the to on page limits contributions thisvolume,I haverestricted number morerecent papersmostlikelyto contain examplescitedand emphasized readers familiar are of work.I assumeherethat additional citations relevant and statistical concepts basic methods. with rudimentary the I briefly four as First, describe reviewproceeds follows: The present of example each straightforward a including relatively resampling techniques, of results a of Next,I summarize from literature ecologyand evolution. the searchof a including computer searchforapplications, literature systematic 1985-1990,and my volumes during and published biological journals edited ecologicaljournals all own searchby handthrough issuesof twoprominent for are techniques usedtotest temporal for period the 1985-1991.Resampling of in and in trends theuse of thesemethods fordifferences frequency use in identified the studies.Publications betweenecologicaland evolutionary seven whichare distilled from searchare classified topicand subtopic, by in with applications moredetail.Focus then majorissuesconsidered example to of applications classical and actively shifts the relation resampling to disadvanI statistical Finally, discussadvantages, methodology. developing questions methodological of methods, highlight tages,andimplications these recommendations. somespecific and that deserve attention, close with

STATISTICAL RESAMPLING METHODS

407

BRIEF DESCRIPTIONS OF THE FOCAL METHODS

programs and and Lucid descriptions examplesof thesemethods computer booksby Edgington them suitable implementing are availablein recent for (64), Noreen (186), andManly(163).

Randomization
tests data,R. A. Fisher Referring theuse of randomization in analyzing to beyondthe have no justification "conclusions once claimedthatstatistical at factthattheyagree withthosewhichcould have been arrived by this test, (75). In a randomization thechanceof type1 error elementary method" random by under nullhypothesis the the (i.e. p-value)is determined repeated is the of levels.Thep-value simply proportion assignment thedatatotreatment in at ofall dataarrangements yielding statistics leastas extreme magnitude test from arrangement (see Figure1). actually observed as thevalueresulting the of statistic is Whenthenullhypothesis that observed the magnitude thetest hypothesis), by is (say) notlarger than wouldbe expected chance(a one-tailed in the values to be counted calculating p-value are those thenthe extreme value. When the null greater thanor equal to the observedtest-statistic is of the magnitude theteststatistic notdifferent hypothesis that observed is countsare hypothesis), separate then (a from chanceexpectation two-tailed and thanor equal to theobserved of valuesless than madeof valuesgreater is by or equal to theobserved; lowerof thesecounts doubledanddivided the the p-value(subject thetotal number dataarrangementsobtain two-tailed of to to theconstraint p < 1). that levels (i.e. the requires poolingall data from treatment Data resampling and then or groups)to be compared established "observed" experimentally to levels, replacement thetreatment data and reassigning randomly without of level the per keeping number observations treatment the same as in the of originaldata. In some cases, all possible redistributions data among in resulting an "exact"randomization treatment canbe readily levels obtained, when the potential numberof different test. In othercases, generally about 5, or redistributions approaches exceeds 104-10 some of these(often for 103) are sampledwithreplacement thetest,whichis thenknownas a randomization test. "sampled" basedon standard statistics test (e.g. t,F); it Randomization areoften tests p is thusthe method resampling data and of calculating thatare of the to potential use special-purpose rather thestatistic than used.Butthe definitive, of or ad hoc statistics a particularly is advantage therandomization important moregenerally), since thismay increasethe approach (and of resampling statistical alternative (64, hypothesis 163). powerto accepttherelevant

408

CROWLEY

nput

Compute Statistic

tatistic As Extreme Or More NO So Than Observed Statistic YES Add 1 To

Counter

Sample
Data

F-ina (*n) YES

N NO

P-Value * Counter Total Divided By Number of Samples (n) Output (P) and tests, Carlomethods, via testing randomization Monte of Figure1 Flowdiagram hypothesis of whenfewsamplesare some types bootstrapping techniques. Exceptforexactrandomization and that program generates processes this implemented a computer using needed, logicis usually programming areneeded, skills a largenumber samples (typically 1000);onlyrudimentary of n n sampling requires sophisticated a unlessthestatistic particularly is complex thedatastructure or This See (64), Noreen(186), and Manly(163) forexampleprograms. algorithm. Edgington is of modification one on thecover(and Figure1, p. 51) of E.S. Edgington's diagram a slight book(64).

The basic rationalefor randomization methodsis thatunderthe null hypothesis for of, example, difference no between treatment-level means,any ofthe possible distributionsdataamong of treatment is equally levels probable. This equiprobability assumedto follow from(i) randomsamplingof is populations beingcompared (contemporary applications generally avoid this but assumption, thosethatinvokeit are knownas permutation tests),(ii) randomassignment experimental of unitsto treatment level, or (iii) for nonexperimental studies, simply taking datato be "exchangeable" the among levelsin theabsenceof treatment effects 163, 250). However, (see random-

STATISTICAL RESAMPLING METHODS

409

izationtestsof differences among means are sensitiveto differences in variances other and moments (e.g. see 15, 219, 242; contrary assertions to in 64, p. vi, and others), implying noneof theenumerated that is assumptions strictly sufficient. lieu ofdefensible In a is alternatives,sufficient assumption that observed of distributionsdataareidentical for except thefeatures actually compared thetest.See Table 1. in For example, considerthe studyby Loreau (156) on temporal niche in in differentiationcarabidbeetles.Loreau was interested whether species shifted their of periods activity suchthat nicheoverlapor another seasonally indextermed "meancompetitive load" is reduced, might expected as be from competition theory. dataconsisted biweekly The of activity levelsby species and habitat (correlated withsuccessional here stage)overa four-year period. Subject to some constraints the timing peak activity on of and on the of boundaries the activeperiodthatwere intended preserve to biological realism, observed the temporal distribution activity of within species was a seasonallyshifted random, at and thetwo indiceswerecalculatedforeach species-habitat combination. each speciesandhabitat, procedure For this was repeated systematically by exactrandomization) (i.e. whenthetotalnumber of distinct was withreplacement reorderings less than4000, or randomly of reorderings bysampled (i.e. randomization) times 2000 when total the number of distinct exceeded4000. The p-valuesweredetermined in as reorderings Figure1. Resultsdiffered amonghabitats, amongconstraints imposed,and among considered. thebeechwood pinewood In and speciessubsets habitats, p-values for both indices or tended approach achievestatistical to response significance as increasingly severeandrealistic on constraints imposed therandomwere izationprocess. In thesecases, nicheoverlapand mean competitive load calculatedfromthe originaldata were lower than 95% or more of the from random in corresponding values generated the seasonal shifts activity It to howthese pattern. is difficult imagine hypotheses couldhavebeentested with these data by standard methods.This example and the statistical debateovertheinterpretationsuchcarabiddata (53, 266, and of continuing their references) indicate someof thechallenges can arisein attempting that to operationalize nullhypothesis specify mostsuitable the and the randomizationalgorithm. the of Nevertheless, overallpattern statistical significance in thepresent does suggest distinct nichedifferentiation study increasingly fromsuccessionalto "climax" beechwoodforest, would be expected as to according competition and theory someother possibleinterpretations.

MonteCarlo
In MonteCarlomethods, particular a random coinflips process (e.g. binomial ora complex stochastic simulation the model)is assumed underlie observed to data in determining confidence or underthe intervals theexpected response

410

CROWLEY

E m

sQ

sQZ

Z ZZ

Zs

:g .s

>z

C~~~~

> Z ZZ Z

vX

Zz

>

9X

<

8.

'

? ~~~~~~~~~~~~s

4)

Z 't E

E Q

't 3

)2 O 't H c; u

STATISTICAL

RESAMPLING

METHODS

411

40.

10

0 cr,
0

.0 v 0) .0
0

CZ

CZ

CZ C's
03

o
j--Q .

cd
CZ cn 0

03
cO

cn

W cli 0 53

= U CZ
O w = 0

C's
C'S cn

to 'o 10 co w
C's

>
C,3

0 4. SZ.(n 0
tn

'ei 0)

0 cn

a 0
w

et 0 u

Q)

10 0
-=1 U 0

an) 4 C13 c

C'S St w

V)
WO

0 a) &

z
0

w
0

C's
0

r. s oo
C13
40.

w '6

SZ to0) a) 'O Iz: 0 = 0)

0

a) 0 Cd

0 0. r
>,

C,3 00 W

C,3U

10 >. 'nosot 0. E C,3

cd w w C's > oi 03

10 CZ .0 o w 0)

C's 4.

a) cw,3

=1

(n cn
C,3 J-W

0
C 14 CZ 0 c -6 0)

03

4;;4

a)
Iz

CZ Ei

c,3

0) O) cn0 0 .- 0)

03

CZ

10
> >

w a) 0 CZ CIS
CIS

0 = > U Ir. oi w C's 0 o1-r. ,

- CZ .

WO
w

.4 > CZ
C'3

03

0 -c

C13 - U Q)

CZ

.0 'O

r. " CO

03

V)
0

0 0)

U m

1-1 tn .

C,3
QL.

bo
o -

0 w 0 =0 C'S 0 W CWI C,3 0 o

03

Cd s

CZ 0 ar = U c: 7:1 r. E-4 Z O 'V cr U

Cd >,

4m.,

0U

"nn w 0 cd U
O

X 7R 0) tn > m 0 Al '-3 '-n 1- - U O.,

cr =
10

.2 C; I CZ
W O =

ct

42 4-4

s W- 0

0 U.

CZ

4.

cl
2

CZ
0

= 0 6 g w .0 w 0 w V E

0 cn

.- QJ
0. 0. .-

a, b

cn W

.-

W
-n 10

0 'a ;G
W

11 O c CZ 0 U 0 CZ
>, =

'c 'O'
cl

4 cwl w
0

g -5. CZ c- x

W
cr W 0 a) I!.,2 : .1.2
W

0 :2
cd

CZ

412

CROWLEY

processis thensampled testing. This random for nullhypothesis hypothesis modelis run or coinflipsaresimulated thesimulation repeatedly (e.g. many testing calculated eachcase. Forhypothesis in with statistics test many times), -or"heads"or"tails"? likely produce to coin (e.g. is a particular unequally significantly themodel'spredictions?), from do empirical observations differ exactly of distribution teststatistics from frequency this thep-valueis found is (see as for randomization Figure 1). In fact,randomization generally considered be thespecialcase of MonteCarlotestsin whichtherelevant to values of randomprocess simplysamples the distribution test-statistic levels of rearrangementsdata amongtreatment associated withequiprobable arise and in somedifferences assumptions restrictions in (163). Nevertheless, moretypical MonteCarlotests(i.e. thosein whichtheobserved comparing tests the process)and randomization data are notused to implement random (see Table 1). intervals Monte Carlo methods confidence are oftenused to generate with and usually cumbersome (whereasthis is possible but uncommon this not difficult, see randomization-e.g. 163,p. 18-20). Though particularly testing because it requires morecomplexthanhypothesis is procedurally of valuesofthestatistic and ordered arrays extreme accumulating maintaining (see as to (corresponding the tails of the distribution) theseare generated method";more "adventurous" the Figure 2, whichillustrates "percentile e.g. in 67 and in methods, withclearadvantages somecases, are described see 12, 82, 85, and their workon Monte Carlo methods 69; for recent references). Leptogorgia Considera demographic studyof the colonial gorgonian to and Monte Carlo methods analyze matrices usingprojection virgulata of Field measurements recruitment, (95). population growth time-varying for size and supplemented colony growth, survival five classesover24 months, 23 projection fecundity wereusedto construct 5 x 5 monthly data, by other in of number individuals the the represented expected matrices. Each entry froman or row size class thatarose by survival,growth, reproduction this before. Multiplying matrix individual thecolumn in size class one month in of the representing numbers individuals each of the by a columnvector yielded of by projected thematrix fivesize classesat thebeginning themonth in later. thenumbers present each size class a month in The less complexof twoMonteCarlo applications thepaperconcerns contributions recruitment, of by determinations elasticity (i.e. proportional and survivalrates to population rate) using the matrix growth growth, the in was whether The of study techniques. question interest thegorgonian form to in simply thegeneral observed patterns thedatacouldbe attributable of thematrices thantheir biologicaldetails.If so, thenan arbitrary rather in should generate of distribution nonzeroentries the matrices elasticity

STATISTICAL RESAMPLING METHODS

413

(Bootstrapping)

anput

Begin

(Monte Carlo)

Sample

NO

Compute Statistic

Value Within

New ~~Order

Tail Array

Sampl

Final

YES

full?

Array Taitr

NO

Delete Least Extreme Value

ai>

NO Statstic More Extreme ThanLes Extreme Tails In Confidence Limits Least Extreme Values
In Tail Arrays

Figure 2

Flow diagram for constructing confidence intervals via Monte Carlo and bootstrapping to the percentile method. This requires a computer program that uses "tail arrays" to collect and reorder the smallest and largest values of the statistics generated. If the confidence according level of interest is 100(1 -a)%, where a is the corresponding significance level, and n values are computed to estimate the confidence interval, then each tail array for two-tail limits will contain 1 + otn/2 values (ignoring any fraction). Thus if n = 10000 and a = 0.05, then the tail arrays ultimately hold the 251 largest and the 251 smallest values of the statistic. Initially, the two arrays are filled by the first 502 values, such that the larger values are ordered in one tail and the smaller values are ordered in the other. Then each subsequent value smaller than the largest in the lower tail or larger than the smallest in the upper tail is ordered within the appropriate array, and the least extreme value is eliminated; intermediate values, insufficiently extreme for either tail, are

not stored. After all n values have been calculated and ordered appropriately, the interval defined by the least extreme value in each of the two tails is the confidence interval. One-tailed confidence intervals are handled similarly. Suitable programs are provided by Noreen (186). With bootstrapping, substantial bias can result from this straightforward approach and 69 for some ways of dealing with this potential problem) in some cases (see 67

patterns vital and rates statistically indistinguishable those from observed. One random wereconstructed the same zero with thousand matrices projection elements in thedatamatrices, with nonzero as from but all elements drawn a uniform distribution ranging fromzero to one. (Note thatif the nonzero elements repeatedly had from beenrandomly scrambled, rather thandrawn a

414

CROWLEY

particular statistical then distribution, thiswould have been classified a as randomization test.) Some of the observed pattern indeedseem to be did mimicked therandom by matrices; MonteCarlo testsshowedthatthe but gorgonian recruitment population and growth rates usedas teststatistics were significantly atypical vitalrates of derived from random the matrices. the So tests helpedseparate general features suchmatrices of from system-spethe cific information contained the data. As in the carabidrandomization in exampleabove, itis difficult see howanyapproach to other than resampling couldhavebeen, useful here.

TheJackknife
Thejackknife, itsall-purpose like namesake, intended offer was to crudebut effective assistance whena more precise toolis unavailable (252). It provides systematic methodsof resampling actual data using relatively the few calculations thatcan often done efficiently a calculator. be on The direct results thesecalculations an "improved" of are (i.e. less sampling-biased) of estimate some sampleparameter (e.g. mean,kurtosis, intrinsic of rate increase)and oftenof the approximate varianceand confidence interval associatedwiththeestimate. The confidence interval sometimes is used in hypothesis tests (occasionallythe jackknifed data themselves, known as are pseudovalues, used; see 114 and 180 forreviews thejackknife). on Though higher-order versions be mayoccasionally useful (e.g. 194, 195), thefirst-order is jackknife by farthemostcommonly used and proceedsas follows: Supposethat parameter interest (e.g. thetrue the of K standard error of the meanfortheunderlying normal distribution means)is estimated of appropriately thewholesampleof m observations k. Pseudovaluesi over as associated with eachobservation then i are as obtained Ki = k - (m - 1)(ki- k), wherek.iis just thestandard parameter calculation withtheithobservation from sample. deleted the Theexpression theright-hand ofthis on side equation is thesampleparameter estimate minus bias term, a reflecting deviation the ofthei-deleted estimate i from fullsampleestimate The meanof the k the k. R pseudovalues is thenthejackknife estimate K. The difference - K of k measures overallsampling the bias of theoriginal estimate (bias can, for k example, distort of estimates populationdensity, when the particularly individuals strongly are in clumped space; see 67 for derivations theabove of relationships). the correlations Ignoring necessarily present among the pseudovalues, s2 their calculating variance in theusualway,anddividing thenumber by of observations thengenerates variance Im of thejackknife the s2 estimate K. Now the assumption suchjackknife that estimates based on normally are distributed yieldstheparametric error confidence interval theestimate: for k

METHODS RESAMPLING STATISTICAL

415

? t*s/m1/2, criticalvalue of the two-tailed where t* is the appropriate of with t-distribution m-I degrees freedom. thatthe correlations requiresassuming Note thatthe varianceestimate in but are amongpseudovalues unimportant, generalconditions whichthis assumption, be (163). The normality might valid have notbeen established for LimitTheorem large samplesizes, is by justifiable the Central though whereit is more applications difficult evaluatewiththe smaller-sample to dubious.See Table 1. successof feral pigeons of intervals reproductive and In a study interclutch (131) & of at on nesting a building theUniversity Kansas,Johnson Johnston and selection parameters between three to the used thejackknife test relation in birds wereincluded thestudy; features. Over600 banded four morphological times weekover per several wereobserved activity and survival reproductive of a 17-month periodand associatedwithmeasurements body mass, tarsus of parameters interest selection The three bill and length, length, bill width. (i), differential theslope of selection directional herewerethestandardized stabiliztrait; on fitness regressed themorphological thestandardized relative disruptive value indicates (C), differential forwhicha positive ing selection and selection; thestandardstabilizing valueimplies and selection a negative of multiple regression from (,(), gradient resulting selection ized directional variables.Thoughthiswas morphological on fitness thecombined relative significantly considered were stated, parameters presumably the notexplicitly interval failed to include zero, confidence nonzerowhen the two-tailed above. for that the the following procedure constructing interval is outlined differential (i) selection directional a significant Resultsindicated highly selection directional andgradient forfemale bodymass,and a significant (,() Thus selection. here for bill interpreted as correlated differential female length, female "targets" interval apparently selection related interclutch to fecundity distribution for of statistical bodymass.In thisexample, parameters unknown and and onlythejackknife bootstrap their overthewholedataset, calculated for required hypothesis the variation closekincouldreadily estimate sampling tests.

TheBootstrap
is rapidly (66) that stilldeveloping is technique Bootstrapping a quiterecent in literature 57, 68, 77). Like (e.g. much andattracting attention thestatistical the focuseson resampling bootstrapping and randomization thejackknife, (in they patterns imply fact,results actualdatato revealsomeof thesubtler the by closelyapproximated thosefrom are from bootstrap often the obtained viewedas is the jackknife-66). Here,thebasic notion that datathemselves, the a frequency represent bestavailableimageof thefrequency distribution,

416

CROWLEY

distribution which from they weredrawn. Thusthebootstrap metaphor refers to thesense in whichthedata are used in their own statistical analysis.To bootstrapconfidence a interval a statistic mean,skewness, species for (e.g. or diversity) calculated from singledata setof m observations, example, a for simply requires random m draws with replacement sample per from original the data,calculating statistic, repeating process the and the many times according to the schemeillustrated Figure2. Here again, thissimplyspecifiesa in random particular processthattechnically a represents special case of the Monte Carlomethod. Bootstrapping also be usedinhypothesis can testing (e.g. 49, 76, 100); for example,withdata from each treatment (or data set to be compared) level sampledwithreplacement separately, testscan be formulated according to the extentof overlapbetweenconfidence intervals by combining or the bootstrapped samplesto calculate teststatistic in Figure1; see 163, p. a (as 28, and 186, p. 80). Thiscan stillbe considered specialcase of theMonte a Carlo method, sinceseparate but random processes used to generate are the separate samples from which comparisons made,theapproach quite the are is different theusualMonte from Carloapproach. is Bootstrappingdistinct from randomization, which redistributes original the datasetovertreatment levels, and it contrasts withtheparametric less computation-intense and jackknife approach. keeping sampling By the processseparate between compared the treatment levels, bootstrapping shouldbe less dependent thanmostother statistical methods similarity underlying on in distributions statistical among treatment levels(B. F. J.Manly,personal see communication; Table 1). In an extensive of study predator parasitoid and selection pressure gall on size of thegoldenrod fly gall Eurostasolidaginis, et Abrahamson al (1) used bootstrapping avoidproblems to withnon-normality correlated and samples that arosein previous analyses.Selection intensities gall size attributable on to natural enemy attack werecalculated thedifference as between mean the of galldiameter theselected individuals the and population-mean diameter, gall divided thepopulation standard For and by deviation. eachof 20 populations twomortality sources (i.e. insects birds), observed and the of number linked observations size in mm,survival (gall from relevant the as natural enemy 0 or 1) were sampledwithreplacement from original the data, the selection was and intensity calculated, this was repeated 1000times generate to process a two-tailed confidence as 2. selection interval, inFigure Anobserved intensity was considered if nonzero itslowerconfidence limit was greater significantly thanzero. Two selection if intensities considered were different significantly their 95% confidence intervals notoverlap;thislatter did wouldbe a very conservative approach hypothesis to testing, except the that p-valueswerenot for of adjusted thelargenumber comparisons implied. The manysignificant selectionintensities imposedby insectswere all

STATISTICAL RESAMPLING METHODS

417

positive, thefewsignificant and selection intensities imposed birdswere by negative. Takentogether, natural enemies therefore to generate seem stabilizingselection gall size,though parasitoid on the effect predominated, resulting in someoveralldirectional selection increasing size as well. for gall

RECENT APPLICATIONS IN ECOLOGY AND EVOLUTION Overview

To determine randomization, how MonteCarlo,jackknife, bootstrapping and methods currently I are beingused in theliterature,conducted searchof a a large computer database(BIOSIS Previewson-linedatabase, 2100 Arch Street, the Philadelphia, Pennsylvania 19103-1399 USA) for publication years 1985-1990 and directly examinedall issues of thejournalsEcology and Oecologia for1985-1991. During period interest, the of approximately biological 9000 journals were being abstracted BIOSIS, apparently by including major all journals ecology in andevolution. Searching titles, abstracts, keywords and yielded references 391 from journals 11 chapters 154 and from books,oncethefewobviousmistakes were eliminated a directscan of the abstracts. reliedon the BIOSIS I by schemeto draw appropriate classification distinctions betweenreferences classified "ecology,""evolution," both.It is possiblethattheMonte as or is Carlo category somewhat inflated relative theothers, to sincetheterm is ofsimulation sometimes for wider used a range methods justthestatistical than of of techniques interest here;butthedirect examination journalsindicated that wouldbe minor. anysucheffect of in Some results the computer searchare presented Table 2. All four methods wellrepresented therecent in resampling are with literature, Monte in Carlomethods overall about twiceas frequent bootstrapping, as which turn was almost twiceas common either tests as randomization or thejackknife. The hypothesis resampling that methods becoming in are morecommon the literature corroborated was statistically, this though evidenceto support the foranyparticular method moreequivocal(randomization, was MonteCarlo, or see bootstrap) clearly contradictory (jackknife) (tested randomization; by Table 2 and Appendix Each of thefour 1). methods (and all taken together) was used disproportionatelyevolutionary in studiesrelativeto ecological studies(MonteCarlo tests;see Table 2, Appendix1), as suggested the by observed v to proportionsof evolutionary studies = 0.144-0.360) relative (v theoverall in proportion all papers published (0.098). I scanned methods results the and sections all figures tablesofthe and and 1485 articlespublishedin Ecology and the 2128 articlespublishedin

418

CROWLEY

1.0

s*

Ico

o0

o-~~~~~~~~~W W)

r-

mt

~ ~ ~ ~ ~ ~ ~ ~~~~I

Z-''

En

r-

cn W)00

WS

Gi

STATISTICAL

RESAMPLING
1 'I','00

METHODS
10, -

419

00

0-so -o =
c=q 0 0 OQ
A

ta B E :5 'tt :,Z, =wg z

2:..00 o, =0 0
o

0 > O.-

-0

:1 'o 0- All - - -0 C, 00 0

u t -0
0

0 O

00

1-

cr O= O==cw Z 'O ro, 0 00.E' Z - - ' Al O, 00 '.., O,,r- > .0 r 0O 0 CZ orb =0 0 Cd A 0 0 0 rIO r>., u CZ. r.
"n 40, 0. m0 m CZ
4..

Q.

00 \0
'Tt

C14

r S= 0) :a Z : C, .
0

8) R

.2 0 0
>

CZ

-O

O O 'O
cz,

O
W) 'No CZ

O =O

cl

-Al

.0 0 A co
O 5 O.,0to

40.

'O

0r0 w '=

Ca

00 ,6

cd

0,_=

00 rkn

00

'No r-

E 00 0 -O w Z 0 Z!

(3

.2

cul

.00

.0

01

01, W) 00

ON

00 O
7s

CZ u 0

0 O >

-0 rCZ
>,

'01 u O 6 c: r.>
E-

ta V
, 1:4

C)

C, ,O >,.4 .- V

00

00

W)
00

0
O

N E > Q 0 oo CS u

w C) cr
4-i

,O r. 0 O.,

>

cl 0 -6

-m 0 cr 0

0 O oo'13- >, 0 u

E U.

tn
o

cn

0 0 0 0

ob
0

C) -t:; -,J0 m c7" ',z 1z) = 42, C\ cul2 O.,0 Qu en 0 U ou oo

0 0) 0

B mmw .0 t lo .-

St

cr

" o
0 O.
.-

0 in.
J:L, O O

0
0

.-

:1

ts

0 0 U .- =- -,= - O C's t -

to
0
0

0 C'S
O

=>

O oi

65 Q

$. oo

C's

0
0 0 ""g : ;-,,O 63 -0

>
0

"a 0
g

C's

79

> r. Z
> u

0 w
2
03 ;a .ou

o 00 0
C-13 o-

0 0 o

wo, 2

420

CROWLEY

were 1270 articles 1985-1991. (For 1985-1990only,there Oecologia from in Ecology and 1841 in Oecologia.) Regardlessof the authors'original random Carlowhen mathematical a methods Monte as designation,classified I variation an of repeatedly generate estimate biological to process executed was Figure 1. following interval) to testa hypothesis, or (e.g. a confidence testswereexcluded(exceptas classified binomial as Analysesspecifically tests to though wereequivalent exactrandomization (see all explicitly noted), below). During1985-1990,87 or 6.9% of theEcologypapersand 53 or 2.9% of techniques. of theOecologiapapers included ormore thefocalresampling one there for (In Ecology,allowing the 8 paperseach using2 of themethods, In and were31 randomization, MonteCarlo, 15 jackknife, 16 bootstrap. 33 therewere 21 Oecologia, with 3 papers each using 2 of the methods, The BIOSIS and randomization, MonteCarlo,5 bootstrap, 8 jackknife.) 22 inefficient overall(17/87= 19.5% forEcology, searchwas thusrelatively differed and 10/53= 18.9% forOecologia,) and theefficiencies probably methods surprising, statistical since amongmethods. (Thisis notparticularly in or warrant mention thetitle,abstract, key words,though may notoften than The overallnumber to somemaybe morelikely be mentioned others.) of ecology/evolution articles during 1985-1990thatused theseresampling of identified as number articles roughly estimated thetotal methods be very can twoefficiencies expressed divided themeanofthese by bytheBIOSIS search as a decimalfraction, whichto thenearest integer equals 2036. Clearly,a would material and reviewof thisand morerecent thorough comprehensive both reviewers readers. and be overwhelming, for data the Combining full 1985-1991 direct-examination withthe 1985Table and the generated 1990 BIOSIS results classifying papersby content 3. Noticethat issuesin ecologyandevolution someparticularly controversial phylogeny) detecting density dependence, size-ratio theory, (e.g. nullmodels, the arewellrepresented perhaps to flexibility here, mainly exploit considerable modelsandtest in nonstandard involving ofresampling methods applications as This can blessing, however, I notebelow. statistics. flexibility be a mixed of and behavioral ecologyin thetable is The poor representation behavior the of from ecologyand artifactual, reflecting separation behavior probably evolution within BIOSIS.

SomeActive AreasofApplication
NULL MODELS, COMPETITION, AND COMMUNITY STRUCTURE

Contemporary structure led in community as underlying interest competition a mechanism of in the 1970s to the formulation null (or neutral) models, withwhich couldbe conducted 223, 229). Since statistical ofpredicted tests (35, patterns

METHODS STATISTICAL RESAMPLING

421

distribution the can modelthat generate statistical of thenotion a probabilistic is witha nullhypothesis theessenceof MonteCarlo hypothesis consistent exactrandomMonteCarlotests(e.g. 201, 223) and closelyrelated testing, term to usingthebroader referred them (e.g. 50 and247, which ization tests that controversy analyses. The in tests") weresoonprominent these "binomial (42-44, 229, for thispurpose null thoseformulating models between erupted (41, 56, aboutthisapproach less 247) and thoseconsiderably enthusiastic null for tale: thepotential differing models, 91-93) providesa cautionary interpretaand errors, alternative procedural of methods, misunderstandings (cf methods MonteCarlo with resampling can results be high tionsof similar seems the in tests 43, 44, and93; see 262). Nevertheless, nullmodelapproach via literature, partly MonteCarlo methods to have takenhold in therecent (Table 3; see 202, 265). testsfor to potential use resampling untapped There is muchrelatively 124, (e.g. guildstructure: 129, 265) and organization community detecting environmental testing whether (118, 263, 264), andfor community similarity (29, structure 83). In assessingspecies factors accountforcommunity can has the with theseestimates, jackknife associated and diversity thevariation (e.g. 23) applications (e.g. 110, 194, 195),butbootstrap often beenusedmost maybecomemorecommon. of the controversy addressed distributionspecies has Muchofthenullmodel on abundance presence/absence islands(214, 244, and analogoussituaand in tions considered 265). A good overviewof this issue and problems is on with constraints randomizationprovided associated choosing appropriate byManly(163, p. 233 fF). to (34, attempts testfornichedifferences 109, 162) and Numerous recent methods, particularly to measureoverlap(2, 107) have used resampling nicheshifts cases, temporal (125, 152, 265; see 202). In other randomization (61, 206, and the binomialor exact have been testedby resampling of analysis Santa & test randomization in50). Simberloff Boecklen'sforensic tests resampling (96, 271, andtheequivalent several Rosalia(230) stimulated from in hypothesis of exact randomization 19) of the constant-size-ratio Hutchinson's paper(120). original on from in has recent work plant Considerable ecology focused competition of distortion thepopulation (27, 136, 251) and related immediate neighbors and of size distribution (133, 143, 227). The geometry access to resources (136, to has thusof potential response competition also been characterized citations and 215). These plantneighborhood-competitionsize-distribution in bothfor of methods considered thisreview, involve gamut resampling the for intervals theGini confidence tests(27, 251) and to estimate hypothesis skewness coefficient indicator size inequality; 227) orto calculate of 133, (an (143). byjackknifing

of papersidentified the in topicand statistical method therelevant Table 3 Classification research by of BIOSIS survey(1985-1990) and by directexamination journals(Ecologyand Oecologia, 19851991)'

TOPIC/Subtopic Competition (total) Null models Niche differentiation, overlap& breadth Size-ratio theory nichedisplacement of intensities Niche-shift dynamics interaction & Plantsize hierarchies Plantneighborhood competition Community structure (total) Detecting organization Diversity Community similarity Temporal variability stability and Detecting density dependence Spatialpatterns processes(total) and Dispersion spatialpattern & Dispersal& migration Scale effects Demography (total) Population size of density Vital rates size & age relationships Growth, Stock-recruitment relations Agricultural/fisheries Environmental factors (total) Absorption scattering light & of Air-quality models& indicators Aquaticenvironmental quality/toxicology Lake & stream acidification soil & groundwater Surface, Behavior/behavioral ecology(total) Social organization Foraging Evolution/evolutionary ecology(total) Selectionintensity response & Geneticdifferentiationcorrelation & Mutation rates Morphometric comparisons Phylogeny

Monte Randomization Carlo 31 3 14 1 3 1 4 14 2 2 7 1 7 15 6 5 4 5 2 2 1 0 0 4 0 0 2 1 1 10 3 6 61 0 1 0 3 5 36 12 9 3 1 0 5 16 5 2 2 5 4 33 17 7 3 47 19 9 7 5 8 58 16 10 8 4 13 16 2 7 44 10 2 7 2 1

Bootstrap Jackknife 20 0 4 0 1 7 0 7 2 3 2 1 0 3 1 1 0 10 6 4 0 0 0 10 1 4 3 1 0 2 0 2 49 3 2 0 2 33 17 0 4 1 0 3 1 7 0 6 1 2 0 9 3 4 1 8 7 7 0 0 2 3 0 0 2 0 1 2 0 1 19 4 2 0 3 9

STATISTICAL methods (total) Statistical modeling and of Analysis variance & Regression correlation Mantel'stest analysis function Discriminant analysis Nearest-neighbor & Sensitivity, error uncertainty Poweranalysis & Confidence intervals variance & Bias estimation reduction Grandtotal

RESAMPLING METHODS 41 3 5 7 2 1 0 0 1 0 145 63 2 8 0 2 6 14 6 4 10 330 43 0 0 0 0 0 1 0 31 7 139

423 44 1 5 0 5 0 3 1 16 8 114

and density dependence almost on all located ofthereferences detecting journals of 'Directsearch thetwoecological BIOSIS andfrom both from were references derived methods and The studies. statistical modeling all ofthecompetition from or almost entirely entirely theBIOSIS search. wereobtained topics thedirect for search.References theremaining fewer accumulating but list usinga lengthier oftopicsand subtopics, thosecategories herewerecollected Data shown and thegrand not from subtopics shown, citations the include additional are than fivecitations notshown;topictotals topicsnotshown.Many of thepapersare talliedin morethanone category. totalincludescitations from

of widepotential application, method very randomization An underutilized in analyses,is Mantel's test (119, 166; see the particularly community testsfor in technique and description examples 163 and 165). This flexible entries distance matrices. Typically, two between (ormore)square correlation some alternative measure) in one matrix expressEuclidiandistances(or multivariate features (e.g. in diets),and between (say) speciesin quantitative species(e.g. zeroes a among pattern matrix representpostulated theother may or membership not in the same guild). By randomly and ones indicating the to rows and columnsof one matrix species, recalculating reassigning matrix elements (wherethe off-diagonal corresponding correlation between of thesecorresponding sumof themultiplicative is products teststatistic the the times, statistical this many repeating sequence and matrix elements), then to the dataforthepostulated pattern match distance in theoriginal tendency be can readily assessed(e.g. see 198). pattern in of Another controversy long standing in the concerns roleofdensity dependence population literature theecological for references evidencethatthe (See e.g. 11, 55, 267, and their dynamics. were milestones theinitiation Two important continues unabated.) controversy of of field tests density (70) andtheformulation dependence ofexperimental in temporal to sequencesof methods detectdensity dependence statistical usefulin bothof these methods have proven data (26). Resampling density thelatter 54, 203, 204, 210, 258, 259; an (48, 51, particularly approaches, T. to of application analysis a fieldtestis in progress-D. M. Johnson, H. in P. L. B. Crowder, H. Crowley, preparation). Martin, for aboutthe potential detecting have been expressed Thoughconcerns
DETECTING DENSITY DEPENDENCE

424

CROWLEY

in sequences(84), recently developedmethods, density dependence density methods Pollardet al the by particularly two testsbased on randomization to powerful be (204) andReddingius denBoer (210), appearsufficiently and havebeen on randomization theme useful (48, 51, 259). Recent variations this used to evaluatebias in k-factor analysis(258), to extendthe notionof level detecting density dependence thecommunity (48, 51), and to derive to testable of changes (48, 51). predictions aboutthedirection density and test" The Pollardet al (204) "randomization and theReddingius den the Boer (210) "permutation test"both involvescrambling orderof the growth observed in density measure population (a of changes log-transformed et overthetime for with sequence.Pollard interval) comparison theobserved of al (204) used thecorrelation coefficient between density thestart each at as during interval thetest the interval theassociated and changein density statistic the implies inverse an correlation; randomization (density dependence noted in parametric approach testavoidstheproblems inherent theanalogous in 159 and 239). Reddingius denBoer (210) used thelog-range and between thehighest lowestdensities reachedin thedensity sequenceas thetest and statistic Other teststatistics implies smalllog-range). a (density dependence or in cases (cf the "violation maybe moreappropriate powerful particular in to different be number" statistic 48 and51). Itmayoften helpful use several on are tests test and statistics thesamedataset,sincethetest results sometimes this aboutadequately complementary 259), though mayraiseconcerns (51, overall tests. protecting chanceof type1 error the and SPATIAL PATTERNS AND PROCESSES Characterizing spatial patterns proA in cesses is a majorchallenge contemporary ecologicalresearch. diverse of has array resampling approaches been used forthispurpose.Descriptive of points methods include the distribution sparsely sampled assessing spatial withpointlocations and thespatialareas mostcloselyassociated (e.g. tree autocorrelation and spatial locations-236 and 136,respectively), particularly of testshave been used to detect nonindependence (149, 232). Hypothesis in size in animallocations (237, 248); variations territory (249); differences of (105, an application dispersion amongsize classes,species,and quadrats in of and Mantel'stest);differences association plantdistribution abundance withtaxonomic vs structure (222, also via Mantel's composition vegetation and dynamics between spatialdistribution temporal test);and an association the haveimproved extended classicnearest and (228). MonteCarlomethods neighbor analysis Clark& Evans (39; see 33; 151; and 163, p. 21-23 and of chapter 7). In other limits populations of have been established cases, geographical of scale (4, 122,215) andofenvironmental (220), andtheimplications spatial heterogeneity 215, 222) havebeenaddressed. (4,

STATISTICAL RESAMPLING METHODS

425

In considering insect dispersal processes, MonteCarlosimulation been has used to evaluatethe need fora stochastic formulation predict to dispersal (253). Error associated with estimates thediffusion of coefficient (160, 188) or of theradius patchdetection of (103) has beenassessedprimarily the with jackknife. ticularly MonteCarlo, are now in fairly commonuse to reducebias and determine error associatedwithestimates population of density (111, 173, 181). The "smoothed bootstrap" (226) and randomization (171, 269) tests havebeenusedtodetect density changes, mainly non-experimental in studies. One of thoselastrandomization examples (i.e. 269) invoked approach an known MRPP (multiresponse as permutation procedures-176-178).MRPP, a specialcase of Mantel'stest(163, p. 209), is conceptually consistent with of graphical representations the data and readilyextendsto multivariate problems. With method, this predefined groups (e.g. sites,treatments) be can for tested differences standard from which using statistical distance measures, a test statistic derivedand then assessed by ordinary is randomization procedures (Figure 1). Interestingly, t standard and F testsand common nonparametric are special cases of MRPP, though tests practitioners argue that nonstandard formulations generally are moreappropriate (269). Following comparison jackknife bootstrap the of and methods Meyeret by for al (175), there has been muchrecent in interest measuring testing and in costsof predator differences demographic defense (13, 212, 260) and of increase other environmental factors 139), as measured theper-capita (88, by rateof zooplankton. (See 95 foran assessment temporal of changesin the of from increase derived rate dominant per-capita eigenvalues a matrix model.) for to testsand error estimates methods have also been applied Resampling othervital rates(birth rate: 58; mortality: relativegrowth rate: 37; 153; transmission of an insectvirus:63; manydifferent rates:254), rate vital effort reproductive (86), andextinction (199, 200). rate environmental ENVIRONMENTAL MODELING As theneedfor reliable predictions monitoring steadily of and has a increased,broad range relatively realistic, in modelshas appeared thebasic and especially theapplied in quantitative A of the is ecologicalliterature. focalissuein many thesestudies evaluating are model'sfitto data;forprobabilistic models,MonteCarlomethods often thebestoption havecommonly and beenused. has Withregardto aquatic environments, resampling been applied in models(20, 24, 221) and laboratory tests(205), time-series toxicological of analysis BOD data (197 via the"Bayesianbootstrap"), testing sensitivity oflakestophosphorus of an loadings (28, 150),estimating index water quality
ESTIMATING POPULATION SIZE AND VITAL RATES

Resampling methods,par-

426

CROWLEY

(97), andassessing impact aciddeposition 128, 132). Applications the of (62, in soil and groundwater systems includethoseon soil hydraulic properties (117, 234), estimating runoff 60,99), andmonitoring (10, groundwater quality (155, 185, 240). In theatmosphere, resampling techniques have been used withmodelsof carbondioxideuptakeand exchange(121, 138, 268), for impact assessment radionuclide of fallout (21, 211, 261), and especially in air-quality modelsandindicators (e.g. 25, 101, 108). An important component manyclimate plant-growth of and models,both is aquaticandterrestrial,absorption scattering incident of and solarradiation. MonteCarlo applications particularly are common thesestudies(e.g. 5, in 38, 94). Resampling methods figure promiin nently analyses natural, of sexual,andgroupselection. Examplesinclude theintroductory studies thejackknife thebootstrap of case and earlyin this review(1, 131), studies determining magnitude sexualselection the of (172, 187) and group selection(78, 102), and othersconcerned with various responses selection to (142, 182, 257). Rates of evolution have been assessed and contrasted resampling via applications (78, 90, 140), as have mutation rates(79, 106, 189, 190) and of of evolutionary implications genetic drift (189, 190, 217). In an analysis taxonextinction rates,Raup & Sepkoski(209 and references used therein) randomization teststo identify of significant periodicity major extinction eventsin the geologic record(also 116; see 116 and 193 on speciation but is periodicity), Quinn(207) argued that bootstrapping moreappropriate forthispurpose in (see 16 andtheoverview 163, p. 192 ff). All four resampling methods havebeenusedtodetect genetic differentiation between populations based on immunologic (225), electrophoretic 59), (45, and nucleotide-difference 246) data. Discriminant function (216, analysis, withthe help of Monte Carlo (225) or randomization particularly (238) methods, wove useful suchstudies. can in
EVOLUTIONARY PROCESSES AND RATES

Phylogenetic has analysis evolved rapidly sincethe1970swith widespread bothofmolecular use and and techniques ofcomputer simulation dataanalysis. Pioneering simulation studies Raupetal (208) demonstrated by the possible importance stochastic of biases in processesand potential on interpreting phylogenies. Some oftheearly work nullmodelsemphasized data or test biogeographic (e.g. thebinomial exactrandomization of247; also see 41 on the avoidableand unavoidable biases in such studies),and the usefulness statistically of contrasting proposedphylogenies againsta null is morewidely pattern becoming recognized see thebootstrap (73; approach of 89 and an exactMonteCarlomethod 233). in
PHYLOGENY

METHODS RESAMPLING STATISTICAL

427

for of was development theformulation techniques recent An important over groups(jackknifing for intervals monophyletic confidence establishing 72; over taxa: 146; bootstrapping characters: see 196 for a comparative used evaluationof these and relatedmethods).Felsenstein'sfrequently matrix. in character contained a species x character data analyses approach of of (or, samplesof characters strictly, thecolumns species-speBootstrap alternative are used to construct characters) cific values for particular group a of trees;thepercentage thesecontaining monophyletic phylogenetic the data thenestimates confidence in present thetreebased on theoriginal that assumptions characters Inherent that groupis indeedmonophyletic. the about raise independently someconcerns and randomly evolved weresampled applicahave validity 73, 224) butapparently notdeterred (72, themethod's (e.g. 126, 127, 270). approaches tionsof this and relatedbootstrapping of studies phylogeny resampling in clearly predominates recent Bootstrapping phenetic (e.g. 47, 145,147) andthe analyses both (Table3), including cladistic studies above. emphasized patterns particular concernswhether interest Another issue of current Here,randomnonrandom. data from phylogenetic canbe considered derived values amongspeciesto character izationtestshave been used to scramble significantly data the derived from original required whether tree the determine than the trees derivedfromscrambled fewerevolutionary step-changes in was values(nonrandomness detected 6 butnotin 7; also see an character in cladistic analysis 192). analogous

Methodology Statistical
methods of the continuing development resampling researchin progress extent thanwithother to the Moreover, a greater (particularly bootstrap). to the tends extend methodologeach new application statistical approaches, of Here,I analysis. ical possibilities becauseofthead hoc nature resampling or methods have been used to supplement improve note how resampling or standard statistical methodsand to stimulate enhance new research initiatives well. as assumptions methodsavoid some of the more restrictive Resampling (e.g. see 64, p. 197), analyses and in regression correlation involved standard 30, are (e.g. regression: 81, 170; andthere nowmany applications published known methods correlation generalized correlation: 249,256). Theuseful 123, (MRPP) have procedures permutation as Mantel's testand multiresponse above. in summary and characterized the literature been described already role in becauseof itscentral of deserves specialattention Analysis variance and of assumptions and thedesign analysis experiments becauseofrestrictive
TYPESOF ANALYSIS RELEVANT

and Thereis muchstatistical biometrical

428

CROWLEY

of thatcan proliferate complexity thedesign(191; 64, p. 58 ff;recent with applications include 130, 162, and 168). Help needed fromstatistical with ANOVA is noted below. researchers problems on associated being and via Manymultivariate methods currently revitalized extended are function resampling (e.g. clusteranalysis: 148, 184, 255; discriminant components analysis:241; indirect gradient analysis:14, 29, 80; principal the readily applicable singlesamples to analysis: 144). It is primarily methods in though (MonteCarlo,bootstrap, jackknife) areofinterest thiscontext, that randomization be usefulfordiscriminant function analysis(163, 238). can reviewof multivariate See Manly (163 and especially165) forthorough resampling applications. in standard issuesandapproaches experimental design data and Manyother withresampling methods. Some of themore analysishave been addressed of associated with sampling 103, 256) (2, important these assessing are errors or direct the tests measurement estimating powerof hypothesis (32, 87, (3), the bias (2, 173, 183), and determining 130), determining reducing and appropriate samplesize (22, 154, 164). In empirical of the studies,the need formethods analyzing ecological (31, response large-scale to perturbations 169) has led to some resampling associatedwithintervention analysis(Monte Carlo: 158) or applications here tests: randomized intervention analysis (randomization 32). Theapproach is usually each is and basedon paired systems, experimental one control; one is monitored before after experimental aiid the system manipulated, extensively so thatsome of theseobservations be assumedessentially can independent is assessed). This generalapproachor a (thoughautocorrelation directly are where replicated experiments successor mayprovevaluable,particularly to that be infeasible, additional should taken ensure thenullhypothesis but care (e.g. by transformingreduce to is testedagainstan appropriate alternative or the distributional differences mayconfound that heteroscedasticityother test;see 71 and 242). in In modeling there muchcurrent is interest incorporating or age studies, size structure individuals representing (52) within (36, 174) or explicitly models. Moreover, modelsnow morecommonly population optimization uncertainties that includestochastic elements (e.g. see 161) or parameter In theseandsimilar methods can cases,resampling complicate interpretation. useful characterizing model'sbehavior evaluating in the and prove particularly itsconsistency empirical with (e.g. age structure: individ137; observations ual-based model:157; optimization: 213). The resampling and of to techniques interest hereare all closelyrelated themorestandard
CLOSE RELATIONSHIPS WITH MORE STANDARD METHODS

STATISTICAL RESAMPLING METHODS

429

widely familiar statistical methods. virtue conceptual By of simplicity the and largenumber nonparametric it has spawned, of tests randomization be can considered fundamental thestandard to methods 46, 134). Transforming (17, datato ranks primarilydeviceto reducedata setsto a general is a form that permits construction nonparametric of significance tables,with entries low at sample sizes determined randomization at higher by and sample sizes by or normal chi-square approximations therandomization to results. The first three commonly used testsin Table 4 are examples these;thesigntestis of also a kindof ranksumtest.Fisher'sexacttestand thebinomial (goodnessof-fit) aredirectly test calculated casesofexactrandomization. tests The listed in thetablearejust a fewof themorecommon nonparametric found tests in theEcology-Oecologia sample. MonteCarlomethods generally to derive are used statistical tablesfor tests basedon dataassumed follow to particular such distributions, as t,F, andx2 tests. theMonte In Carlotests interest theactualstatistical of here, distribution maybe unknown, longas therelevant so stochastic process be simulated can to in 1 according thescheme Figures and2. In somecases among Ecology the and Oecologia articles, smaller a of number simulations used to draw was conclusions without formal a test(e.g. 104, 245) or were compared with observations usingstandard categorical (e.g. 55, 141) orparametric tests tests (e.g. 74, 98). Such hybrid approaches may often proveusefulwherethe underlying assumptions be met, in several these can but of cases,thestandard MonteCarlotest might havebeenmore defensible straightforward. and of Applications thejackknife involving hypothesis testsor determination of confidence limits on parametric rely critical valuesand significance tables (see above).Though bootstrap notinherently toparametric the is tied methods, oneareaofactive development known theparametric is as in bootstrap, which thestandard error themeanis bootstrapped thenused in parametric of and analysesas withthe jackknife (e.g. see 67, 186, 231). Of course, the
Table 4 Percentages papers publishedin Oecologia of some common (1985-1991) featuring nonparametric tests, all of whichare (or are equivalent randomization tests. to) Test Mann-Whitney rankcorrelation Spearman Wilcoxonmatched-pairs signed-ranks Fisher'sexacttest Sign test Binomialtest Percentage 11.3 7.1 4.7 2.7 1.3 1.2

430

CROWLEY

asymptotic convergence statistical of sampling distributions the normal on distribution sufficiently samplesizes is implied theCentral at large by Limit Theorem.

DISCUSSION Advantages Disadvantages TheseMethods and of

An attempt sortouttheprosand cons of resampling to methods relative to themorestandard statistical techniques raisesmany issuesofvarying subtlely and complexity (Table 5). Whentheir stringent assumptions met,paraare metric procedures maximize power(i.e. thechanceof rejecting falsenull a hypothesis favor a true in of alternative), a specified for type1 error (i.e. rate chanceof falsely rejecting true a null)(186). Butrarely smallor moderate at samplesizes can all of theassumptions knownor convincingly be demonstrated apply. The conservative to approachis thento resort standard to or nonparametric methods resampling. Nonparametric methods generally are to slightly considerably weaker than of the stronger parametric resampling and methods for several reasons. in Essentially nonparametric all techniques commonuse were necessarily for In designed minimizing computation. somecases,this resulted inherently in lowpower (e.g. thesign test-see 135). Inothers, lossofpower inadequate a or of to protection thetype1 error maybe attributed reducing to ranks, rate data at relatedto ties in ranktests,continuity corrections low approximations frequencies categorical for tests, thepossibility inaccurate or of approximationsin sometablesat intermediate numbers observations of (64). Often, the
methods of Table 5 Key features three of categories statistical Standard Nonparametric Methods Methods Methods Standard Parametric Resampling High(whenassumptions met) Verywidely Widespread Veryhigh Low Moderate-strong (robust somedepartures) to Population Moderate Moderate Widely Widespread High Moderate Moderate Sample Somewhat lower Moderate High Sometimes increasing & Common& increasing Moderate High Weak-moderate Population (exceptrandomization) Higher& decreasing Low-moderate

Feature Statistical power Knownby researchers Acceptance Standardization Flexibility Assumptions (see Table 1) or Population sample Time & effort cost

Conceptual complexity High

METHODS RESAMPLING STATISTICAL

431

(e.g. the constrain analysesby incompleteness available tables seriously or two-tailed only by Friedman's ANOVA-by-ranks), providing one-tailed only or respectively), by significance values (Fisher'sexacttestand chi-square, (mostnonparaindication thep-value'smagnitude of providing a rough only metric tests)(see 64). level as tests In contrast, randomization yieldaboutthesamesignificance are assumptions met(115, 218) but methods whentheparametric parametric non-normethods whendataarefrom parametric mayhavemorepowerthan in about circumstances which (64, maldistributions p.94; 135). Less is known than methods maybe morepowerful and MonteCarlo,bootstrap, jackknife in standard analysis (butsee examples 67). parametric and parametric nonparaAn obviouscurrent advantage usingstandard of by and known accepted editors and is they widely are metric techniques that in use methods nowclearly common are though resampling other researchers, of as well. Yet bootstrapping have been sweptintothe mainstream may ahead of a full, research somewhat evolutionary ecologicaland particularly Bootstrap confidence and of balanced evaluation itscapabilities shortcomings. testing not alwaysreliable(e.g. see 67); some are limitsand hypothesis some ad-hoc with advancesand perhaps methodological familiarity current to couldproveimportant, use shouldbe restricted cases where and checking (163, 186). methods inappropriate are randomization andparametric tests of methods must trade to some off, Standardization flexibility statistical and shiftassociatedwith the conceptually simple extent.The psychological controls and resampling approach,in whichthe data analystnecessarily to formation designing sufficiently a understands stepfrom each hypothesis to an the itself) calculating intuitively test (and powerful statistic perhaps test infeasible experi(186). Otherwise meaningful p-value,can be "liberating" samplingor requiring mentaldesigns (e.g. those based on nonrandom methods. resampling nonstandard variables) becomeavailablewith response other defensible statistics test the Butthis of carries costthat degree versatility conclusions or procedures conducting testitself for the maylead to different thesurvey "nullmodels"above)-or as a worst case evenundermine of (e.g. to of theobjectivity thedata analysis(see 9 and 113). It is thusimportant the of alternative statistics procedures tojustify and and consider range a test the It be choicesmade,ideally before dataareanalyzed. should clearthat the the be alternative wouldindeed supported rejecting by hypothesis appropriate between is the randomization of differences test case null; an instructive for meansbut can means,in whichthenullhypothesis be rejected identical different variances (242). in the The striking differences assumptions underpinning classical and constrain options the methods (Table 1; 186,p. 84-92)necessarily resampling and or overlooked ignored editors, to an extent thatis often by referees,

432

CROWLEY

researchers. Making dubious assumptions should obviously avoidedwhere be possible,butthisshouldbe balancedagainst tacitassumptions the involved in usingsome of themorespeculative resampling methods (particularly the bootstrap jackknife). and Randomization tests(and by implication theirderivative nonparametric to tests) apply only thesamples not themselves, tosomeunderlying population that mayhavebeensampled. Thisobviates needfor the random samplesand forcertain assumptions aboutthepopulation origin, it also lengthens of but theinterpretive from extrapolation theobserved results thegeneral to situation orpopulation interest. of Some suchextrapolationalmost is alwaysnecessary in anycase, andthisproblem of greater is theoretical appliedrelevance than (64). Finally, resampling methods currently require average on more expenditure of timeand effort analysis per than classicalmethods, do largely becauseof thenecessary In is computer programming. fact,therequired programming often quitestraightforward,programs many and for common are applications widelyavailablein theliterature 64, 163, 186) or as shareware. (e.g. Soon, mainframe microcomputer commercial and packageswillbe available(186).

Implications Interpreting Communicating for and Results

In hypothesis the of testing, interpretationresults necessarily hingeson the p-value,oratleaston itsmagnitude relative thecritical to value.An attractive of feature resampling methods thatthedirect is of calculation thep-value obviates the discretedecision-theory distinction and betweensignificant if nonsignificant cleft at critical value. results, sharply arbitrarilya knife-edge to Instead,thep-value can simplybe understood measurethe degreeof between data and thenull hypothesis, the consistency the though classical significance levels (0.05, 0.01, etc) retaintheirutility benchmarks. as the Moreover, directly calculated easiertocommunicate p-valuemaybe much to nontechnical as in decision-makers; notedin theintroduction, a randomizationtestof a difference two means,thep-value is simplythe between of of proportion random assignments datatotreatments givesa difference that in betweengroupaveragesat least as largeas the difference obtained the experiment p. 10). (64, As withotherstatistical for an estimate the parameter estimates, error p-value is desirable, wherethiserror a moderate particularly reflects only number repetitions 1000) for resampling of a method random(e.g. (sampled ization,MonteCarlo, or bootstrapping). thelatter In the 100(1-a)% case, confidence interval well approximated p + ti_,x(p(1-p) X2 withan is In) by infinite number degrees freedom, of of ao where is thesignificance level,tlis thecritical for and valueof thet distribution significance level cX, n is the of number repetitions see 186, p. 34). Notethat boundson theseerror (e.g.

METHODS STATISTICAL RESAMPLING

433

More of the derived from intensity resampling. variation thep-valuereflect on error bounds the it speculatively, maybe possiblein somecases to obtain or usingthebootstrap variation sampling withempirical p-valueassociated I am unaware thishas yetbeenattempted. that but jackknife, little generally require via estimates resampling reports, error In research of (if and the number repetitions fanfare-only name of the method the in moreinformation the methods testing requires applicable).Hypothesis of and specification justification thenull and alternative section,including centralto the Unless the approachis exotic or particularly hypotheses. The are methods unnecessary. resampling references support to presentation, of magnitude theteststatistic, (if number repetitions applicable), of method, appear generally whenappropriate) interval and p-value (withconfidence with of test. parenthetically results a hypothesis

Issues ThatDeserveAttention Methodological

improve, evaluate and remains be donetodevelop, to research Muchstatistical both themeof robustness, theseresampling Underthe general techniques. and examined compared methods needtobe further and parametric resampling (e.g. of in their to non-equivalence distributions sensitivity non-normality, for unequalvariances),and samplesize. Also, whatare the implications other levelsandfor to assignment treatment tests randomization ofnonrandom suchstudies Ideally, sampling? tests and resampling parametric ofnonrandom samplesand the shouldfocuson features typical small-to-moderate-sized of on thanexclusively thecharacrather gathered, wayssuchdata are actually fromwhichsuch samplesmay be of populations teristics large statistical drawn (64). the feasible recently, relatively Until methods becamegenerally resampling that weresufficiently methods constraining theformulaavailablestatistical and has straightforward tionofnullandalternative hypotheses beenrelatively has widened Now thatthe horizonfor these hypotheses unambiguous. willhelp that newanduseful can guidelines be devised considerably, perhaps and teststatistics. to moreeffectively tests to hypotheses practitionersmatch and techniques Manyof themorecomplexdescriptive hypothesis-testing can methods be based parametric assumption-bound traditionally onrelatively as methods refitted resampling (see above and 163, 165). Of effectively that in and are directly importance ecology evolution methods relate particular to commonexperimental designs,like analysisof variance.ANOVA has to from robust departures thestandard been parametric traditionally considered variations not complex though all agree(e.g. 18,243), andmore assumptions, to ANOVA, ANCOVA, MANOVA) can be morevulnerable (e.g. factorial studies with further of violations parametric assumptions 191). Inparallel (e.g. ANOVA and its variations of of robustness, development resampling the

434

CROWLEY

ANOVA; ANOVA; 130, randomization shouldcontinue (e.g. 8, jackknife viewson interlike 162, bootstrap ANOVA), and controversies contrasting actionterms factorial in ANOVA (see 64 vs 163) needto be resolved. on and especiallyon Further statistical research Monte Carlo methods and additional immediate interest bootstrapping continueto attract will in applications ecologyandevolution.

SomeSpecific Recommendations
training in should be part of basic statistical 1. Resampling methods into and At these methods incorporated are ecology evolution. leastuntil this somecomputer-programmainframe statistics packages, willrequire can perhaps less emphasis be placed mingskillsas well. In exchange, on standard methods. nonparametric of 2. Parameter by estimates shouldbe accompanied estimates theassocimethods makeit possibleforthisprinciple atedvariation. Resampling to be very applied. broadly notuniversally) (if a 3. Withsmall-to-moderate skepticism sample sizes, maintain healthy to abouttheappropriatenessparametric Evenfailures reject of analysis. conclusive, as are normality equal variances nullhypotheses rarely and is sample them low at therelevant sincethepowerof teststo evaluate the processthat generated data is statistically sizes. Whentherandom is methods the approach to use defensible uncharacterized, conservative and assumptions. making fewest the strong unverifiable 4. Transformations shouldbe usedto improve equivalence distributhe of the tionsin randomization in essentially same waythattheseare tests used in parametric a analyses.This shouldhelp neutralize potential randomization methodsthathas oftenbeen problemwith standard unrecognized. 5. Where is to equivalence distributionsunlikely holdor to be achieved of by multisample hypothesis testscan be conducted by transformation, randomizaWiththisapproach (termed "bootstrapped bootstrapping. tion" in 186), data for different treatment levels are bootstrapped is In state before teststatistic calculated. ourpresent the independently of ignorance, shouldnotordinarily used whereparabe bootstrapping metric randomization or methods apply. 6. Forconfidence more theoretically intervals hypothesis and testing, other shouldgenerally used insteadof thejackknife. be methods defensible is usefulforeliminating in parameter bias The jackknife particularly estimates of (68), orin cases (67), as a checkorextension thebootstrap the where other methods notfeasible are computation-intensive heavily (67). 7. Where thattheir shouldattempt ensure to experpossible,researchers

STATISTICAL RESAMPLING METHODS

435

of of a sample somepopulation interest. imental units constituterandom this,theobjectis howto accomplish be Thoughit mayoften unclear analysis.Whenthe case for options thestatistical to retain defensible that identify population the to can be made,it maybe useful explicitly randomly has beensampled two comparing or more sampling, hypotheses 8. In theabsenceofrandom be by should tested randomization. distributions samples with equivalent intervals, confidence can Randomization also be used to construct conservative (64). thesetendto be relatively though instead of methods MonteCarlo, or bootstrapping 9. Use randomization, maximizing power when particularly methods, nonparametric standard test is essential.Any nonparametric can be replacedby a potentially test. equivalent resampling but moreflexible powerful otherwise and use of possibleand appropriate, a largenumber repetitions 10. Whenever whenitcan important inresampling (?20000). Thisis particularly tests (e.g. thedataareinterpreted whenthep-valueis near influence way the n 0.05). For randomization, = 1000 and n = 5000 are generally levels, fortestsat the 5% and 1% significance considered minimal 65, 167). (64, respectively hypotheses null define and alternative resampling methods, using 11. When sectionof the thesechoicesin themethods with special care. Justify report. research of should be paid to the applicability assumptions 12. More attention editors,and referees. statistical analysesby researchers, underlying known commonly and methods used, With widely becoming resampling data analysisshould for and standards acceptably thorough rigorous continue rise. to in arising data analysisand the A summary some common situations of in is for with them presented Table 6. methods dealing mostappropriate
ACKNOWLEDGMENTS

of somepeoplewhohave aidedthepreparation this It is a pleasure thank to and Dan Johnson, Chesson, of Peter and review development theideasherein: and Eugene me Nagle forintroducing to randomization bootstrapping; Jerry the and and Manlyforencouragement forbreaking ground Edgington Bryan with BIOSIS andjournalsearches the so effectively; Hudsonfor Ken helping Clare of theliterature; BryanManly,BrianMcArdle,CidambiSrinivasan, in in and Veltman, participants the 1992 courseon experimentationbenthic of by ecology sponsored the University Ume'a, Sweden, for stimulating John Manly,Brian MickCrawley, Dan Johnson, Lawton, Bryan discussions; on for andcommenting themanuscript; and McArdle, ClareVeltman reading his at Mick Crawley letting rummage will through journals;Brenda for me

436

CROWLEY

Table 6 Recommended methods. Purpose Sampling Empirical samplesize' Large Small-moderate Small-moderate Small-moderate Any Any Any Any Any Distribution(s) (or underlying model) Preferred method

Confidence intervals Random and single-sample Random hypothesis tests Random Random Non-random Multisample hypothesis tests Random Random Non-random Non-random

Any Parametric Normal Parametric MonteCarlo Known,butnon-normal2 Unknown2 Bootstrapping Any Randomization3 Equivalent Non-equivalent2 Equivalent Non-equivalent2 Parametric Bootstrapping Randomization
---

'In practice, andmoderate large sample sizesaregenerally distinguished subjectively. ofthe for 2Transformations datacanat leastsometimes the reshape distribution(s) adequately parametric analysis (with random sampling) randomization (with or to tests non-random this difficult demonstrate sampling), though mayprove convincingly. 3Awkward implement, tends yield to and conservative limits. to confidence

and ChaunceyCurtzfornew perspectives null hypotheses; on Production EditorNancyDonhamforgraceunderpressure; Marcel Dekker,Inc., for to permission use a slightly modified version E.S. Edgington's of diagram as Figure 1; JohnLawtonforhosting my sabbaticalyear at the Centrefor Population Biology; and Lillie, Sarah, and Martinforbeing patientand when the crunchcame. I acknowledge understanding funding fromthe NationalScience Foundation grant INT 9014938 and a Visiting Research from RoyalSociety London. the Fellowship of
APPENDIX: HOWTHEDATAOF TABLE2 WEREANALYZED

I used randomization a Monte Carlo methodto test two a-priori and hypotheses the concerning data summarized Table 2. The rationale in and procedure used in each case are briefly described hereas examplesof how thisgeneral approach be implemented. can Hypothesis Thefocal methods 1: considered thisreview becoming in are morecommonly overthelastseveralyearsin ecology used and evolution. The BIOSIS literature search overthesix publication years1985-1990on topicsin ecologyand evolution (Table 2) provides basis fora test.To the assure sizesfor adequate eachseparate sample references method, either fitting or bothof thesecategories a particular for publication year were pooled, yielding publication frequencies eachofthesixyears be tested trend. for to for 1 Hypothesis predicts positive a in trend publication frequency years, over a generating one-tailed against nullhypothesis no trend. chose test the of I the linear regression coefficient or least-squares b slope of publication as frequency a function publication of yearto be theresponse measure.(It

METHODS RESAMPLING STATISTICAL

437

r coefficientforthispurpose that can be demonstrated usingthecorrelation metric the moreover, muchsimpler results; producesidentical necessarily of thecoordinates theithdatapoint,is equivalent xi lxiyi,where andyi are used in mytests-see 163, actually to either theseand was themeasure of b the required calculating observed = 1 Hypothesis then 91-92.) Testing pp. that and equivalent) demonstrating a valueas largeorlarger bo (orthesimpler by to unlikely have arisen chancealone. is very then there sequence, the generated observed null Now ifa trend-free process frequencies in the the specialabout order which publication be should nothing equivalent should producea statistically were observed;any reordering of orderings sixnumbers. are (163, p. 92). There 6! = 720 different sequence for b coefficient(or equivalent) each andcomparing a Calculating regression thanbo. This of the it withbo indicated proportion theseas largeor larger sequencecould the p to was proportion taken be theprobabilitythat observed the produced that process by havebeengenerated thesamekindoftrend-free level significance than relevant the p When was smaller 719 other sequences. (i.e. in of the 0.05), I rejected nullhypothesis favor thealternative (generally 1 I 1); Hypothesis otherwise was unableto rejectthenull,and Hypothesis was notsupported. exactp-valwhich determines randomization, systematic Thisexemplifies in (written TurboPascal 6.0) to calculate program ues. My Pascal computer of b, r, andp was justover 100 lineslong,themajority whichwereneeded have could instead of to generate 720 reorderings thedata. The solution the (see algorithm usinga muchsimpler randomization been found sampled by 30 run lines,and someadditional time(roughly program 163), halfas many 386/387 microcomputer). of sec rather thana fraction a secondon a typical in considered thisrevieware used in 2: Hypothesis Thefocal methods studies. evolutionary in frequency ecologicalversus differing for to this The data of Table 2 permit hypothesis be tested each method. rowsof thetablewereused. Onlythedatain the"ecology"and "evolution" and ecological evolutionary as within classified both years the papers First, few of a eliminatingsource positive from observed the frequencies, wereremoved and alongrows(i.e. overyears), summed The werethen correlation. residuals studiesout of all ecologicaland theoverallproportion of evolutionary vo row as was studies thegivenmethod calculated theevolution for evolutionary row sumdivided thesumof theevolution sumandtheecologyrowsum. by of to (This is of courseequivalent usingthe overallproportion ecological is which simply 1-vo.) studies, 2, Hypothesis I used a MonteCarlo To determine whether corroborated vo withvo. If of for a method generate nulldistribution v-values comparison to the then kinds studies, of in method wereusedjustas frequently both a given as in yearcould have been generated a observed frequencies a particular of a based on thefrequencies all ecologyand process samplefrom binomial

438

CROWLEY

all evolution in studies thatyear.I thusrandomly generated number the of studies observed each year, in with chanceofanyparticular the study's being "evolutionary" equalling "evolutionary" the proportion thegivenyear,and in I tallied distribution the between evolution ecology.Whenthesedistribuand tions beensimulated all sixyears thesequence, overall for had of the proportion v of evolutionary studies found abovefrom evolution ecology was as the and rowtotals. Each of 20,000suchv-values determined, theproportion was and as extreme moreextreme or thanvo becamethep-valueestimate. Though this p-valuewas derived from random of a sample v-values was therefore and inexact, largenumber iterations the of assured adequate precision. (See 163, pp. 32-36. The precisionof thisp-value could be assessed using the confidence-interval calculation described theDiscussion in subsection entitled Implications Interpreting Communicating for and Results.) Special provision mustbe made fortwo-tailed testsof hypotheses like Hypothesis I notedwhether was largeror smallerthanthe overall 2. vo expectedvalue or meanof v, setting computer the program calculatep to from v-values far from for equally orfarther themean.As is typical two-tailed this that level tests, required thesignificance be halvedto 0.025. Thusif499 or fewer thev-values of wereas extreme moreextreme or thanvo (as in all 2 cases in Table 2), thenthe null hypothesis rejected was and hypothesis corroborated. Pascal program calculate andp wasjustover50 lines My to vo andwas straightforward to write; substantial of the number iterations required on about 10-60 sec to run on a 386/387microcomputer, depending the observed publication frequencies themethod interest. for of Literature Cited
1. Abrahamson, G., Sattler,J. F., W. Inversion Monte Carlo model for of estimating vegetation canopyparameMcCrea, K. D., Weis, A. E. 1989. in on ters.Remote Sens. Environ. 33:201-10 Variation selection pressures the goldenrod flyand thecompetitive gall 6. Archie,J. W. 1989. A randomization in of test for phylogenetic information interactions its naturalenemies. systematic data. Syst.Zool. 38:239-52 Oecologia 79:15-22 2. Abrams, Nyblade, Sheldon, P., C., 7. Archie,J. W. 1989. Phylogenies of S. A 1986. Resourcepartitioning comand plant families: demonstrationphyof logenetic randomness DNA sequence in petition shellsin a subtidal for hermit data derivedfromproteins. Evolution crabspeciesassemblage. Oecologia69: 43:1796-800 429-45 3. Althawadi,A. M., Grace, J. 1986. J. T. 8. Arveson, N., Schmitz, H. 1970. Robust procedures variancecomfor Water use by the desert cucurbit Citrullus colocynthis(L.) Schrad. ponentproblems using the jackknife. Biometrics26:677-86 Oecologia 70:475-80 4. Antonovics, Clay, K., Schmitt, 9. Basu, D. 1980. Randomization analysis J., J. 1987. The measurement small-scale of of experimental data: the Fisherranenvironmentalheterogeneityusing domization test. J. Am. Stat. Assoc. of clonal transplants Anthoxanthum 75:575-82 odoratum and Danthonia spicata. 10. Benjamini, Harpaz, Y. 1986. ObY., runoff servational rainfall analysisfor Oecologia 71:601-7 5. Antyufeev, S., Marshak, L. 1990. V. A. of estimating effects cloud seedingon

STATISTICAL RESAMPLING METHODS

waterresources northern in Israel. J. Hydrol. 83:299-306 A. Berryman, A. 1991. Stabilization 26. or regulation: whatit all means.Oecologia 86:140-43 Besag, J.,Clifford, 1991.Sequential 27. P. MonteCarlop-values.Biometrika 78: 301-4 Black, A. R., Dodson, S. I. 1990. Demographic costs of Chaoborus-in- 28. ducedphenotype in plasticity Daphnia pulex. Oecologia 83:117-22 Block,W. M., Brennan, A., Gutierrez, L. R. J. 1991. Ecomorphological relation- 29. ships of a guild of ground-foraging birds in northern California,USA. 30. Oecologia 87:449-58 Boik, R. J. 1987. The Fisher-Pitman permutation a non-robust test: alternative to the normal theory F-test when 31. variances are heterogeneous. Br. J. Math. Stat. Psychol. 40:26-42 Boyajian,G. E. 1991. Taxon age and of selectivity extinction. Paleobiology 32. 17:49-57 Bradley, V. 1968. Distribution-Free J. Statistical Tests.Englewood NJ: Cliffs, Prentice-Hall Br. 33. Bradley,J. V. 1978. Robustness? J. Math. Stat. Psychol. 31:144-52 Brandl, Topp,W. 1985. Size strucR., tureof Pterostichus spp. (Carabidae): 34. Oikos 44:234aspectsof competition. 38 Breck,J. E., DeAngelis,D. L., Van S. Winkle, W., Christensen, W. 1988. Potentialimportance spatial and of 35. in temporal heterogeneity pH, alumisurvival num,and calciumin allowing of a fishpopulation: modeldemon- 36. a Ecol. Modell. 41:1-16 stration. D. T. 37. Breshears, D., Kirchner, B., Otis, M. D., Whicker, W. 1989. UncerF. in of tainty predictions falloutradionuclidesin foods and of subsequent Health Phys. 57:943-53 ingestion. Briggs,J. M., Knapp, A. K. 1991. biomass in 38. Estimating aboveground tallgrass prairie with the harvest method: size determining proper sample and Monte Carlo using jackknifing simulations. Southwest Nat. 36:1-6 Brokaw,N. V. L., Scheiner,S. M. 39. in 1989. Speciescomposition gaps and structure a tropical of forest. Ecology 70:538-41 Brueggemann, Trapp, Matthies, 40. R., S., M. 1991. Behavior assessment a of volatilechemicalin the RhineRiver. Environ. Toxicol.Chem. 10:1097-104 41. D. Brusasca, Tinarelli, Anfossi, G., G., between results the 1989. Comparison of a MonteCarloatmospheric diffusion

439

11. 12. 13.

14.

15.

16. 17. 18. 19.

20.

21.

22.

23.

24.

25.

model and tracer experiments. Atmos. Environ. 23:1263-80 Bulmer,M. G. 1975. The statistical analysisof densitydependence.Biometrics31:901-11 Cain, M. L. 1990. Patterns Solidago of altissimarametgrowth and mortality: the role of below-ground rametconnections.Oecologia 82:201-9 Canale, R. P., Effler,S. W. 1989. Stochastic phosphorus model for Onondaga Lake, New York, USA. WaterRes. 23:1009-16 Capone, T. A., Kushlan,J. A. 1991. Fishcommunity in structure dry-season stream pools. Ecology 72:983-92 Carnes, A., Slade, N. A. 1988. The B. use of regression detecting for comwithmulticollinear Ecoldata. petition ogy 69:1266-74 Carpenter, R. 1990. Large-scale S. perturbations: for opportunities innovation. Ecology 71:2038-43 Carpenter, R., Frost, M., Heisey, S. T. D., Kratz, T. K. 1989. Randomized intervention analysisand the interpretation whole-ecosystem of experiments. Ecology 70:1142-52 host Casas, J. 1990. Multidimensional distribution nonrandom and parasitism: model. a case studyand a stochastic Ecology 71:1893-903 of Case, T. J.1990.Patterns coexistence in sexual and asexual species of lizards.Oecologia 83: Cnemidophorus 220-27 Caswell,H. L. 1976. Community structure: a neutral model analysis.Ecol. Monogr. 46:327-54 Caswell, H. 1989. MatrixPopulation Models. Sunderland, Mass: Sinauer F. R. Chapin, S. III, Groves, H., Evans, determinants L. T. 1989. Physiological of growth rate in responseto phosphorussupplyin wild and cultivated Hordeum species. Oecologia 79:96105 Chuh, H. T., Tan, H. S. 1990. A multiconstituent multilayer and micromodelfor vegetative a wavebackscatter medium.RemoteSens. Environ. 31: 137-54 P. Clark, J.,Evans,F. C. 1954.Distance to nearestneighbor a measureof as in spatial relationships populations. Ecology 35:445-53 W. Cochran, G. 1954. Some methods hi2 forstrengthening common tests. the Biometrics10:417-51 D. Colwell,R. K., Winkler, W. 1984. A null model fornull modelsin bioIn geography. EcologicalCommunities: ConceptualIssues and the Evidence,

440

CROWLEY
D. ed. D. R. Strong, Simberloff, G. L. Abele, A. B. Thistle, pp. 344-59. Princeton, Princeton NJ: Univ. Press Connor,E. F., Simberloff, 1979. D. The assembly speciescommunities: of Chance or competition? Ecology 60: 1132-40 Connor,E. F., Simberloff, 1984. D. Neutralmodels of species' co-occurrence patterns. Ref.41, pp. 316-31 See D. Connor,E. F., Simberloff, 1984. Rejoinder. See Ref. 41, pp. 41-43 K. Corbin, W., Livezey,B. C., HumP. phrey, S. 1988. Geneticdifferentiationamongsteamer-ducks (Anatidae: Tachyeres): electrophoretic an analysis. Condor 90:773-81 J. Cotton, W. 1973. Even better than before.Contemp. Psychol. 18:168-69 of Cracraft, 1991. Patterns diversiJ. fication withincontinental biotas: hierarchical congruence amongtheareas of endemism Australian of vertebrates. Aust. Syst. Bot. 4:211-27 Crowley,P. H. 1992. Density dependence,boundedness, attraction: and Dein tecting stability stochastic systems. Oecologia. 90:246-54 P. J. Crowley, H., Gillett, Lawton, S., H. 1988. Contests between larval damselflies: empirical steps towarda better ESS model. Anim. Behav. 36:1496-511 P. D. Crowley, H., Johnson, M. 1982. Habitatand seasonality niche axes as in an odonate community. Ecology63: 1064-77 P. D. Crowley, H., Johnson, M. 1992. of Variability stability a dragonfly and assemblage.Oecologia. 90:260-69 DeAngelis,D. L., Gross,L. J., eds. 0000. Individual-BasedApproaches and Models in Ecology. New York: Chapman-Hall den Boer, P.J. 1989. Comment the on "On testing article temporal nichedifin ferentiation carabidbeetles,"by M. Loreau. Oecologia 81:97-98 den Boer, P. J. 1990. On the stabilizationof animalnumbers. Problems of 3. testing. Whatdo we concludefrom tests? Oecologia 83:38significant 46 den Boer, P. J. 1991. Seeingthetrees for wood:random walksorbounded the fluctuations of population size? Oecologia 86:484-91 Diamond,J. M., Gilpin,M. E. 1982. Examination the "null" model of of Connorand Simberloff speciescofor occurrences islands.Oecologia 52: on 64-74 DiCiccio, T. J., Romano,J. P. 1989. inconfidence A reviewof bootstrap tervals.J. R. Stat. Soc. B 50:338-54 popDorazio, R. M. 1986. Estimating when ulation birth ratesof zooplankton and hatching rates of egg deposition are periodic.Oecologia 69:532-41 Douwes, P., Stille,B. 1988. Selective processesin the geversusstochastic of of neticdifferentiation populations (Lepthebutterfly ErebiaemblaThnbg. Hereditas109:37idoptera: Satyridae). 44 S. J., Dressing, A., Spooner, Kreglow, P. J. M., Beasley,E. O., Westerman, sampler W. 1987. Waterand sediment forplot and fieldstudies.J. Environ. Qual. 16:59-64 comDuBowy,P. J. 1988. Waterfowl munitiesand seasonal environments: temporal variabilityin interspecific Ecology 69:1439-53 competition. Duff,K. E., Smol, J. P. 1991. Morphological descriptionsand stratiof graphic distributions the chrysofroma recently phyceanstomatocysts Park, New acidified lake, Adirondack York,USA. J. Paleolimnol.5:73-113 Dwyer,G. 1991. The rolesof density, and in stage, patchiness thetransmission of an insectvirus.Ecology72:559-74 E. Edgington, S. 1987. Randomization Tests.New York:MarcelDekker.2nd ed. Edwards,D. 1985. Exact simulationwithaddibased inference: survey, a tions. J. Stat. Comput. Simul. 22:307-26 Efron,B. 1979. Bootstrapmethods: anotherlook at the jackknife.Ann. Stat. 7:1-26 the Efron,B. 1982. The Jackknife, Plans. and Resampling Bootstrap Other SIAM Philadelphia: Efron,B. 1992. Jackknife-after-bootfuncerrors influence and strap standard tions. J. Roy. Stat. Soc. B 54:83127 R. Efron, Tibshirani, 1986. BootB., confor methods standard errors, strap of methods and fidence intervals, other statistical accuracy.Stat. Sci. 1:54-77 R. Eisenberg, M. 1966. The regulation of in of density a naturalpopulation thepondsnail,Lymnaeaelodes. Ecology 47:889-906 A. Faust,K., Romney, K. 1985. The on of distributions matrix effect skewed tests.Br. J. Math. Stat. permutation Psychol. 38:152-60 limits J. Felsenstein, 1985. Confidence An on phylogenies: approach usingthe Evolution39:783-91 bootstrap. from J. Felsenstein, 1988. Phylogenies

42.

58.

43. 44. 45.

59.

60.

46. 47.

61.

62.

48.

49.

63. 64. 65.

50.

51. 52.

66. 67. 68.

53.

54.

69.

70.

55.

71.

56.

72. 73.

57.

STATISTICAL RESAMPLING METHODS

molecular sequences:Inference reand liability. Annu.Rev. Genet.22:521-65 A. 90. Firbank, G., Watkinson, R. 1987. L. On the analysisof competition the at leveloftheindividual plant.Oecologia 91. 71:308-17 Fisher, I., Hall,P. 1990.Onbootstrap N. hypothesis testing. Aust. J. Stat. 32: 177-90 Fisher, I., Hall, P. 1991. Bootstrap 92. N. algorithms small samples.J. Stat. for Plan. Infer. 27:157-69 R. of Fisher, A. 1936. The coefficient racial likenessand the future craof 93. niometry. R. Anthropol. 66:57J. Inst. 63 Fix, A. G. 1984. Kin groupsand trait 94. groups:population structure epiand Am. J. Phys. demicdisease selection. Anthropol. 65:201-12 Foley, P. 1987. Molecularclock rates 95. as lociunder selection. Proc. stabilizing Natl. Acad. Sci. USA 84:7996-8000 Forina, M., Armanino, Leardi,R., C., Drava, G. 1991. A class-modelling 96. based on potential technique functions. 5:435-54 J. Chemometrics E. G. Gallagher, D., Gardner, B., Jumars, P. A. 1990. Competition among the 97. pioneers a seasonalsoft-bottom in benand thicsuccession:fieldexperiments analysisof the Gilpin-Ayala competition model. Oecologia 83:427-42 98. P. Garthwaite, H., Buckland,S. T. Monte Carlo confi1992. Generating dence intervals theRobbins-Monro by process.Appl. Stat. 41:159-71 and 99. Gascon,C. 1991.Population- community-level analyses of species occurrences of central Amazonian tadpoles.Ecology 72:1731- 100. raipforest 46 Gaston,K. J., Lawton,J. H. 1987. A testof statistical for techniques detect- 101. in ing density dependence sequential censuses of animal populations. 102. Oecologia 74:404-10 Gates, J. 1991. Exact Monte Carlo testsusing several statistics. Stat. J. 103. Comput.Simul. 38:211-18 Gebhardt, M., Ribi, G. 1987. Reproin ductiveeffort growth the proand sobranch snail, Viviparus ater. Oecologia 74:209-14 T. Gerrodette, 1991. Modelsfor power 104. of detecting trends-a replyto Link and Hatfield. Ecology 72:1889-92 J. effects Gilbert, J. 1990. Differential of Anabaena affinison cladocerans 105. and and rotifers: mechanisms implications.Ecology 71:1727-40 N. R. Gilinsky, L., Bamback, K. 1986. The evolutionary a bootstrap: new ap- 106.

441

74.

75. 76. 77.

78.

79. 80.

81.

82.

83.

84.

85. 86.

87. 88.

89.

diproachto the studyof taxonomic versity. Paleobiology12:251-68 of Gillespie,J. H. 1986. Variability evolutionary rates of DNA. Genetics 113:1077-92 Gilpin,M. E., Diamond,J. M. 1982. to Factors contributingnon-randomness on in species co-occurrences islands. Oecologia 52:75-84 Gilpin,M. E., Diamond,J. M. 1984. on Are species co-occurrences islands and non-random, are null hypotheses in ecology?See Ref. useful community 41, pp. 297-315 Gilpin,M. E., Diamond,J. M. 1984. Rejoinder.See Ref. 41, pp. 332-41 and Gordon,H. R. 1991. Absorption estimatesfrom irradiance scattering Monte Carlo simulameasurements: tions.Limnol. Oceanogr. 36:769-77 modGotelli, J. 1991. Demographic N. a els forLeptogorgia virgulata, shallow-water gorgonian. Ecology 72: 457-67 N. Gotelli, J.,Lewis,F. G. III, Young, in C. M. 1987. Body-sizedifferences assema colonizing amphipod-mollusc blage. Oecologia 72:104-8 G-Valdecasas,A., Baltanas,A. 1990. of and estimation Jackknife bootstrap of Water index water quality. biological Res. 24:1279-84 in Haefner, W. 1988. Niche shifts J. Antillean Anoliscommunities: Greater of and biological effects niche metric resolution on null model tests. Oecologia 77:107-17 regional Haith,D. A. 1986. Simulated in runoff. EnviJ. variations pesticide ron. Qual. 15:5-8 Hall, P., Wilson, S. R. 1991. Two for testhypothesis guidelines bootstrap 47:757-62 ing. Biometrics Hanna,S. R. 1988. Air qualitymodel J. evaluation and uncertainty. Air Pollut. ControlAssoc. 38:406-12 H., Harpending, Rogers,A. 1987. On for semechanism intergroup Wright's lection.J. Theor. Biol. 127:51-62 disHarrison, 1989. Long-distance S. persal and colonizationin the Bay Checkerspot butterfly,Euphydryas editha bayensis. Ecology 70:123643 A. Harvey,B. C., Stewart, J. 1991. Fishsize andhabitat depth relationships in headwater streams.Oecologia 87: 336-42 L. Harvey, E., Davis, F. W., Gale, N. 1988. The analysisof class dispersion patternsusing matrix comparisons. Ecology 69:537-42 rates Hastings,A. 1987. Substitution

442

CROWLEY
under stabilizingselection. Genetics 123. 116:479-86 Hayward, D., Garton, 0. 1988. G. E. Resource partitioning among forest 124. owls in the Riverof No Return Wilderness, Idaho. Oecologia 75:253-65 Heidam, N. Z. 1987. Bootstrap estimates of factor model variability. 125. Atmos.Environ.21:1203-18 Heller, K.-G., vonHelversen, 1989. 0. of Resource partitioning sonar frequency bands in rhinolophoid bats. 126. Oecologia 80:178-86 Heltshe, F., Forrester, E. 1985. J. N. of Statistical evaluation the jackknife of estimate diversity when using quadrat samples.Ecology 66:107-11 Henle, K. 1989. Populationecology and life history the diurnalskink 127. of in Morethia boulengeri aridAustralia. Oecologia 78:521-32 Hinkley,D. V. 1978. Improving the withspecialreference corto jackknife relation estimation. Biometrika 65:1322 128. Hinkley,D. V. 1980. Discussion of Basu's paper.J. Am. Stat. Assoc. 75: 582-84 D. Hinkley, V. 1983. Jackknife meth- 129. ods. Encyclopedia Stat. Sci. 4:280-87 W. Hoeffding, 1952. The large-sample powerof testsbased on permutations 130. of observations. Ann. Math. Stat. 23: 169-92 Holman,E. W. 1989. Some evolutionof taxa.Paleobioarycorrelates higher logy 15:357-63 E. Hopmans,J. W., Guttierez-Rave, 1988.Calibration a root of water uptake 131. model in spatiallyvariable soils. J. Hydrol. 103:53-66 D. Horvitz, C., Schemske, W. 1990. C. in variation insectmuSpatiotemporal of tualists a neotropical herb.Ecology 132. 71:1085-97 L. Hubert, J., Schultz,J. 1976. Quaas dratic assignment a general data Br. analysisstrategy. J. Math. Stat. Psychol. 29:190-241 G. Hutchinson, E. 1959. Homage to 133. Santa Rosalia, or why are thereso Am. Nat. 93: manykindsof animals? 145-59 Ito,D., Oikawa,T. 1989.A simulation of studyon characteristics light-interand assimilation 134. dioxide ception carbon in a densely field planted mulberry and their variation. Sericul.Sci. J. regional 135. Jpn. 58:131-39 D. H. Jackson, A., Harvey, H. 1989. associations fishasin Biogeographic localvs. regional semblages: processes. 136. Ecology 70:1472-84 D. Jackson, A., Somers,K. M. 1991. of correlations. The spectre "spurious" Oecologia 86:147-51 Jaksic,F. M., Medel, R. G. 1990. recognition guilds:testing of Objective significant speciesclusforstatistically ters. Oecologia 82:87-92 variation James, D. 1991. Temporal C. by in diets and trophicpartitioning lizards(Ctenotus: Scincidae) coexisting incentral Australia. Oecologia85:55361 K. Jansen, K., Holsinger, E., MiR. chaels, H. J., Palmer, J. D. 1991. analysis of chloroplast Phylogenetic DNA restriction site data at higher the taxonomic levels:an examplefrom Asteraceae.Evolution44:2089-105 R. Jansen, K., Wallace, R. S., Kim, K. Chambers, L. 1991.Systematic K.-J., DNA variof implications chloroplast ation in the subtribeMicroseridinae (Asteraceae:Lactuceae). Am. J. Bot. 78:1015-27 Jenkins, A., Whitehead, P. G., T. Musgrove, J., Cosby, B. J. 1990. in A regionalmodel of acidification Wales, UK. J. Hydrol. 116:403-16 Joern, Lawlor,L. R. 1981. Guild A., in structure grasshopper assemblages rebased on food and microhabitat sources. Oikos 37:93-104 D. Johnson, M., Pierce,C. L., Martin, R. T. H., Watson, N., Bohanan, E., C. Crowley,P. H. 1987. Preydepletion evidence larvae:combining by odonate Ecolfrom multiple fieldexperiments. ogy 68:1459-65 R. Johnson, G., Johnston, F. 1989. S. in A multifactorial studyof variation interclutch interval and annual reproductive success in the feral pigeon, Columba livia. Oecologia 80:87-92 C. Jones, L., Minns, K., Marmorek, M. D. R., Heltcher, J. 1991. Assessing K. of thepotential extent damageto inland Canada due to acidic lakes in eastern of IV. analysis deposition. Uncertainty model.Can. J. Fish. Aquat. a regional Sci. 48:599-606 distriT. Jurik, W. 1991. Population butionsof plant size and lightenvironment giantragweed(Ambrosia of densities. Oecologia trifida at three L.) 87:539-50 0. Kempthorne, 1955. The randomizaof inference. tiontheory experimental J. Am. Stat. Assoc. 50:946-67 T. O., Kempthorne, Doerfler, E. 1969. The behaviourof some significance randomization. tests under experimental Biometrika 56:231-48 J. N. Kenkel, C., Hoskins, A., Hoskins,

107.

108. 109.

110.

111.

112.

113. 114. 115.

116. 117.

118.

119.

120.

121.

122.

STATISTICAL RESAMPLING METHODS

W. D. 1989. Edge effects the use in of area polygons study to competition. Ecology 70:272-74 Kimura,D. K. 1990. Approaches to age-structured separable sequential pop- 151. ulationanalysis.Can. J. Fish. Aquat. Sci. 47:2364-74 A. King, W.,O'Neill, R. V., DeAngelis, D. L. 1989. Using ecosystem models 152. to predict regional carbondioxideexchange betweenthe atmosphere and the terrestrial biosphere.Global Bio153. geochem.Cycles 3:337-62 J. Kirk,K. L., Gilbert, J. 1990. Suspendedclay andthepopulation dynamics of planktonic and cladorotifers cerans. Ecology 71:1741-55 154. Kitchell, A., Estabrook, MacLeod, J. G., N. 1987. Testing equality rates for of of evolution. Paleobiology13:272-85 Kjellsson, G. 1991. Seed fate in an ant-dispersed sedge, Carex pilulifera 155. L.: recruitment seedlingsurvival and intests models spatial of for dispersion. Oecologia 88:435-43 W. Knapp,S. J.,Bridges, C. Jr., Yang, M. -H. 1989.Nonparametric confidence interval estimators heritability for and expectedselectionresponse.Genetics 156. 121:891-98 Knox,R. G., Peet,R. K., Christensen, N. L. 1989. Population in 157. dynamics loblollypine stands: changesin skewness and size inequality. Ecology 70: 1153-66 Knox, R. G., Peet,R. K. 1989. Boot- 158. ordination: method esa for strapped timating samplingeffectsin indirect 80:153-66 gradient analysis.Vegetatio T. E. Krantz, S., Theriot, C., Zimmer, 159. E. A., Chapman,R. L. 1990. The and MicromonaPleurastrophyceae dophyceae:a cladisticanalysisof nuclear ribosomalRNA sequence data. J. Phycol. 26:711-21 160. intermal Lanyon, M. 1985.Detecting S. inconsistencies distancedata. Syst. in Zool. 34:397-403 Lavin, M., Doyle, J. J. 1991. Tribal relationships of Sphinctospermum161. of (Leguminosae): integration traditionaland chloroplast DNA data. Syst. Bot. 16:162-72 162. L. Legendre, Dallot,S., Legendre, P., 1985. Successionof species within a community: chronologicalclustering 163. withapplications marine to and freshwater Am. zooplankton. Nat. 125:25788 164. M. Legendre, Fortin, J.1989.Spatial P., and patterm ecological analysis. Vegetatio 80:107-38 B. T. Lesht, M., Fontaine, D. III, Dolan, 165.

443

137.

138.

139.

140. 141.

142.

143.

144.

145.

146. 147.

148.

149. 150.

D. M. 1991. Great Lakes USA-Canada totalphosphorus modelpost auditand analysis. J. regionalized sensitivity Great Lakes Res. 17:3-17 W. D. Lindberg, J.,McElroy, A. 1991. in Limitations nearestneighbortests of dispersion association. and Fla. Sci. 54:106-11 E. Linton, R., Edgington, S., Davies, L. R. W. 1989. A view of nicheoverlap amenableto statistical analysis.Can. J. Zool. 67:55-60 H. Loader, Damman, 1991.Nitrogen C., of content foodplants vulnerability and of Pieris rapae to naturalenemies. Ecology 72:1586-90 D. Lobel,P. S., Anderson, M., DurandClement, M. 1988. Assessmentof populations: Ciguatera dinoflagellate and sample variability algal substrate preference. Biol. Bull. 175:94-101 G. S. Loganathan, V., Shrestha, P., Dillaha, T. A., Ross, B. B. 1989. Variablesourcearea conceptforidencritical areas runoff-generating tifying in a watershed. Inst. Virginia Polytech. StateUniv.Water Resour.Res. Center Bull. I-XII:1-119 temporal Loreau,M. 1989. On testing in carabidbeetles. nichedifferentiation Oecologia 81:89-96 Lumsden, C. J. 1988. Gene-culture a coevolution: testof the steady-state for translation. hypothesis gene-culture J. Theor. Biol. 130:391-406 Madenjian,C. P., Jude,D. J., Tesar, F. J. 1986. Intervention analysis of on impact fish populations. power plant Can. J. Fish. Aquat. Sci. 43:819-29 Maelzer,D. A. 1970. The regression of log Nn+I on log Nn as a test of an density-dependence: exercise with density-indepencomputer-constructed dentpopulations. Ecology 51:810-22 Mallet, J. 1986. Dispersal and gene flow in a butterfly with home range behavior: Heliconius erato(Lepidoptera: Oecologia68:210Nymphalidae). 17 Mangel,M., Clark,C. W. 1988. Dynamic in Ecology. Modeling Behavioral NJ: Univ. Press Princeton, Princeton Manly,B. F. J. 1990. On thestatistical analysisof niche overlap data. Can. J. Zool. 68.1420-22 Manly,B. F. J. 1991. Randomization and MonteCarlo Methodsin Biology. London: Chapman& Hall Manly, B. F. J. 1992. Bootstrapping to determine samplesizes in biological studies.J. Exp. Mar. Biol. Ecol. In press Manly, B. F. J. 1993. A review of

444

CROWLEY
intensive computer multivariate meth- 179. ods in ecology. In Multivariate Environmental Statistics, G. F. Patil, 180. ed. C. R. Rao, N. P. Ross. Amsterdam: 181. North Holland/Elsevier. press In Mantel, N. 1967. The detectionof disease clustering and a generalized regression approach.Cancer Res. 27: 182. 209-20 Marriott, H. C. 1979. Barnard's F. MonteCarlo tests:how manysimulations?Appl. Stat. 28:75-77 R. T. Martin, H., Wright, A., Crowder, 183. L. B. 1989. Non-additive impactof blue crabs and spot on their prey 184. assemblages. Ecology 70:1935-42 S. Matson,P. A., Carpenter, R. 1990. Statisticalanalysis of ecological response to large-scale perturbations. Ecology 71:2037 McArdle,B. H. 1988. The structural 185. in relationship: regression biology. Can. J. Zool. 66:2329-39 P. G., Skewes, McKillup,S. C., Allen, M. A. 1988. The naturaldecline of 186. an introduced its speciesfollowing initial increasein abundance:an explain 187. nation for Ommatoiulus moreletii Australia.Oecologia 77:339-42 McLain, D. K. 1986. Null modelsand Evol. the intensity sexual selection. of Theory8:49-52 188. Menkens, E. Jr.,Anderson, H. G. S. 1988. Estimationof small-mammal population size. Ecology 69:195259 Metz,J.A. J.,Diekmann, eds. 1986. O., StrucTheDynamics Physiologically of tured Populations. Springer-Verlag 189. 68: LectureNotes in Biomathematics 475-94 J. C. Meyer, S., Ingersoll, G., McDonald, 190. L. L., Boyce, M. S. 1986. Estimating in rates: uncertainty population growth jackknife vs. bootstraptechniques. 191. Ecology 67:1156-66 Mielke, P. W. 1984. Meteorological of applications permutation techniques 192. In functions. Handbased on distance Vol. 4, ed. P. R. book of Statistics, P. Krishnaiah, K. Sen, pp. 813-30. North-Holland 193. Amsterdam: K. G. Mielke,P. W., Berry, J.,Brier, of multiresponse W. 1981. Application for permutation procedures examining in meanseaseasonalchanges monthly level pressure patterns. Monthly 194. Rev. 109:120-26 Weather K. Mielke,P. W., Berry, J.,Brockwell, P. J., Williams, S. 1981. A class of 195. J. tests based on multinonparametric response permutation procedures. Biometrika 68:720-24 196. variances. Miller, 1968. Jackknifing R. Ann. Math. Stat. 39:567-82 Miller,R. G. 1974. The jackknife-a 61:1-17 review.Biometrika Minta,S., Mangel,M. 1989. A simple basedon simulation estimate population and capturefor capture-recapture resight data. Ecology 70:1738-51 T., Mitchell-Olds, Shaw, R. G. 1987. selection: of Regression analysis natural inference biologicalinand statistical Evolution41:1149-61 terpretation. L. Mueller,L. D., Altenberg, 1985. on Statisticalinference measures of niche overlap.Ecology 66:1204-10 R. Nemec, A. F. L., Brinkhurst, 0. to 1988. Using the bootstrap assess in statistical significance the cluster analysis of species abundance data. Can. J. Fish. Aquat. Sci. 45:965-70 L. Newell,C. J.,Hopkins, P., Bedient, database P. B. 1990. A hydrogeologic for ground-water modeling. Ground Water 28:703-14 IntenNoreen,E. W. 1989. Computer Hypotheses: sive Methods for Testing New York: Wiley An Introduction. Ocana, J., Ruiz de Villa, C., Ribo, G. in 1991. Bias andefficiency estimating Levene's Z indexand sexualselection: alternatives. Heredity someresampling 67:95-102 P., Odendaal,F. J.,Turchin, Stermitz, F. R. 1989. Influenceof host-plant on densityand male harassment the anof distribution femaleEuphydryas icia (Nymphalidae). Oecologia 78: 283-88 Ohta, T. 1988. Evolution by gene advantaand duplication compensatory Genetics120:841-48 geous mutations. simulation Ohta, T. 1988b. Further studieson evolution gene duplicaby 42:375-86 tion. Evolution a Olson, C. L. 1976. On choosing test of statistic multivariate in analysis variance. Psychol.Bull. 83:579-86 Page, R. D. M. 1988. Quantitative cladistic biogeography:constructing area cladograms. and comparing Syst. Zool. 37:254-70 Page, R. D. M. 1991. Clocks, clades, rates of and cospeciation: comparing evolutionand timingof cospeciation events in host-parasite assemblages. Syst. Zool. 40:188-98 M. W. 1990. The estimation Palmer, of species richnessby extrapolation. Ecology 71:1195-98 M. species Palmer, W. 1991.Estimating jackknife richness:the second-order reconsidered. Ecology 72:1512-13 tests of Pamilo, P. 1990. Statistical

166.

167. 168.

169.

170. 171.

172. 173.

174.

175.

176.

177.

178.

STATISTICAL RESAMPLING METHODS

phenograms basedon genetic distances. Evolution44:689-97 Papadopoulos,A. S., Tiwari,R. C., 211. Muha,M. J.1991.Bootstrap procedures fortimeseriesanalysisof BOD data. Ecol. Model. 55:57-66 G. Patterson, B. 1986. A statistical 212. for methodfor testing dietary differences. N. Zeal. J. Zool. 13:113-15 Peltonen, Hanski,I. 1991. Patterns A., of islandoccupancy explained col- 213. by onization extinction inshrews. and rates Ecology 72:1698-708 D. 214. Phillips, A., Neal, L., Johnson, R. T. M., Nagel,J.W., Laughlin, F., Lane, R. T. 1987. Simulating local extinction of trout In populations. Tools for the 215. Simulation ed. Profession, R. Hawkins, K. Klukis,pp. 24-27. San Diego: Soc. Simul. Computer MontePimm,S. L. 1983. Appendix: Carlo analysesin ecology. In Lizard 216. Ecology:Studies a ModelOrganism, on ed. R. B. Huey,E. R. Pianka,T. W. Schoener, pp. 290-96. Cambridge, Mass: Harvard Univ. Press J. Pleasants, M. 1990. Null-model tests 217. for competitive displacement: falthe lacy of not focusingon the whole community. Ecology 71:1078-84 M. Poethke,H. J., Kirchberg, 1987. 218. On thestabilizing of effect density-dependentmortality factors.Oecologia 74:156-58 Pollard,E., Lakhani,K. H., Rothery, 219. P. 1987. The detection density-deof from seriesof annualcena pendence suses. Ecology 68:2046-55 K. Pontasch, W., Smith, E.P., Cairns, 220. J. Jr.1989. Diversity commuindices, nity comparison indices,and canonical discriminant the 221. analysis interpreting resultsof multispecies toxicity tests. WaterRes. 23:1229-38 Powell, G. L., Russell, A. P. 1985. Growth sexualsize dimorphism and in Alberta populations theeastern of short- 222. hornedlizard, Phrynosoma douglassi brevirostre. Can. J. Zool. 63:139-54 Quinn,J. F. 1987. On the statistical of detection cycles in extinctions in 223. the marine fossilrecord. Paleobiology 13:465-78 T. Raup, D. M., Gould,S. J., Schopf, J. M., Simberloff, S. 1973. StoD. chasticmodels of phylogeny the 224. and of evolution diversity. Geol. 81:525J. 42 Raup, D. M., Sepkoski,J. J. 1988. 225. Testingfor periodicity extinction. of Science 241:94-96 Reddingius, den Boer, P. J. 1989. J., On thestabilization animal of numbers. 226.

445

197.

198. 199.

200.

201.

202.

203.

204.

205.

206.

207.

208.

209. 210.

Problems testing. Powerestimates of 1. andestimation errors. Oecologia78:1-8 Ribi,J.,Tomczak, 1990.An impact M. assessment for the French nuclear weapontestsitesin French Polynesia. Mar. Pollut. Bull. 21:536-42 Riessen,H. P., Sprules,W. G. 1990. costsof antipredator deDemographic fensesin Daphnia pulex. Ecology 71: 1536-46 M. Ritchie, E. 1990. Optimal foraging in and fitness Columbian ground squirrels. Oecologia 82:56-67 Roberts, A., Stone, L. 1990. Islandsharing by archipelago species. Oecologia 83:560-67 M. G. Robertson, P., Huston, A., Evans, F. C., Tiedje, J. M. 1988. Spatial variability a successional in plantcomof munity: patterns nitrogen availability. Ecology 69:1517-24 Rogers,D. S. 1990. Genic evolution, and historical biogeography, systematic relationships amongspiny pocket mice, J. subfamily Heteromyinae. Mammal. 71:668-85 A. Rogers,A. R., Eriksson, W. 1988. Statisticalanalysis of the migration of component genetic drift.Am. J. 77:451-58 Phys. Anthropol. Romano, J. P. 1989. Bootstrapand randomization testsof some nonparametric Ann. Stat. 17:141hypotheses. 59 Romano,J. P. 1990. On thebehavior of randomization without group a tests invarianceassumption. Am. Stat. J. Assoc. 85:686-92 on Root, T. 1989. Energyconstraints avian distributions:replyto Castro. a Ecology 70:1183-85 Rose, K. A., McLean,R. I., Summers, J. K. 1989. Development and Monte Carlo analysisof an oyster bioaccumulation modelappliedto biomonitoring data. Ecol. Model. 45:111-32 J. Rotenberry, T. 1985. The role of habitatin avian community composition:physiognomy floristics. or Oecologia 67:213-17 Sale, P. F. 1974. Mechanisms coof in existence a guildof territorial fishes at Heron Island. In Proc. 2nd Int. CoralReefSymp.1:193-206.Brisbane: GreatBarrier Reef Comm. Sanderson,M. J. 1989. Confidence limits on phylogenies: the bootstrap revisited. Cladistics5:113-29 Schill, W. B., Dorazio, R. M. 1990. of Immunological discrimination Atlanticstriped bass stocks.Trans.Am. Fish. Soc. 119:77-85 Schluter, D., Repasky,R. R. 1991.

446

CROWLEY
densities of limitation finch Worldwide by food and otherfactors.Ecology 243. 72:1763-74 D. J., J., Schmitt, Eccleston, Ehrhardt, flowering W. 1987. Density-dependent and outcrossing, reproducphenology, capensis.Oecologia tion in Impatiens 72:341-47 244. Setzer,R. W. 1985. Spatio-temporal popin of patterns mortality Pemphigus on 245. ulicaulis and P. populitransversus Oecologia 67:310-21 cottonwoods. D. Simberloff, 1978. Usingislandbioto distributions determine geographic Am. Nat. 246. is if colonization stochastic. 112:713-26 Simberloff, Boecklen,W. 1981. D., size Santa Rosalia reconsidered: ratios 35:1206-28 Evolution andcompetition. and 247. Simonoff, S. 1986. Jackknifing J. bootstrapping goodnessof fitstatistics J. in sparse multinomials. Am. Stat. Assoc. 81:1005-11 H. M., Slatkin, Arter, E. 1991. Spatial 248. in methods population autocorrelation genetics.Am. Nat. 138:499-517 C. Slowinski, B., Guyer, 1989.Test- 249. J. of of ing the stochasticity patterns null an diversity: improved organismal model. Am. Nat. 134:907-21 N. K. Smettem, R. J., Collis-George, of ponded 250. 1985. Prediction steady-state in infiltration distributions a soil with verticalmacropores. Hydrol. 79: J. 115-22 F. Sokal, R. R., Rohlf, J. 1981.Biometry:The Principlesand Practice of in Statistics BiologicalResearch.New 251. York:Freeman. 2nd ed. Solow, A. R. 1989. Bootstrapping point patterns. spatial sampled sparsely 252. Ecology 70:379-82 Solow, A. R. 1989. A randomization of testfor independence animalloca253. tions. Ecology 70:1546-49 Solow, A. R. 1990. A randomization in probability testformisclassification discriminant analysis. Ecology 71: 254. 2379-82 J. St. Amant, L. S. 1970.The detection Ecolin of regulation animalnumbers. 255. ogy 51:823-28 G. Starks,T. H., Flatman, T. 1991. decimonitoring RCRA ground-water conviewedas quality sionprocedures Monit.Assess. trolschemes.Environ. 256. 16:19-38 E. D. Stauffer, F., Garton, O., Steinhorst, of R. K. 1985. A comparison principal data. from components realandrandom 257. Ecology 66:1693-98 A., Stewart-Oaten, Bence, J. R., Oseffects C. enberg, W. 1992. Assessing no perturbations: simple ofunreplicated Ecology. In press solutions. Still,A. W., White,A. P. 1981. The test randomization as an approximate to alternative the F testin analysisof variance.Br. J. Math. Stat. Psychol. 34:243-52 L. Stone, Roberts, 1990.ThecheckL., erboard scoreand speciesdistributions. Oecologia 85:74-79 L. Stone, Roberts, 1991.Conditions L., for a species to gain advantagefrom Ecology the presenceof competitors. 72:1964-72 Strobeck,C. 1987. Average number in differences a sample of nucleotide a a from singlesubpopulation: testfor Genetics117: subdivision. population 149-54 L. D. Strong, R., Szyska, A., Simberloff, D. 1979. Tests of community-wide characterdisplacementagainst null Evolution33:897-913 hypotheses. R. Swihart, K. and Slade, N. A. 1985. in Testing for independence animal Ecology 66:1176-84 movements. imTemeles,E. J. 1987. The relative and inportanceof prey availability size in territory truder pressure feeding Circuscyaneus. by regulation harriers, Oecologia 74:286-97 terBraak,C. J. F. 1991. Permutation tests in significance versusbootstrap and ANOVA. In regression multiple and Bootstrapping RelatedTechniques, Berlin: Springer-Vered. K.-H. Jockel. lag. In pressc Thomas,S. C., Weiner,J. 1989. Inin asymmetry meacompetitive cluding in sures of local interference plant Oecologia 80:349-55 populations. J. Tukey, W. 1958.Bias andconfidence in not quite large samples(Abstract). Ann. Math. Stat. 29:614 P. Turchin, 1987. The role of aggreof in gation theresponse Mexicanbean to Oecologia beetles host-plant density. 71:577-82 N. vanStraalen, M. 1985. Comparative floor Collembola of demography forest Oikos 45:253-65 populations. L., Vassiliou,A., Ignatiades, Karydis, of phyM. 1989. Clustering transect collections with a quick toplankton J. randomization algorithm. Exp. Mar. Biol. Ecol. 130:135-46 in E. Venrick, L. 1990. Phytoplankton ocean: speciesstructure an oligotrophic Ecology71: variability. andinterannual 1547-63 Verrier, Colleau, J. J., Foulley,J. E., cumulated response L. 1990.Predicting selectionin finite panto directional

227.

228.

229.

230. 231.

232. 233.

234.

235.

236. 237. 238.

239. 240.

241.

242.

STATISTICAL RESAMPLING METHODS

Theor. Appl.Genet. mictic populations. 79:833-40 W. L. 1991. An evaluation Vickery, of bias in k-factor analysis.Oecologia 85:413-18 W. Vickery, L., Nudds,T. D. 1991. effects Testingfor density-dependent censuses.Oecologia 85: in sequential 419-23 Walls, M., Caswell, H., Ketola, M. 1991. Demographiccosts of Chaodefenses in Daphnia borus-induced Oecologia analysis. pulex: a sensitivity 87:43-50 B. Whicker, W., Kirchner,T. 1987. F. PATHWAY: a dynamic food-chain modelto predict radionucllide ingestion after falloutdeposition. Health Phys. 52:717-38 Wilson,J. B. 1987. Methodsfordein non-randomness species cotecting occurrences: contribution. a Oecologia 73:579-82 Wilson,J. B. 1989. A null modelof appliedto stratiguildproportionality, fication a New Zealand temperate of rain forest.Oecologia 80:263-67 J. J. Wilson, B., Hubbard, C. E., Rapson, of G. L. 1988. A comparison realized of nicherelations speciesin New Zealandand Britain. Oecologia76:106-10 E. K. Winemiller, O., Pianka, R. 1990.

447

258. 259.

266.

260.

267.

261.

268.

262.

269.

263.

270.

264.

271.

265.

of assemblages in Organization natural Ecol. fishes. desert lizardsand tropical Monogr. 60:27-55 Wolda, H. 1989. Commenton the nichedif"On testing temporal article in ferentiation carabidbeetles"by M. on Loreauandthe"Comment thearticle of M. Loreau" by P. J. den Boer. Oecologia 81:99 of Wolda,H. 1991. The usefulness the dyconceptin population equilibrium namics: A reply to Berryman. Oecologia 86:144-45 D. Yearsley, J. R., Lettenmaier, P. and 1987.Modelcomplexity dataworth of an assessment changesin theglobal carbonbudget. Ecol. Modell. 39:20126 G. Zimmerman, M., Goetz,H., Mielke, P. W. Jr. 1985. Use of an improved statistical methodfor groupcompariof sons to studyeffects prairiefire. Ecology 66:606-11 D. Zink,R. M., Dittmann, L., Rootes, DNA variW. L. 1991. Mitochondrial of and ation thephylogeny Zonotrichia. Auk 108:578-84 Zwolfer, Brandl,R. 1989. Niches H., in and size relationships Coleoptera associatedwithCardueae host plants: adaptations to resource gradients. Oecologia 78:60-68