1 views

Uploaded by Heyleen Peñaloza

- vu_Modulegroups__8-25-2013.pdf
- Rheumatology 2001 Sutcliffe 37 47
- IntJRemoteSens_Psomas_2011
- Data Mining Record
- Bs Hypothesis
- 2379 Midterm Sheet
- Collateral Value Risk-commodity Linked Exposures
- Ps 1 Solutions
- Earthdawn dice roll probabilities
- Statistics Problems 2 Chapter 9
- Variances of Direct Material
- Functional Connectivity of the Cortical Swallowing Network in Humans
- Analytics With R - Course Contents
- 27719-95508-1-PB
- Principal Component Analysis
- David Brown
- Notes Probability
- m.yusni
- Sss
- perception revised

You are on page 1of 9

86, 2011

The American Ornithologists’ Union, 2011.

Printed in USA.

A CRITICAL APPRAISAL

FR ANÇOIS -X AVIER D ECHAUME -M ONCHARMONT,1,3 K ARINE M ONCEAU,1

AND F R ANK C EZILLY1,2

1

Université de Bourgogne, Equipe Ecologie Evolutive, UMR CNRS 5561 Biogéosciences, Dijon, France; and

2

Institut Universitaire de France

Abstract.—Discriminant function analysis (DFA) based on morphological measurements is a quick, inexpensive, and eﬃcient

method for sex determination in ﬁeld studies on cryptically monomorphic bird species. However, behind the apparent standardization

and relative simplicity of DFA lie subtle diﬀerences and pitfalls that have been neglected in some studies. Most of these concerns directly

aﬀect assessment of the discriminant performance, a parameter of crucial importance in practice because it provides a measure of the

quality of an equation that may be used in later ﬁeld studies. Using results from published studies and simulations based on a large

data set collected on adult Zenaida Doves (Zenaida aurita), we assessed the eﬀects of sexual dimorphism, sample size, and validation

methods on discrimination rates. We compared the three most common methods used to estimate the proportion of correctly classiﬁed

males and females by DFA: resubstitution, jackknife, or sample splitting. Results from simulations indicate that these procedures may

lead to opposite conclusions, especially when the sample size is small. In particular, the resubstitution techniques appear to be over-

optimistic, and we therefore recommend that DFA accuracy be estimated by the jackknife cross-validation procedure. In addition, we

show that most previous studies failed to present DFA accuracy with % conﬁdence intervals, which hampers comparisons among

studies. Finally, our results suggest that large sample sizes should be preferred over repeated measurements of the same individuals,

because random measurement error is likely to have only a weak eﬀect on the accuracy of the discriminant rate. Received May ,

accepted November .

Key words: cross-validation, DFA, measurement errors, morphological measurements, sample size eﬀect, sexual dimorphism, Zenaida

aurita.

Identiﬁcation du sexe des oiseaux par analyse factorielle discriminante: une évaluation critique

Résumé.—L’analyse factorielle discriminante (AFD) à partir de mesures morphologiques constitue une méthode rapide, peu

chère et eﬃcace pour un sexage sur le terrain d’oiseaux appartenant à des espèces cryptiquement monomorphes. Cependant, l’apparente

standardisation et la relative simplicité de l’AFD cachent en réalité plusieurs diﬀérences et écueils subtils négligés dans de nombreuses

études. La plupart de ces problèmes nuisent directement à l’évaluation du pouvoir de discrimination, un paramètre crucial en pratique

puisqu’il correspond à une mesure de la qualité de l’équation destinées à être utilisée dans des études de terrain ultérieures. En nous

appuyant sur publications et sur des simulations basées sur un large jeu de données collectées sur des Tourterelles à queue carrée

Zenaida aurita adultes, nous avons évalué l’eﬀet du dimorphisme sexuel, de la taille de l’échantillon et de la méthode de validation sur

le taux de discrimination. Nous avons comparé les trois méthodes les plus utilisées pour estimer la proportion de males et de femelles

correctement sexés par l’AFD: la resubstitution, le jackknife et le sample-splitting. Les résultats issus des simulations indiquent que

ces procédures peuvent conduire à des conclusions opposées, surtout en cas de petites tailles d’échantillons. Plus précisément, les

techniques de resubstitution semblent beaucoup trop optimistes. Nous recommandons que la précision d’une AFD soit évaluée au

moyen d’un jackknife. De plus, nous montrons que la plupart des études antérieures indiquent cette précision sans l’assortir d’un

intervalle de conﬁance à %, ce qui limite la possibilité de comparaisons entre études. Enﬁn, nos résultats suggèrent que de grandes

tailles d’échantillons devraient être préférées à des mesures répétés sur les mêmes individus puisqu’il est probable que l’erreur de

mesure n’ait qu’un eﬀet limité sur la précision du taux de discrimination.

3

E-mail: fx.dechaume@u-bourgogne.fr

The Auk, Vol. , Number , pages
. ISSN -, electronic ISSN -. by The American Ornithologists’ Union. All rights reserved. Please direct all

requests for permission to photocopy or reproduce article content through the University of California Press’s Rights and Permissions website, http://www.ucpressjournals.

com/reprintInfo.asp. DOI: ./auk..

— 78 —

J ANUARY 2011 — S EXING B IRDS USING D ISCRIMINANT A NALYSIS — 79

Sex-related differences in the behavior and ecology of bird of DFA is to provide equations based on morphological measure-

species are crucial to our understanding of sexual selection and ments in order to predict the sex of birds. This equation is pri-

mating systems (Andersson ) and may also have important marily calibrated on individuals of known sex. Every discriminant

consequences for management and conservation (Zavalaga and equation comes with its estimated proportion of correctly sexed

Paredes , Fernandez-Juricic et al. ). Therefore, the ability individuals.

to identify the sex of an individual is of paramount importance in Biologists justiﬁably pay great attention to the proportion of

avian studies. Although sexual dimorphism can be conspicuous misclassiﬁed birds, because a discriminant equation is not pub-

(e.g., plumage-based sexual dichromatism), especially in the case lished as an end in itself but as a tool for reliably and rapidly sexing

of polygynous species (Darwin , Andersson ), diﬀerences birds in later ﬁeld studies. Various methods can be used to esti-

between male and female birds are often subtle or escape the hu- mate the proportion of correctly classiﬁed males and females by

man eye (Cuthill et al. ). Several techniques have been devel- DFA, and yet the choice of a method is rarely justiﬁed in the avian

oped to alleviate this diﬃculty, including anatomical examination literature. In addition, the estimated rate of correct discrimination

(Petrides , Miller and Wagner ), vocalization analyses has been reported to be sensitive to sample size (Morrison ).

(Bourgeois et al. ), sex-speciﬁc behavior analyses (Castoro Among or within subjects, variance in measurements may also

and Guhl , Flux and Innes , Fletcher and Hamer ), have an eﬀect on the ability to discriminate males from females.

and, more recently, molecular techniques (Dubiec and Zagalska- Such variance has two components: the true variance among indi-

Neubauer ). Although the latter methods are by far the most viduals and the measurement error (ME) that arises from random

popular techniques and are thought to be very reliable (but see or systematic errors. Large MEs increase the risk of Type II er-

Robertson and Gemmell ), they require training and a license ror (Lougheed et al. , Yezerinac et al. ). Francis and Mat-

to collect blood or tissues, which raises ethical issues, and imply tlin () showed that discriminant power could fall from %

ﬁnancial costs and delays for processing of samples. to % when a small amount of bias exists in morphometrics

Sexing based on morphometrics is often a reasonable choice measurement. The need for ME assessment has been emphasized

for quick and inexpensive but eﬃcient sex identiﬁcation in ﬁeld (Bailey and Byrnes , Arnqvist and Mårtensson ), yet few

studies on cryptically monomorphic bird species. Signiﬁcant, al- studies that used DFA have reported ME (Flux and Innes ,

beit small, diﬀerences in biometric measurements often exist be- Devlin et al. , Kenward et al. ), and its potential eﬀect on

tween females and males, oﬀering the possibility of discriminating discrimination rate has rarely been investigated (but see Mallory

between the sexes (Murphy , Cardoni et al. ). To that end, and Forbes ).

various statistical analyses have been used, such as linear models Here, we address the problem using a twofold approach. First,

(Jeﬀrey et al. , Iko et al. , Ura et al. , Gill and Vonhof using simulations based on a large data set, we assess the choice of

, Hallgrimsson et al. ) or multivariate methods, includ- validation method and the eﬀect of sample size or measurement

ing principal component analysis (Rubega , McCracken et al. errors on the estimated accuracy of the DFA. Second, we review

, Remisiewicz and Wennerberg , Urﬁ and Kalam , the literature to quantify the methodologies used, assess the eﬀects

Schroeder et al. ) and discriminant function analysis (DFA). of the extent of sexual dimorphism, sample size, or number of vari-

The latter is the most popular of these statistical methods, and its ables on discriminant rate, and test predictions from our simula-

use has increased steadily in recent decades (Fig. ). The principle tions. Finally, we provide recommendations for future studies.

M ETHODS

rita; females and males) were captured on Barbados from

March to May (Monceau et al. ). This sexually mono-

morphic species is widely distributed throughout the Caribbean.

Each individual was molecularly sexed (Fridolfsson and Ellegren

). The reliability of molecular sexing techniques has been re-

ported to vary according to which PCR ampliﬁcation procedure is

used (Dubiec and Zagalska-Neubauer , Robertson and Gem-

mel , Daniel et al. ). In a preliminary study (Monceau

), we ﬁrst assessed the reliability of our procedure by compar-

ing results from molecular sexing with behavioral observations

based on pairs.

For each bird, we measured three dimensions of the bill at

nostrils (length, depth, and width), head plus bill length, left and

right tarsus length, left and right wing chord, and tail length. All

measurements were made by the same person (K.M.) with a digital

caliper (precision: o. mm), except for wing chord and tail length,

FIG. 1. Histogram of the number of papers in our survey that reported which were measured with a ruler (precision: o mm). Each char-

sexing birds by means of discriminant function analysis (n 141 articles) acter was measured twice, and the caliper was removed between

over the past 50 years. measurements in order to assess MEs (Bailey and Byrnes ).

80 — DECHAUME-MONCHARMONT ET AL. — AUK, VOL. 128

TABLE 1. Comparisons between male and female Zenaida Doves on Barbados for all characters with measurement error (ME) and mean difference

index (MDI) calculated as (mean female/mean male) r 100.

Characters (mean o SD) (mean oSD) (95% CI) t-test P ME (%) (%)

Bill length (mm) 10.67 o 0.48 10.45 o 0.51 0.45 (0.28–0.62) −5.11 10 −5 7.92 97.94

Bill width (mm) 4.07 o 0.22 3.93 o 0.26 0.58 (0.41–0.77) −6.73 10 −5 9.63 96.56

Bill depth (mm) 4.15 o 0.21 3.98 o 0.21 0.78 (0.61–0.96) −8.83 10 −5 21.86 95.90

Head plus bill length (mm) 48.59 o 0.99 47.40 o 1.02 1.18 (1.01–1.36) −13.47 10 −5 3.65 97.55

Mean tarsus length (mm) 26.76 o 0.86 25.94 o 0.77 1.01 (0.92–1.21) −11.30 10 −5 6.83 96.94

Mean wing chord (cm) 15.31 o 0.54 14.74 o 0.46 1.61 (0.98–1.36) −12.93 10 −5 2.22 96.28

Tail length (cm) 10.38 o 0.56 9.94 o 0.56 0.79 (0.60–0.97) −8.94 10 −5 4.07 95.76

Body mass (g) 149.19 o 15.21 141.81 o 13.95 0.51 (0.33–0.70) −5.73 10 −5 — 95.05

Because MEs were low for all traits except bill depth (Table ), we from the complete data set. For each of these , simulated

used the mean of ﬁrst and second measurements of each variable data sets, we performed DFA and assessed the discriminant rates

in subsequent analyses. Birds were weighed with a Pesola digital using resubstitution, jackknife, and sample-splitting methods.

pocket scale MS (precision: o. g). Males diﬀered signiﬁcantly We performed a second series of simulations using the com-

from females in all morphological traits, and the mean diﬀerence plete original data set (n individuals) to assess the eﬀect of

index (MDI r mean female/mean male; Delestrade , ME. Each variable used in the DFA was the mean of two repeated

Helfenstein et al. ) was % for all characters (Table ). measures recorded in order to assess ME (Table ). We simulated

We performed DFA on the original data set using the eight higher ME for a given variable by adding random noise to both

morphological variables. Because of a violation of the assumption repeated measures. This additional noise followed a normal dis-

of homogeneity of the variance–covariance matrices (box’s M test, tribution with mean μ and standard deviation σ. We adjusted

C ., df , P .), we performed quadratic DFA (Ste- σ up to obtain the chosen ME (o.%). Note that it was not pos-

vens ) using the qda function from the MASS package (Ven- sible to artiﬁcially decrease ME, because we could not improve the

ables and Ripley ) for R, version .. (R Development Core accuracy of the data. Variables with ME already higher than the

Team ). We searched for potential multivariate outliers by mea- ones requested for the simulation were not modiﬁed. For example,

suring robust Mahalanobis distances (Rasmussen , Jarrell , for a minimal ME of %, we increased ME for every variable (in-

Tabachnick and Fidell , Nordhausen et al. ). We assessed cluding mass), except for bill depth, which already showed a larger

the leverage of outliers by removing them from the data set and esti- ME (Table ). For each value of minimal ME, we simulated

mating the new discriminant rate. The eﬀect of automated variable data sets and performed a DFA on each data set. For each DFA, the

selection was assessed using the stepclass function and minimiza- proportion of correctly sexed individuals was estimated using the

tion of Wilks’s lambda criterion (Mardia et al. ) as implemented jackknife procedure.

in the klaR package for R (Weihs et al. ). Literature survey.—We searched the literature and found

We compared three validation methods commonly used in studies (from to ) on species from families (on-

sexing birds to estimate the proportion of correctly classiﬁed in- line supplement; see Acknowledgments) that used DFA to identify

dividuals by a DFA: resubstitution, jackknife (Manly ), and an individual’s sex or describe sexual dimorphism in birds. For a

sample splitting (Picard and Berk ). In the ﬁrst method, the given group (same population and same age class), some authors

sex of each individual is predicted using the discriminant func- reported more than one discriminant rate using several discrim-

tion calculated from the complete data set. Using the jackknife (or inant equations or several validation methods or both. In order

leave-one-out) method, the sex of an individual is predicted from to avoid giving too much weight to studies that reported multiple

the discriminant equation calculated after that individual has DFAs, we retained only the equation that led to the best discrim-

been removed from the data set. This procedure is repeated until ination between sexes. If the resubstitution method or sample-

a sex is assigned to each individual (Tabachnick and Fidell ). splitting was used along with the jackknife, we report only the

With sample splitting, the data set is randomly divided into two discriminant rate estimated for the jackknife validation proce-

subsamples, and the training set (two-thirds of the individuals, as dure. We collected a total of DFAs on diﬀerent populations,

recommended by Picard and Berk ) is used to compute the age classes, and species. For each of these analyses, we recorded

discriminant function that is then used to assess the accuracy of the sample size used to calculate the discriminant function, the

the discriminant function by classifying the remaining third of the validation method, and the discriminant rate. Some authors used

individuals. a stepwise procedure to reduce the number of variables used in

We simulated both smaller data sets and data sets with larger the discriminant equation. We reported the stepwise procedure

MEs in order to assess the eﬀect of sample size, the choice of vali- used (if available) and the number of variables retained in the dis-

dation method, and MEs on the discriminant power. We deﬁned criminant equation. Finally, the mean tarsus length of males and

diﬀerent sample sizes ranging from to individuals, reg- females was also collected, if available, in order to estimate the

ularly spaced every ﬁve individuals. For each sample size, we sim- relative sexual dimorphism (Storer’s index: absolute value of the

ulated diﬀerent data sets by randomly sampling individuals diﬀerence between male and female divided by the mean tarsus

J ANUARY 2011 — S EXING B IRDS USING D ISCRIMINANT A NALYSIS — 81

morphism on the number of birds sampled using analysis of vari-

ance and on the number of variables selected in the discriminant

equation using generalized linear models (Poisson regression). We

used linear models to assess the eﬀect on the discriminant rate of

sample size, relative sexual dimorphism, validation method, num-

ber of variables in the discriminant equation, variable-selection

methods (automated or not), and year of publication. Percentages

were inspected for normality and normalized using square-root

arcsine transformation if needed (Zar ).

R ESULTS

tween the three validation methods were far from negligible, espe-

cially for small sample sizes (Fig. ). The proportions of correctly

sexed birds (discriminant rate) were larger when estimated with

the resubstitution method, and decreased with increasing sample

size (Fig. A). For small data sets (n birds), the mean esti-

mate of the discriminant rate was %. By contrast, the mean

estimates with the jackknife (Fig. B) or sample-splitting (Fig. C)

methods increased with increasing sample size and were consis-

tent, although the variance of the estimates was smaller with the

jackknife than with the sample-splitting method.

Diﬀerences between methods were less pronounced with

large data sets. Considering the DFA based on the complete origi-

nal data set (n adults), the resubstitution method estimated

a proportion of correctly sexed adults of .% (% conﬁdence

interval [CI]: .–.%), whereas the same value was .% (% FIG. 2. Simulation of the effect of sample size on the estimated propor-

CI: .–.%), and .% (% CI: .–.%) for the jackknife tion of correctly classiﬁed individuals (discriminant rate) in simulated

and the sample-splitting methods, respectively. Because their discriminant function analyses (DFAs) of the Zenaida Dove data from

bootstrapped % CI largely overlapped, the estimated proportion Barbados. From the complete data set (n 525 individuals), smaller sub-

of correctly classiﬁed birds did not signiﬁcantly diﬀer according samples were randomly selected (ranging from 25 to 520 individuals,

to the validation method. By measuring robust Mahalanobis dis- with 500 subsamples per size). For each of these 50,000 subsamples, we

tances, we identiﬁed potential multivariate outliers ( males performed a DFA and evaluated the discriminant rate by three methods:

and females). However, these outliers had a limited eﬀect in our (1) resubstitution; (2) jackknife cross-validation; and (3) sample-splitting,

analyses, because removing these extreme individuals from the in which two-thirds of the data set was used as the training sample and

original data set only slightly increased the proportion of correctly the remaining third as the test sample. Each gray dot represents one DFA.

sexed birds estimated with a given validation method: resubstitu- Each dot was slightly randomly jittered to reduce overplotting. The thick

tion, .% (% CI: .–.%); jackknife, .% (% CI: .– line and the dotted lines, respectively, represent the mean discriminant

.%); and sample splitting, .% (% CI: .–.%). rate and the 95% limits computed from the 500 DFA performed for each

The stepclass function for automated variable selection sug- sample size. The horizontal dashed line represents the discriminant rate

gested several subsets of variables. The most frequent set was estimated from the complete data set.

made up of three variables: wing chord, head length, and tarsus

length. Stepwise variable selection based on minimization of Eﬀect of measurement errors.—An increase of ME led to a

Wilks’s lambda criterion, a frequently used procedure in the bird decrease in the proportion of correctly sexed individuals (Fig. ).

literature, led to much larger variable sets, with up to eight vari- Despite a large drop of accuracy in the simulated measurements,

ables being included in the complete sample (n ). We used the the mean proportion of correctly classiﬁed individuals only de-

reduced set of three variables obtained from the stepwise proce- creased from .% to .%, which corresponds to ~ addi-

dure (wing chord, head length, and tarsus length) and performed tional adults incorrectly classiﬁed. Using the smaller subset of

the same simulations detailed above. The shape of the relationship variables (wing chord, head length, and tarsus length) selected

between mean discriminant rate and sample size was unchanged, by automated stepwise procedure, we found similar results. The

though it was slightly less steep for small sample sizes. In addi- mean discriminant rate dropped marginally from .% to .%

tion, asymptotic discriminant rates calculated for the complete with a simulated ME of %.

sample size (n ) were slightly smaller, with .% (% CI: Literature survey.—Sample size varied greatly, from birds

.–.%), .% (% CI: .–.%), and .% (% CI: .– ( males and females) to , ( males and females). The

.%) of correctly sexed birds for resubstitution, jackknife, and median sample size was birds (% CI: .–.). The median

sample-splitting, respectively. sex ratio (male/female) was . (% CI: .–.). Discriminant

82 — DECHAUME-MONCHARMONT ET AL. — AUK, VOL. 128

phism (absolute difference of tarsus length between males and females

divided by mean tarsus length) on the discriminant rate (proportion of

FIG. 3. Boxplot of the jackknifed estimated discriminant rate (proportion correctly sexed birds). The line and the gray area depict the predicted

of correctly classiﬁed individuals) as a function of the minimal measure- trend and its 95% conﬁdence interval ﬁtted by linear regression (F 64.7,

ment error (ME). ME was increased to a given value by adjusting random df 1 and 132, P 10 −5) after square-root arcsine transformation for nor-

noise to the repeated measures except when real ME was already larger. malization of the percentages.

For each value of minimal ME, 500 data sets were simulated from the

complete data set. Every discriminant equation relied on the eight vari-

ables listed in Table 1. For a given minimal ME, the black circle depicts the number of variables was signiﬁcant (F ., df and ,

the mean discriminant rate, the thick line the median, and the box the P .), with the discriminant rate decreasing with an increas-

interquartile range. The horizontal dashed line represents the reference ing number of variables for small sample sizes. No other interac-

discriminant rate estimated from the original data, with no noise added. tion had a signiﬁcant eﬀect on the discriminant rate (all P .).

The discriminant rate signiﬁcantly increased with the relative

sexual dimorphism (Fig. ). One might expect that the authors

rates ranged from % to % (median .%, % CI: .– adjusted their measurement eﬀort (sample size or number of mor-

.). Among the DFAs, .% relied on the resubstitution vali- phological variables) accordingly. However, there was no signiﬁ-

dation method or simply reported the proportion of birds correctly cant association between relative sexual dimorphism and number

sexed (presumably using resubstitution method), whereas the jack- of birds sampled (F ., df and , P .) or the number

knife and sample-splitting methods, respectively, accounted for of variables used in the discriminant equation (Poisson regres-

.% and .%. The ﬁrst occurrence of the jackknife validation sion: C ., df , P .).

method in our survey dates back to the mid-s (Brennan et al. Contrary to our simulations, validation methods had no sig-

), and one might argue that use of the jackknife is less common niﬁcant eﬀect on the discriminant rate in published studies, and

in our survey because it is a more recent and computer-intensive lower sample sizes had an eﬀect only in interaction with the num-

method than the other two. However, although most of the modern ber of variables. This might be explained by publication bias. We

statistical packages (e.g., R, SAS, and SPSS) implement DFAs with therefore examined the funnel plots of discriminant rate as a func-

jackknife procedures, only .% of DFA studies published since tion of the log-transformed sample size (Palmer ). Because

relied on jackknife procedures. Among the DFAs, .% of sampling error, the variance in estimates of the discriminant

used the automated stepwise procedure to reduce the number of rate is expected to be higher for studies with smaller sample sizes

variables in the discriminant function. There was no diﬀerence in (Møller and Jennions ). For each validation method, there

number of variables in the discriminant equation between studies was no signiﬁcant diﬀerence between the variance of discrimi-

that relied on automated stepwise variable selection and the oth- nant rates of the smallest % of sample sizes and the largest %

ers (Wilcoxon test, W ,, P .). Measurement errors were (Levene test; resubstitution: P ., jackknife: P ., sample-

mentioned in % of the papers. splitting: P .), which reveals a lack of variance for small sam-

Relative sexual dimorphism had the largest eﬀect on the dis- ple sizes. Because there was no signiﬁcant relationship between

criminant rate (F ., df and , P −) and explained sexual dimorphism and sample size, the lack of variance for small

.% of the variance. Neither the validation method (F ., df sample sizes cannot be attributed to studies on strongly dimor-

and , P .), the year of publication (F ., df and phic species (which require fewer individuals to generate a high

, P .), nor the variable selection procedure (automated discriminant rate) and is possibly, then, attributable to a publica-

or not) (F ., df and , P .) had an eﬀect on the dis- tion bias. Small data sets that led to small discriminant rates were

criminant rate. However, the interaction between sample size and probably rejected by editors or underreported by the authors.

J ANUARY 2011 — S EXING B IRDS USING D ISCRIMINANT A NALYSIS — 83

D ISCUSSION not too diﬀerent, they concluded that there was no sampling bias.

This kind of argument is poorly sustained, because important dif-

Eﬀect of the validation method.—The simulations based on our Ze- ferences between these two estimates were possible in our simula-

naida Dove data set indicate that the choice of validation method tions even in the absence of sampling bias.

may have a strong eﬀect on the estimated discriminant rate. This Importance of large sample size.—Our simulations with small

eﬀect was particularly sensitive for small to intermediate sample to medium sample sizes ( birds) highlighted the wide variance

sizes ( individuals). Estimates of the discriminant rate based in the estimated proportion of misclassiﬁed birds. High discrimi-

on subsampling techniques (such as jackknife cross-validation or nant success can be obtained by chance, raising reasonable doubts

splitting of the data into training and test sets) should be preferred about the subsequent use of equations constructed from small data

over estimates based on resubstitution, which appeared to be sets. In accordance with previous studies (Brennan et al. , We-

overly optimistic for small sample sizes. It has been well illustrated, hberg and Schumacher , Shealer and Cleary , Isaksson

in both the statistical (Johnson and Wichern , Huberty , et al. ), we recommend caution in dealing with discriminant

Manly , Piraux and Palm , Wehberg and Schumacher equations computed from small data sets. In addition, the sample

) and the applied literature (Eisenbeis , Lance et al. ), size in interaction with the number of variables in the equation had

that reclassiﬁcation of the original individuals used in construct- a signiﬁcant eﬀect on the discriminant rates from our literature

ing the discriminant equation leads to a biased estimated discrimi- survey. If there are too few individuals in relation to the number of

nant rate. In one of the ﬁrst papers to use DFA to sex birds, Ryder variables, the analysis leads to a poor discriminant rate (Marks and

() explicitly raised the problem of resubstitution and recom- Dunn , Burnham and Anderson ). This result is known as

mended that it not be used, especially in the case of a comparison the “curse of dimensionality” (Bellman ). Several authors have

among studies. But as the technique became more popular within advised carefully adjusting the number of birds measured to the

the ornithologist community, this advice was somehow over- number of morphological variables in order to have a sample size

looked. The two subsampling techniques are not equivalent, how- qr larger than the number of variables used in the DFA (Williams

ever. Sample splitting led to a mean estimate of the proportion of and Titus , McGarigal et al. ), but this criterion is some-

correctly classiﬁed individuals consistent with the mean estimate times claimed to be arbitrary (James and McCulloch ).

using the jackknife procedure, but with a much larger variance. Variable selection.—Estimating the optimal number of vari-

Therefore, jackknife cross-validation should always be preferred ables to measure is of obvious importance. Morphological mea-

over the sample-splitting procedure or resubstitution. surements are usually highly correlated, which can lead to unstable

Surprisingly, we found no diﬀerence in the discriminant parameter estimates. Eliminating multicollinearity among the

power between these three validation methods in our survey. How- variables could improve the discriminant rate, although the pro-

ever, most published discriminant rates were quite high (median cedure is highly debated (James and McCulloch ). A common

discriminant rate .%). Even in the case of minute diﬀerences procedure is to reduce each subset of highly correlated variables

between males and females (% diﬀerence), the mean discrimi- to only one variable. More importantly, morphometric measures

nant rate reported in published studies was still ~%. It is possi- are costly in time and can be stressful for the birds. Field biolo-

ble that many authors (e.g., Clark et al. ) restrained themselves gists have to limit themselves to a narrow set of morphological

from publishing DFAs with low discriminant rates, which has led variables. Two-thirds of the authors relied on automated stepwise

to a “ﬁle-drawer eﬀect” (Scargle , Møller and Jennions ) techniques in order to identify, from a larger number of poten-

and underestimation of the inﬂuence of the validation method. tially relevant measures, a reasonable set of variables. Biologists

Interstudy comparisons.—The authors often compared their should be extremely cautious with the output of automated analy-

discriminant rates to those of previous studies, most of the time to sis. Several authors strongly recommend that stepwise procedure

claim that the new discriminant function led to better classiﬁca- be avoided entirely (James and McCulloch , Snyder et al. ,

tion, even if it increased the discriminant rate by only a small per- Thompson ). One peer-reviewed journal even made it an edi-

centage. Such narrow diﬀerences may be meaningless, especially torial policy to summarily reject any article that used the tech-

when diﬀerent validation techniques were used to estimate the nique (Babyak ). Stepwise methods capitalize on sampling

discriminant rate. In addition, all the studies that we reviewed error and yield results that are not replicable (Thompson );

failed to report % CIs around discriminant rates, thus hamper- there is no reason to think that they lead to the best equation for

ing comparisons among studies (Nakagawa and Cuthill , Ga- sexing the model species, and no guarantee that the results will

ramszegi et al. ), even when conducted on the same species. be useful for later studies (James and McCulloch , Derksen

Just as publishing means without indication of variance is not ac- and Keselman ). Most importantly, the greatest danger of the

ceptable, one should never report discriminant rates without their use of stepwise analysis is the temptation to leap directly from

conﬁdence intervals. Bootstrap resampling provides a simple routine procedure to straightforward conclusions about ecologi-

and robust method for calculating this range. Another possibil- cal relevance. These procedures are unable to select from a set of

ity would be to consider Bayesian posterior probabilities instead variables those that are most inﬂuential (Burnham and Anderson

of just looking at overall error rates (McCarthy , Garamszegi ); therefore, the selected variables are not necessarily more

et al. , Hastie et al. ). Another kind of comparison is also biologically relevant than nonselected ones.

highly questionable. Some authors compared apparent error (i.e., We oﬀer the reminder to authors that stepwise procedures are

estimated from resubstitution procedure) and error estimated by exploratory and only suggest combinations of variables that do an

sample-splitting in order to detect possible sampling bias (e.g., adequate job, not necessarily the most discriminating or the sim-

Hanners and Patton ). As long as these two error rates were plest. Our survey of the literature showed that studies that relied

84 — DECHAUME-MONCHARMONT ET AL. — AUK, VOL. 128

on automated stepwise procedures did not diﬀer from the others G. Prato for help in catching birds and W. Worrell for hospital-

in the number of variables used in the discriminant equation or on ity. K.M. was supported by a doctoral grant from the Ministère de

their discriminant power. Scientiﬁc judgment and choice of bio- la Recherche et de l’Enseignement Supérieur. Financial support

logically meaningful combination of variables must play a leading was partly provided by a research grant from the Agence Nation-

role in selecting variable (James and McCulloch , Burnham ale pour la Recherche (program Monogamix ANR--BLAN-

and Anderson ). If stepwise procedures are used, Manly () -). Calculations were performed using HPC resources from

advised that the analyses be rerun several times with randomized DSI-CCUB (Université de Bourgogne).

subsamples of individuals to check the validity of the results.

Measurement errors.—We also found that the eﬀect of ME LITERATURE CITED

on discriminant rates is generally moderate but nonetheless sig-

niﬁcant. Simulating large ME in our measures only marginally Andersson, M. . Sexual Selection. Princeton University Press,

decreased the estimated proportion of correct classiﬁcation. This Princeton, New Jersey.

result contrasts sharply with that obtained by Francis and Mattlin Arnqvist, G., and T. Mårtensson. . Measurement error

(), who reported high sensitivity of DFA to small amounts of in geometric morphometrics: Empirical strategies to assess and

bias in measurement. They simulated systematic bias by adding a reduce its impact on measures of shape. Acta Zoologica Academiae

constant to each of the dimension of their data set, which was later Scientiarum Hungaricae :–.

analyzed with a discriminant function constructed from the un- Babyak, M. A. . What you see may not be what you get: A brief,

manipulated data set. We, on the other hand, investigated the eﬀect nontechnical introduction to overﬁtting in regression-type mod-

of random bias. As long as these MEs equally aﬀected males and fe- els. Psychosomatic Medicine :–.

males in the same proportion and with no systematic bias, they had Bailey, R. C., and J. Byrnes. . A new, old method for assessing

a limited eﬀect on the probability of misclassiﬁcation. This suggests measurement error in both univariate and multivariate morpho-

that increasing the number of sampled birds might be preferable metric studies. Systematic Zoology :–.

to additional measurements of the same bird. Yet, because mea- Bellman, R. . Dynamic Programming. Princeton University

surement errors were rarely reported in the papers we reviewed, we Press, Princeton, New Jersey.

could not compare our results from the simulation with actual ﬁeld Blakesley, J. A., A. B. Franklin, and R. J. Gutiérrez. . Sexual

data. More studies are needed before ﬁrm conclusions can be made dimorphism in Northern Spotted Owls from northwest California.

regarding the importance of ME for the accuracy of DFA. Journal of Field Ornithology :–.

Finally, several authors warned that multivariate outliers can Bourgeois, K., C. Curé, J. Legrand, E. Gómez-Díaz, E. Vidal,

lead to violation of multinormality assumption and can severely T. Aubin, and N. Mathevon. . Morphological versus

impair the accuracy of DFA (Manly , Tabachnick and Fidell acoustic analysis: What is the most eﬃcient method for sexing

, Osborne and Overbay ). Rather extreme observations Yelkouan Shearwaters Puﬃnus yelkouan? Journal of Ornithology

can inﬂate classiﬁcation error rates and bias parameter estimation. :–.

However, search methods for outliers were almost never reported Brennan, L. A., J. B. Buchanan, C. T. Schick, and S. G. Herman.

in the ornithological literature that we reviewed (but see Blakes- . Estimating sex-ratios with discriminant function analysis:

ley et al. , Santiago-Alarcon and Parker ). We advise that The inﬂuence of probability cutpoints and sample size. Journal of

future studies that use DFA to sex birds systematically test for the Field Ornithology :–.

presence of outliers and, if they are present, decide, on the basis of Brennan, L. A., J. B. Buchanan, C. T. Schick, S. G. Herman,

established criteria (discussed at length in Tabkachnick and Fidell and T. M. Johnson. . Sex determination of Dunlins in winter

), whether or not to remove them. plumage. Journal of Field Ornithology :–.

Our purpose was not to criticize the use of DFA, which re- Burnham, K. P., and D. R. Anderson. . Model Selection

mains in our opinion a very eﬃcient and straightforward technique. and Multimodel Inference: A Practical Information-Theoretic

Molecular methods are not foolproof, but they are unquestionably Approach, nd ed. Springer-Verlag, New York.

more reliable and popular than DFA. If a person does not have ac- Cardoni, D. A., J. E. Maldonado, J. P. Isacch, and R. Greenberg.

cess to a molecular laboratory, proper performance of DFA should . Subtle sexual dimorphism in the Bay-capped Wren-Spinetail

follow the guidelines that we have recommended. Most of the com- (Spartonoica maluroides; Furnariidae) uncovered through molec-

mon pitfalls addressed here can easily be avoided. Authors should be ular sex determination. Onitología Neotropical :–.

extremely cautious with the predictions of discriminant equations Castoro, P. L., and A. M. Guhl. . Pairing behavior of pigeons

constructed from small data sets, even when the reported discrimi- related to aggressiveness and territory. Wilson Bulletin :–.

nant rate is high. The accuracy of DFA should be estimated with the Clark, R. G., P. C. James, and J. B. Morari. . Sexing adult

jackknife cross-validation procedure, and this accuracy should al- and yearling American Crows by external measurements and dis-

ways be presented with % CIs to allow interstudy comparisons. criminant analysis. Journal of Field Ornithology :–.

Cuthill, I. C., A. T. D. Bennett, J. C. Partridge, and E. J. Maier.

ACKNOWLEDGMENTS . Plumage reﬂectance and the objective assessment of avian

sexual dichromatism. American Naturalist :–.

Supplementary material (Table S) is available at dx.doi.org/./ Daniel, C., C. D. Millar, S. M. H. Ismar, B. M. Stephenson,

auk... We thank S. Devonish of the Natural Heritage and M. E. Hauber. . Evaluating molecular and behavioural

Department for providing us with a permit to capture and ring sexing methods for the Australasian Gannet (Morus serrator).

Zenaida Doves in Barbados. We thank J. Moreau, S. Motreuil, and Australian Journal of Zoology :–.

J ANUARY 2011 — S EXING B IRDS USING D ISCRIMINANT A NALYSIS — 85

Darwin, C. R. . The Descent of Man, and Selection in Relation Isaksson, A., M. Wallman, H. Göransson, and M. G.

to Sex. John Murray, London. Gustafsson. . Cross-validation and bootstrapping are

Delestrade, A. . Sexual size dimorphism and positive assor- unreliable in small sample classiﬁcation. Pattern Recognition

tative mating in Alpine Choughs (Pyrrhocorax graculus). Auk Letters :–.

:–. James, F. C., and C. E. McCulloch. . Multivariate analysis

Derksen, S., and H. J. Keselman. . Backward, forward and in ecology and systematics: Panacea or Pandora’s box? Annual

stepwise automated subset selection algorithms: Frequency of Review of Ecology, Evolution, and Systematics :–.

obtaining authentic and noise variables. British Journal of Math- Jarrell, M. G. . A comparison of two procedures, the Mahalano-

ematical and Statistical Psychology :–. bis distance and the Andrews-Pregibon statistic, for identifying

Devlin, C. M., A. W. Diamond, and G. W. Saunders. . Sexing multivariate outliers. Research in the Schools :–.

Arctic Terns in the ﬁeld and laboratory. Waterbirds :–. Jeffrey, J. J., S. G. Fancy, G. D. Lindsey, P. C. Banko, T. K. Pratt,

Dubiec, A., and M. Zagalska-Neubauer. . Molecular tech- and J. D. Jacobi. . Sex and age identiﬁcation of Palila. Journal

niques for sex identiﬁcation in birds. Biological Letters :–. of Field Ornithology :–.

Eisenbeis, R. A. . Pitfalls in the application of discriminant Johnson, R. A., and D. W. Wichern. . Applied Multivariate

analysis in business, ﬁnance, and economics. Journal of Finance Statistical Analysis, rd. ed. Prentice-Hall, Englewood Cliﬀs, New

:–. Jersey.

Fernandez-Juricic, E., A. J. del Nevo, and R. Poston. . Kenward, B., C. Rutz, A. A. S. Weir, J. Chapell and A. Kacelnik.

Identiﬁcation of individual and population-level variation in . Morphology and sexual dimorphism of the New Caledo-

vocalizations of the endangered Southwestern Willow Flycatcher nian Crow Corvus moneduloides, with notes on its behaviour and

(Empidonax traillii extimus). Auk :–. ecology. Ibis :–.

Fletcher, K. L., and K. C. Hamer. . Sexing terns using bio- Lance, R. F., M. L. Kennedy, and P. L. Leberg. . Classiﬁcation

metrics: The advantage of within-pair comparisons. Bird Study bias in discriminant function analyses used to evaluate putatively

:–. diﬀerent taxa. Journal of Mammalogy :–.

Flux, I., and J. Innes. . A ﬁeld technique for determining the Lougheed, S. C., T. W. Arnold, and R. C. Bailey. . Measure-

sex of North Island Kokako (Callaeas cinerea wilsoni). Notornis ment error of external and skeletal variables in birds and its eﬀect

:–. on principal components. Auk :–.

Francis, R. I. C. C., and R. H. Mattlin. . A possible pitfall in Mallory, M. L., and M. R. Forbes. . Sex discrimination and

the morphometric application of discriminant analysis: Measure- measurement bias in Northern Fulmars Fulmarus glacialis from

ment bias. Marine Biology :–. the Canadian Arctic. Ardea :–.

Fridolfsson, A. K., and H. Ellegren. . A simple and universal Manly, B. F. J. . Multivariate Statistical Methods: A Primer, nd

method for molecular sexing of non-ratite birds. Journal of Avian ed. Chapman & Hall, London.

Biology :–. Mardia, K. V., J. T. Kent, and J. M. Bibby. . Multivariate Anal-

Garamszegi, L. Z., S. Calhim, N. Dochtermann, G. Hegyi, ysis. Academic Press, New York.

P. L. Hurd, C. Jørgensen, N. Kutsukake, M. J. Lajeunesse, Marks, S., and O. J. Dunn. . Discriminant functions when

K. A. Pollard, H. Schielzeth, and others. . Changing covariance matrices are unequal. Journal of the American Statis-

philosophies and tools for statistical inferences in behavioral tical Association :–.

ecology. Behavioral Ecology :–. McCarthy, M. A. . Bayesian Methods for Ecology. Cambridge

Gill, S. A., and M. J. Vonhof. . Sexing monochromatic birds in University Press, Cambridge, United Kingdom.

the ﬁeld: Cryptic sexual size dimorphism in Buﬀ-breasted Wrens McCracken, K. G., D. C. Paton, and A. D. Afton. . Sexual

(Thryotorus leucotis). Ornitología Neotropical :–. size dimorphism of the Musk Duck. Wilson Bulletin :–.

Hallgrimsson, G. T., S. Palsson, and R. W. Summers. . Bill McGarigal, K., S. Cushman, and S. Stafford. . Multivari-

length: A reliable method for sexing Purple Sandpipers. Journal of ate Statistics for Wildlife and Ecology Research. Springer-Verlag,

Field Ornithology :–. New York.

Hanners, L. A., and S. R. Patton. . Sexing Laughing Gulls Miller, W. J., and F. H. Wagner. . Sexing mature columbi-

using external measurements and discriminant analysis. Journal formes by cloacal characters. Auk :–.

of Field Ornithology :–. Møller, A. P., and M. D. Jennions. . Testing and adjusting for

Hastie, T., R. Tibshirani, and J. Friedman. . The Elements publication bias. Trends in Ecology & Evolution :–.

of Statistical Learning: Data Mining, Inference, and Prediction. Monceau, K. . Biologie des populations de tourterelles à queue

Springer-Verlag, Berlin. carrée sur l’île de la Barbade: Apport de l’outil moléculaire. Ph.D.

Helfenstein, F., E. Danchin, and R. H. Wagner. . Assor- dissertation, Université de Bourgogne, Dijon, France.

tative mating and sexual size dimorphism in Black-legged Kitti- Monceau, K., R. Wattier, F.-X. Dechaume-Moncharmont,

wakes. Waterbirds :–. S. Motreuil, and F. Cézilly. . Territoriality versus ﬂock-

Huberty, C. J. . Applied Discriminant Analysis. Wiley, New ing in the Zenaida Dove (Zenaida aurita): Resource polymor-

York. phism revisited using morphological and genetic analyses. Auk

Iko, N. M., S. J. Dinsmore, and F. L. Knopf. . Evaluating the :–.

use of morphometric measurements from museum specimens for Morrison, M. L. . Inﬂuence of sample size on discriminant

sex determination in Mountain Plovers (Charadrius montanus). function analysis of habitat use by birds. Journal of Field Orni-

Western North American Naturalist :–. thology :–.

86 — DECHAUME-MONCHARMONT ET AL. — AUK, VOL. 128

Murphy, M. T. . A cautionary tale: Cryptic sexual size dimor- Shealer, D. A., and C. M. Cleary. . Sex determination of adult

phism in a socially monogamous passerine. Auk :–. Black Terns by DNA and morphometrics: Tests of sample size, tem-

Nakagawa, S., and I. C. Cuthill. . Eﬀect size, conﬁdence poral stability and geographic speciﬁcity in the classiﬁcation accu-

interval and statistical signiﬁcance: A practical guide for biologists. racy of discriminant functions models. Waterbirds :–.

Biological Reviews :–. Smith, R. J. . Statistics of sexual size dimorphism. Journal of

Nordhausen, K., H. Oja, and D. E. Tyler. . Tools for exploring Human Evolution :–.

multivariate data: The package ICS. Journal of Statistical Software Snyder, P., B. Thompson, and P. Snyder. . Three reasons why

:–. stepwise regression methods should not be used by researchers.

Osborne, J. W., and A. Overbay. . The power of outliers (and Pages – in Advances in Educational Research: Substantive

why researchers should always check for them). Practical Assess- Findings, Methodological Developments (B. Thompson, Ed.). JAI,

ment, Research and Evaluation (). Greenwich, Connecticut.

Palmer, A. R. . Quasi-replication and the contract of error: Stevens, J. . Applied Multivariate Statistics for the Social Sci-

Lessons from sex ratios, heritabilities and ﬂuctuating asymmetry. ences, nd ed. Lawrence Erlbaum, Hillsdale, New Jersey.

Annual Review of Ecology and Systematics :–. Tabachnick, B. G., and L. S. Fidell. . Using Multivariate

Petrides, G. A. . Notes on determination of sex and age in the Statistics, th ed. HarperCollins, New York.

Woodcock and Mourning Dove. Auk :–. Thompson, B. . Stepwise regression and stepwise discriminant

Picard, R. R., and K. N. Berk. . Data splitting. American Stat- analysis need not apply here: A guidelines editorial. Educational

istician :–. and Psychological Measurement :–.

Piraux, F., and R. Palm. . Etude empirique des estimateurs Thompson, B. . Signiﬁcance, eﬀect sizes, stepwise methods,

des taux d’erreur en analyse discriminante. Revue de statistique and other issues: Strong arguments move the ﬁeld. Journal of

appliquée :–. Experimental Education :–.

R Development Core Team. . R: A Language and Environ- Ura, T., N. Azuma, S. Hayama, and S. Higashi. . Sexual

ment for Statistical Computing. R Foundation for Statistical Com- dimorphism of Latham’s Snipe (Gallinago hardwickii). Emu :

puting, Vienna. [Online.] Available at cran.r-project.org/doc/ –.

manuals/refman.pdf. Urfi, A. J., and A. Kalam. . Sexual size dimorphism and mating

Rasmussen, J. L. . Evaluating outlier identiﬁcation tests: pattern in the Painted Stork (Mycteria leucocephala). Waterbirds

Mahalanobis D squared and Comrey D. Multivariate Behavioral :–.

Research :–. Venables, W. N., and B. D. Ripley. . Modern Applied Statistics

Remisiewicz, M., and L. Wennerberg. . Diﬀerential migra- With S, th ed. Springer-Verlag, Berlin.

tion strategies of the Wood Sandpiper (Tringa glareola)—Genetic Wehberg, S., and M. Schumacher. . A comparison of

analyses reveal sex diﬀerences in morphology and spring migra- nonparametric error rate estimation methods in classiﬁcation

tion phenology. Ornis Fennica :–. problems. Biometrical Journal :–.

Robertson, B. C., and N. J. Gemmell. . PCR-based sexing in Weihs, C., U. Ligges, K. Luebke, and N. Raabe. . klaR ana-

conservation biology: Wrong answers from an accurate method- lyzing German business cycles. Pages – in Data Analysis

ology? Conservation Genetics :–. and Decision Support (D. Baier, R. Decker, and L. Schmidt-Thieme,

Rubega, M. A. . Sexual size dimorphism in Red-necked Phala- Eds.). Springer-Verlag, Berlin.

ropes and functional signiﬁance of nonsexual bill structure varia- Williams, B. K., and K. Titus. . Assessment of sampling sta-

tion for feeding performance. Journal of Morphology :–. bility in ecological applications of discriminant analysis. Ecology

Ryder, J. P. . Sexing Ring-billed Gulls externally. Bird-Banding :–.

:–. Yezerinac, S. M., S. C. Lougheed, and P. Handford. .

Santiago-Alarcon, D., and P. G. Parker. . Sexual size dimor- Measurement error and morphometric studies: Statistical power

phism and morphological evidence supporting the recognition of and observer experience. Systematic Biology :–.

two subspecies in the Galápagos Dove. Condor :–. Zar, J. H. . Biostatistical Analysis, th ed. Prentice Hall, Upper

Scargle, J. D. . Publication bias: The “ﬁle-drawer” problem in Saddle River, New Jersey.

scientiﬁc inference. Journal of Scientiﬁc Exploration :–. Zavalaga, C. B., and R. Paredes. . Sex determination of adult

Schroeder, J., P. M. Lourenço, M. van der Velde, J. C. E. W. Humboldt Penguins using morphometric characters. Journal of

Hooijmeijer, C. Both, and T. Piersma. . Sexual dimor- Field Ornithology :–.

phism in plumage and size in Black-tailed Godwits Limosa limosa

limosa. Ardea :–. Associate Editor: M. E. Hauber

- vu_Modulegroups__8-25-2013.pdfUploaded byDorin Rusu
- Rheumatology 2001 Sutcliffe 37 47Uploaded byAlina Cretu
- IntJRemoteSens_Psomas_2011Uploaded byEdwin Delta Tambunan
- Data Mining RecordUploaded byMohan Chandaluru
- Bs HypothesisUploaded byPham Hai Yen
- 2379 Midterm SheetUploaded byLiam Maw
- Collateral Value Risk-commodity Linked ExposuresUploaded byKrishnan Chari
- Ps 1 SolutionsUploaded byJonathan Shao
- Earthdawn dice roll probabilitiesUploaded bydabar aher
- Statistics Problems 2 Chapter 9Uploaded byALyzha Mae Lacuesta
- Variances of Direct MaterialUploaded byChintan Tejani
- Functional Connectivity of the Cortical Swallowing Network in HumansUploaded byMauricio Escobar Castañeda
- Analytics With R - Course ContentsUploaded byHarikrishna Bajjuri
- 27719-95508-1-PBUploaded bySaifullah Khalid
- Principal Component AnalysisUploaded byVicky Vk
- David BrownUploaded bySrinu Bonu
- Notes ProbabilityUploaded byWadnerson Boileau
- m.yusniUploaded byYuznie Amelia
- SssUploaded byKona Kiran
- perception revisedUploaded byShivaniSharma
- gilmour_separable1997Uploaded byNojordari Sordar
- Bios Tat AUploaded byLucy Lucerne
- Intelligent Prediction of Sieving Efficiency in Vibrating ScreensUploaded bySHAIJU
- RuggednessUploaded byiabureid7460
- Guidance for Writing Lab ReportsUploaded byalawi1889
- ExploreUploaded byTria
- VeganUploaded bySiddharta Gauthama
- chapter 3.docUploaded bynilesh bhojani
- Soroosh Solhjoo et al- Classification of Chaotic Signals Using HMM Classifiers: EEG-Based Mental Task ClassificationUploaded byAsvcxv
- MATH 231-Statistics-Dr. Hanif MianUploaded bySubayyal Ilyas

- A New Genus for the Waterthrushes (Parulidae)Uploaded byHeyleen Peñaloza
- 6-Seminario-I-II-TEG-NORMATIVA CITAS BIBLIOGRAFICAS.pdfUploaded byHeyleen Peñaloza
- 6-Seminario-I-II-TEG-NORMATIVA CITAS BIBLIOGRAFICAS.pdfUploaded byHeyleen Peñaloza
- 6-Seminario-I-II-TEG-NORMATIVA CITAS BIBLIOGRAFICAS.pdfUploaded byHeyleen Peñaloza
- Planilla Del Seminario I (NUEVA)Uploaded byHeyleen Peñaloza
- A Review of Waterthrush Identification With Particular Reference to the 1968 British RecordUploaded byHeyleen Peñaloza
- Guía Para La Evaluación de ML y MIUploaded byHeyleen Peñaloza
- 5 Seminario i II Teg ReglamentoUploaded byHeyleen Peñaloza
- 5 Seminario i II Teg ReglamentoUploaded byHeyleen Peñaloza
- Asignatura Que Ofrece La UDPPUploaded byHeyleen Peñaloza
- Asignatura Que Ofrece La UDPPUploaded byHeyleen Peñaloza
- Migracion diferenciada.pdfUploaded byHeyleen Peñaloza
- Planilla Del Proyecto ML y MIUploaded byHeyleen Peñaloza
- Abundance, Feeding Behavior, And Body Condition of Nearctic Warblers Wintering in Venezuelan MangrovesUploaded byHeyleen Peñaloza
- Anexo Al Reglamento de Seminario iUploaded byHeyleen Peñaloza
- CLIENTE Propuesta SuplidoraUploaded byHeyleen Peñaloza
- planilla_modificacionUploaded byHeyleen Peñaloza
- Covino 2015 WJOUploaded byHeyleen Peñaloza
- CITASUploaded byHeyleen Peñaloza
- Prueba t de StudentUploaded byHeyleen Peñaloza
- Gill & Vonhof Sexing Monochromatic Birds on 2006Uploaded byHeyleen Peñaloza
- Kardex_18358806_BUploaded byHeyleen Peñaloza
- Past 3 ManualUploaded byHeyleen Peñaloza
- Dechaume-Moncharmont et al 2011 Auk Supp1.pdfUploaded byHeyleen Peñaloza
- Dechaume-Moncharmont et al 2011 Auk Supp1.pdfUploaded byHeyleen Peñaloza
- Planilla Elab Del Proyecto de TEG SEMINARIO IUploaded byHeyleen Peñaloza
- Pagos de Bancos Net UnoUploaded byHeyleen Peñaloza
- Migracion diferenciada.pdfUploaded byHeyleen Peñaloza
- Directv - Resultado EpagoUploaded byHeyleen Peñaloza

- Mohr's CircleUploaded byaap1
- Microsoft Dynamics CRM 2011 User's GuideUploaded byRafael Souza
- Gauss Seidel MethodUploaded byGerhan Pradana
- AS5050A_Datasheet_EN_v6 (1)Uploaded byAbhijeet Chaudhari
- ScribdUploaded byJiT2U
- Pedn Bhs Inggris_noun VocabUploaded byIka Meri
- Improving Your DelegationUploaded byBurt Gummer
- User Manual ADU-555CUploaded byromaditya
- Ch03_InducstrialHygieneW17Uploaded byRadhi
- Ho EstimateUploaded bySakib Baghoust
- ME 329 Chapter 5 Oct 28 2009Uploaded byMitch
- 6446penelitian DMUploaded byharis
- MCom Risk Management of Financial Markets Handbook EntryUploaded byMzukisi
- MELJUN IT102 Syllabus Computer Fundamental and ProgrammingUploaded byMELJUN CORTES, MBA,MPA
- Energy Transport Notes 2Uploaded byFaisal Mumtaz
- Casas e Ascendente ScorpioUploaded bykbludos
- hee young nohUploaded byapi-339212734
- Rizki Lbm 3 HormonUploaded byAndri Tri Atmojo
- ajb822003Uploaded byJustin Mak
- CbsUploaded byShawn Davidson
- Ncle-CDW-BME-Mental-Health-leaflet-complete1.pdfUploaded byaghorbanzadeh
- Evolution of Archery in AmericaUploaded byArchery Duns
- peebles2003.pdfUploaded byRuskin Patel
- Aristotle_ Politics [Internet Encyclopedia of Philosophy]Uploaded byWes Alwan
- Anatomy and Physiology of the Circulatory and Ventilatory Systems, ThirietUploaded bySubhajit Chakraborty
- Pages From Vincenzo RuggieroUploaded bycrismg1
- Application of Artificial Neural Networks in Groundwater Table Forecasting a Case Study in a Singapore Swamp ForestUploaded byAaron Arango Ayala
- The Genomic Challenge to the Social Construction of Race - Lee Shiao, Bode, Beyer, SelvingUploaded bySofía Muñoz
- Introduction to Sociology and the Sociological ImaginationUploaded byJohn Nicer Abletis
- VETASSESS Update on Application Processes for Skills Assessments for General Professional OccupationsUploaded byaga_andre