J Forensic Sci, 2017

doi: 10.1111/1556-4029.13564
PAPER Available online at: onlinelibrary.wiley.com

ANTHROPOLOGY

Kelly R. Kamnikar,1 M.A.; Amber M. Plemons,1 M.A.; and Joseph T. Hefner,1 Ph.D.

Intraobserver Error in Macromorphoscopic

Trait Data*,,

ABSTRACT: As part of a much larger investigation into the use of macromorphoscopic trait data by forensic anthropologists to estimate
ancestry from unidentified skeletal remains, we conducted a fourteen-year (20022016) intraobserver error study. Motivated by the development
of a large macromorphoscopic databasewhich will potentially utilize data collected in 2002quantification of observer error, the impact of
technological improvements in macromorphoscopic trait data collection and observer experience is necessary. To maximize comparisons
between the two samples, ten macromorphoscopic traits were assessed. Results revealed three patterns of error relating to observer experience,
the introduction of new technologies, and error inherent in the method. Overall, this study found the effect of error on macromorphoscopic trait
analysis could be predicted and did not significantly impact their utility.

KEYWORDS: forensic science, forensic anthropology, ancestry, macromorphoscopic traits, quantitative data analysis, observer experience

Creation of the biological profile for unidentified human skeletal
remains is a starting point in many forensic anthropological inves-
tigations. Ancestry estimation is one key aspect of the biological
profile and is accomplished through multiple methods, including:
cranial metrics (13), postcranial metrics (4), dental metrics (58),
dental morphology (912), and macromorphoscopic trait analysis
(1316). Macromorphoscopic traits are quasicontinuous morpho-
logical traits of the cranium, which are reflected as soft-tissue dif-
ferences in the living (16,17). These traits qualify a bones shape,
the presence or absence of a feature, and/or a features degree of
expression (14,18). Macromorphoscopic traits have historical ties
to Hootons Harvard List and were traditionally applied post hoc
to explain and support ancestry estimations (14,18). Using 11 non-
metric traits, Hefner (14) refined, quantified, and standardized their
use in ancestry estimation, placing macromorphoscopics in a more
objective framework. Further studies demonstrated the value of
macromorphoscopic trait analysis when applied to populations
expanded beyond Hefners original study (1923).
The current research addresses long-term intraobserver error on
macromorphoscopic traits in two ways. First, we quantify the
implications of observer error introduced as observer experience
increases or technological improvements are included, in an effort
to measure the impact those factors have on macromorphoscopic
1
Department of Anthropology, Michigan State University, 655 Auditorium
Drive, East Lansing, MI 48824.
*Presented in part at the 69th Annual Scientific Meeting of the American
Academy of Forensic Sciences, February 13-18, 2017, in New Orleans, LA.
Funded in part by the Award No. 2015-DN-BX-K012, awarded by the
National Institute of Justice, Office of Justice Programs, United States
Department of Justice.
The opinions, findings, and conclusions or recommendations expressed in
this presentation are those of the authors and do not necessarily reflect the
views of the Department of Justice.
Received 17 Mar. 2017; and in revised form 28 April 2017; accepted 28
April 2017.
trait data collection. The second objective targets the data collected
in 2002 to ensure reliability and cohesion between the datasets for
inclusion of those data into the Macromorphoscopic Databank
(MaMD), a reference and research tool for macromorphoscopic
trait analysis. To that end, we piloted a fourteen-year (20022016)
intraobserver error study.
Interobserver error quantifies differences in independent
observations of the same feature (or measurement) between two
or more individuals in an effort to identify potential sources of
error. Intraobserver error assesses consistency in observations of
the same feature recorded by a single observer over multiple
attempts (3). We address interobserver error for macromorpho-
scopic data in a separate paper, focusing herein on intraobserver
error in macromorphoscopic data collection.
Metric and nonmetric analyses of cranial and postcranial ele-
ments are commonly used to document and quantify human varia-
tion in biological anthropology. A single observer or multiple
observers may participate in the data collection process, which can
last days, weeks, and/or months. Additional data collection can
occur years after the original data were collected. Inter- and intraob-
server analyses provide insight into the precision of observations
and can aid in the identification of potential sources of error.
Although long-term intraobserver error studies (i.e., >10 years) are
limited in the literature, several studies address the implications of
inter- and intraobserver error, illuminating potential factors con-
tributing to data collection error. In general, these studies come to
similar conclusions about the factors influencing error, which can
include measurement technique, observer experience, and error
inherent in a research design. One major assumption of data analy-
sis is that observations can be replicated over time (2426). In
reality, they can be difficult to replicate with a high degree of preci-
sion (24). Incorrectly repeating some measure or observation is the
very nature of intraobserver error, with roots in two factors.
One factor influencing observer-introduced error (and bias) is
the observers level of experience. Several studies report high

2017 American Academy of Forensic Sciences 1

2 JOURNAL OF FORENSIC SCIENCES
levels of error in data collected by multiple observers of varying
experience levels. The proposed solution is often training prior
to data collection (2527), but this is not always possible. New
method development should consider the role of experience to
achieve consistent scores between observers and between periods
of observation. Additional sources of error are biases built into a
method through problematic protocols such as unclear definitions
or more serious mistakes in method design and implementation.
Jamison and Zegura (28) recommend assessing intravariable cor-
relations to identify error sources. Low levels of correlation may
indicate difficulty in accurately measuring or scoring a variable
(28), which would necessitate refinement and revision of the
method before implementation in research and data collection.
Macromorphoscopic data analysis has been subjected to rela-
tively few intraobserver error studies. Hefners (14) original
paper presented the results of a small-scale intraobserver error
test, but observations were separated by only two weeks.
Regardless, significant intraobserver error was identified for
anterior nasal spine and supranasal suture (14). A separate inter-
observer error study on macromorphoscopic traits found consid-
erable variation in scoring between observers of different
experience levels; some traits had agreement levels lower than
expected for random allocation (29). The authors, again, recom-
mended observers become familiar with identifying and assess-
ing macromorphoscopic traits prior to data collection.
In this study, discrepancies between observations are expected
to result from the following changes in data collection protocols:
(i) the introduction of images in the MMS (Macromorphoscopic)
interface; (ii) alterations in scales/values between 2002 and
2016; (iii) the introduction and implementation of a contour
gauge to evaluate nasal bone contour; and (iv) increased obser-
ver experience. Each of these is briefly addressed.
The MMS v1.61 is a data collection program that represents a
modified, free-standing version of the Macromorphoscopic Mod-
ule in Osteoware (Smithsonian Institution 2016). MMS is a
graphical user interface comprising a standard window with a
definition for each trait and character state, as well as illustra-
tions and descriptive comments. The illustrations offer visual
clarification for each character state (i.e., the possible expres-
sions of a trait) and are supplemented with anatomically based
definitions (18). Second, MMS uses a numerical scoring scale
for the traits. Since its implementation in 2005, some of these
scales have shifted slightly, generally by one digit. For example,
in 2002, a scoring scale ranged from 0 to 4, while in 2016, the
same scale had been adjusted to range between 1 and 5. Third, a
contour gauge was first recommended for use in 2005. This tool
is used for assessing trait expression in nasal bone contour
FIG. 1One example of original paper forms used to score macromorphoscopic traits in 2002.
(NBC) and was first publicly introduced in 2009 (14). A contour
gauge is critical for standardizing NBC scoring; it is a more
objective method to assess the nuances of the individual charac-
ter states. Finally, observer experience will likely have an impact
on trait scores. Observer experience and familiarity with macro-
morphoscopic traits influence trait scoring because more experi-
enced observers have been exposed to a greater range of human
variation. Extreme trait values, often associated with the trait list
approach (15), are much more rare when an observer has docu-
mented crania from all over the world.
Materials and Methods
Ten macromorphoscopic traits were assessed to ensure consis-
tency between the two data collection periods. These include
anterior nasal spine (ANS), inferior nasal aperture (INA), interor-
bital breadth (IOB), malar tubercle (MT), nasal aperture width
(NAW), nasal bone contour (NBC), nasal overgrowth (NO),
postbregmatic depression (PBD), posterior zygomatic tubercle
(PZT), and zygomaticomaxillary suture (ZS). These ten traits
were selected to maximize sample sizes and to evaluate a range
of trait-value possibilities.
Paired data were collected for American Black (n = 127) and
American White (n = 58) individuals from the Robert J. Terry
Collection. The Terry Collection is a large documented skeletal
collection housed at the Smithsonian Institution, National
Museum of Natural History, in Washington, DC. The collection
includes over 1500 American Black and American White indi-
viduals of known biological parameters with birth-years ranging
from 1822 to 1943 (30). The Terry Collection was chosen to
facilitate data comparisons. Macromorphoscopic trait data for the
2002 sample were collected for thesis research using paper forms
and then standard texts, references, and line drawings (Fig. 1).
The same data were recollected in 2016 using MMS v1.61
(Fig. 2), specifically to assess intraobserver error and observer
bias.
Statistical Analyses
The 2002 and 2016 samples were combined into a single data
table for analysis. Several approaches to identify intraobserver
error were considered. First, we implemented a quadratic-
weighted Cohens kappa test to assess error. This test was cho-
sen over a traditional (unweighted) Cohens kappa, as the
weighted approach is better suited to test the agreement of
ordered data (1 < 2 < 3) (31) unlike the traditional test, which
does not control for the grade of disagreement. Following
 KAMNIKAR ET AL. . INTRAOBSERVER ERROR 3

FIG. 2Updated MMS v1.61 interface.

Jamison and Zeguras (28) recommendations for assessing TABLE 1Results of Cohens kappa test for all ten macromorphoscopic
intraobserver error, we also incorporated correlation analyses. A traits.
correlation analysis using the corrplot package in R (32)
explored rater agreement. Kernel density graphs were created to Observed Standard Frequency of
visualize differences between the two years. Separated by ances- Trait Levels Kappa Error Agreements/N %
try and data collection year (2002 or 2016), these graphs provide NBC 5 0.69 0.04 91 77.78
insight into observer error. Finally, classification accuracies from NAW 3 0.64 0.06 116 62.70
a canonical analysis of the principal coordinates using all 10 IOB 3 0.64 0.05 113 61.08
INA 5 0.63 0.01 92 78.63
traits and a reduced model using seven traits with the highest
PBD 2 0.62 0.05 137 74.05
levels of observer agreement were calculated and assessed for NO 2 0.58 0.06 136 73.51
statistically significant differences between the 2002 and 2016 ANS 3 0.49 0.06 78 68.42
samples. MT 4 0.10 0.29 68 36.76
ZS 3 0.06 nc 52 28.11
PZT 4 <0.01 nc 63 34.05
Results
Bolded values represent moderate-to-good observer agreement.
Kernel density estimate plots, by year, are provided in Figures nc = not calculated.
4 through 8. No obvious data collection errors are noted. The
results of the intraobserver error assessments are provided shows neither a positive nor negative correlation for PZT, sug-
below. gesting no agreement between the two collection intervals.
The Cohens kappa test indicates moderate-to-good observer (Fig. 3, left plot). Reassessing the level of agreement (Cohens
agreement for seven of the ten traits between the two observa- kappa) between MT and PZT identified a higher than expected
tion periods. The frequency of agreement between observation level of agreement, particularly compared to the original calcula-
periods ranges between a high of 74.05% (PBD) to as low as tions (Table 2).
28.11% (ZS) (Table 1.). To determine how observer bias and/or observer error affected
Three of the ten macromorphoscopic traits had significantly classification accuracies, two CAP analyses were conducted
different rates of agreement, indicating relatively high intraob- using the chi-square distance metric (x-validated with substitu-
server error. The three underperforming traits include MT (confi- tion). Classification accuracies were assessed between years
dence interval jw = 0.0290.046; levels = 4); ZS (CI using all ten traits and, again, using the seven traits with high
jw = 0.0310.093; levels = 3); and PZT (CI jw = 0.0290.046; observer agreement in a reduced model. These accuracies
levels = 4). assessed significant differences, by year. Despite the observer
Figure 3 illustrates the correlation coefficients between ANS, error noted above, all of the models had high classification accu-
INA, IOB, NAW, NO, PBD, and PZT. All intertrait compar- racies and are similarly distributed. The 2002 sample correctly
isons, except PZT, were positively correlated, indicating gener- classified 92.2% of the sample using the seven-variable model
ally good observer agreement, or low observer bias. The matrix and 93.5% of the sample using the ten-variable model. Similarly,
4 JOURNAL OF FORENSIC SCIENCES

FIG. 3Plots illustrating correlation coefficients before and after data reconciliation. Blue values indicate a positive correlation between traits, and red
values indicate a negative correlation between traits.

TABLE 2Updated Cohens kappa tests for malar tubercle (MT) and poste- TABLE 4Results of the CAP analysis using seven variables (chi-square
rior zygomatic tubercle (PZT). distance metric).

Observed Standard Frequency of Group Percent Correct Black White

Trait Levels Kappa Error Agreement/N %
(2002)
MT 4 0.36 0.07 89 76.07 Black 92.71 89 7
PZT 4 0.38 0.06 103 65.19 White 95.35 2 41
Total: 93.53 91 48
(2016)
Black 89.66 104 12
TABLE 3Results of the CAP analysis using all ten variables (chi-square White 94.12 3 48
distance metric). Total: 91.02 109 60

Group Percent Correct Black White

(2002) Observer Experience
Black 90.91 100 10
White 95.56 2 43 Trait frequencies for the individual character states differed
Total: 92.26 102 53 between the two datasets. These differences are potentially
(2016) related to observer experience and an increased exposure to a
Black 87.29 103 15 wider range of human variation between the data collection peri-
White 90.38 5 47
Total: 88.24 108 62 ods. Augmented observer experience likely results from more
familiarity with the intricacies of the macromorphoscopic method
and with the individual character state expressions.
Some of the differences noted in the kernel density plots
the 2016 sample correctly classified 91.1% and 88.2% of the (Figs 48) illustrate shifts in character state frequencies from
sample using these same models, respectively (Tables 3 and 4.) extreme expressions (either minimum or maximum scores) to more
intermediate expressions. Extreme character expressions were less
frequent in the 2016 data for the following traits: ANS, IOB, and
Discussion
NAW (Fig. 4a,b); however, scores for INA included more extreme
To identify intraobserver error and document potential obser- trait values. All four of these traits are expressed as three or more
ver bias, three general patterns (i.e., observer bias and/or intraob- character states. Comparison of the ANS data highlights this shift;
server error) influencing trait scores were recognized. These the most frequent expression in 2002 for both American Whites
patterns and data discrepancies include (i) differences attributed and American Blacks was the minimum character expression
to observer experience; (ii) an increase in objectivity through the (ANS = 1). In the 2016 sample, however, the greatest frequencies
introduction of new technologies for macromorphoscopic trait corresponded to the intermediate character expression (ANS = 2).
scoring; and (iii) the identification of errors inherent in the origi- Trait frequencies for IOB and NAW were similarly shifted. INA
nal methodological design. Each of these is outlined in detail scores were the exception. Within both the American Black and
below. the American White sample, the most frequent expressions shifted

FIG. 4Density plots for traits exhibiting movement toward their middle expression. Dashed lines represent sectioning points used in OSSA.
to a more extreme character states (i.e., American Black = INA 2
to INA 1; American White sample = INA 4 to INA 5).
The distribution of binary traits, dichotomous variables defin-
ing presence/absence data, also proved interesting. The kernel
density plots for NO illustrate a very low rate of change in trait
frequencies between observation periods. Values for PBD show
discrepancies (Fig. 5), which we attribute to observer experience.
As an observer becomes more familiar with the manifestation of
PBD, more subtle depressions are scored as present.
A proper understanding of macromorphoscopic trait expression
is linked to experiencenovice observers should rely heavily on
written definitions and images, while more experienced observers
will only do so initially or when a unique expression is encoun-
tered. Any difficulty distinguishing character state manifestations
may be attributed to poorly worded or inadequate definitions or
the failure of the method to capture all of a traits character state
expressions. We propose amendments to the MMS method to
clarify unclear trait definitions in a forthcoming publication,
specifically for IOB and NAW. They suggest scoring these fea-
tures as a ratio of the relative size of the feature to the entire mid-
facial region, which Hefner (14) attempted but failed to do.
KAMNIKAR ET AL. . INTRAOBSERVER ERROR 5
Observer experience is potentially the most interesting factor
influencing observer error and bias. Intermediate values for all ten
traits demonstrated a lower proportion of agreement between
observation periods; the 2016 sample demonstrated a higher pro-
portion of intermediate scores, suggesting more experienced
observers are less likely to assign extreme trait values, at least for
the American Black and American White samples used herein.
This prompts an interesting question: When scoring IOB and
NAW, would you preferentially accept observations provided by a
seasoned expert over those provided by a novice? The only valid
way to answer this question is validation of both observations in a
quantifiable manner. Future research should identify methods to
quantify observer experience and assess its true role in the collec-
tion of macromorphoscopic trait data. Observer experience seem-
ingly plays a role when scores are binary, and the identification of
modest expressions is fundamental, such as when scoring PBD.
New and Improved Technologies
As Hefner (14,33,34) first introduced the MMS method, adjust-
ments have been made to the scaling system (i.e., character states)
6 JOURNAL OF FORENSIC SCIENCES

FIG. 5Density plots for binary traits. Dashed lines represent sectioning points used in OSSA.

through the institution of new technologies, such as a contour
gauge and necessary minor adjustments in scoring strategies.
These modifications may impact trait frequencies, particularly if
errors occur during data maintenance (e.g., database management).
Kernel density plots identified scaling difference for ZS (Fig. 6)
and the impact of a contour gauge on scores for NBC (Fig. 7). In
2002, the scoring scale for ZS ranged from 0 to 3; by 2016, the
scale excluded a value for unobservable and decreased in range
to only include values between 0 and 2. Issues like this one may
indicate that some of the data from 2002 should not be included in
the final, released version of the MaMD. Two issues were also
identified in the NBC density graph. First, using a contour gauge
 KAMNIKAR ET AL. . INTRAOBSERVER ERROR 7

FIG. 6Density plots illustrating differences in scales.

FIG. 7Density plot illustrating the improvement of scores with the introduction of new technologies. Dashed lines represent sectioning points used in
OSSA.

produced well-defined, although relatively evenly distributed well-defined character state boundaries. This issue is less easily
clusters, indicating greater objectivity in NBC scores. Second, the remedied. During the development of the OSSA method (15), data
2002 trait score frequencies show data smoothing without for NBC were inadvertently transformed (optimized) in the main
8 JOURNAL OF FORENSIC SCIENCES
FIG. 8Density plots for Posterior Zygomatic Tubercle (PZT) and Malar Tubercle (MT).
data file. As such, these data no longer represent actual observa-
tions and as such cannot be included in the MaMD.
Error Inherent in the Method
Hefners (14) original study shows moderate-to-high intraob-
server agreement between all traits and suggests traits can be
scored with repeatability, although that study did identify SPS
and ANS as the traits with the lowest level of repeatability. We
did not test intraobserver error on SPS in the current study, as
Hefner (14) suggested eliminating SPS from consideration until
a better definition is provided. However, we were able to iden-
tify several problematic traits. In an attempt to understand the
observed bias or error in PZT scores between 2002 and 2016,

we reexamined Hefners (33) thesis, where the majority of these
traits were first clarified. We noted inconsistencies therein
between the current definitions for PZT and MT and a striking
correspondence to older definitions of traits coded with similar
three letter abbreviations. In his thesis, Hefner used the abbrevi-
ation MT for the marginal tubercle, which is now identified
as the posterior zygomatic tubercle (PZT); ZMT originally
identified the zygomatic tubercle, but that trait is now referred
to as the malar tubercle (MT). Further investigation of the fre-
quency table data in that thesis identified the inadvertent inter-
change between these two variables. Exchanging the 2002 data
between MT and PZT corrected the issues we identified above
in the intertrait correlations, which now demonstrate a signifi-
cant correlation (Fig. 3, right matrix). Jamison and Zegura (28)
noted the importance of testing intraobserver error to identify
such issues in a dataset. Once the data were exchanged, density
plots for trait frequencies of MT and PZT (Fig. 8) show no
significant differences. In fact, PZT frequencies in the 2002 and
2016 data for American Whites are almost identical. Trait
frequencies for MT do shift to lower values in the American
White sample, but score frequencies for MT remain similar
among the American Black sample for both data collection
periods.
Results from this study may explain some of the error identi-
fied by other practitioners. For example Klales and Kenyhercz
(29) compared their results to Hefners original study (14) and
suggest sectioning points between groups similar to Hefner and
Ousleys OSSA (15), with the exception of NBC. Klales
and Kenyhercz (29) also report lower frequencies of extreme
scores for ANS, INA, NAW, and PBD, but a higher frequency
of more extreme scores reported for IOB and transverse palatine
suture (not included in the current study). They suggest these
differences may reflect temporal differences between the two
sample populations; however, the differences between the Terry
and Todd collection could also reflect errors in the original data
collection protocol from 2002 and greater experience with
macromorphoscopic traits.
Conclusions
Quantifying short- and long-term intraobserver error is impor-
tant for validating more broadly used methods. Our objectives
were to identify causes of intraobserver error in macromorpho-
scopic trait data collection and to determine whether data col-
lected in 2002 are reliable and appropriate for use as reference
data in the MaMD.
Intraobserver error tests, in this case, identified mistakes with
trait nomenclature that occurred early on in the methods intro-
duction and subsequent development. This error was not noted
earlier, because frequency tables between the two traits were not
significantly different. This mistake could potentially impact val-
idation studies and interobserver error tests. At times, these traits
were identified as inadequate for ancestry assessments. In point
of fact, the opposite may be true. Statistical reanalysis of PZT
and MT could clarify this predicament.
Overall, error between traits in this study was negligible; thus
confirming that most of the 2002 data [ANS, INA, IOB, MT,
NAW, NO, PBD, PZT, and ZS] are suitable for inclusion in the
MaMD. The 2002 data for NBC and SPS will not be included
in the reference databank.
This study set out to evaluate and quantify long-term intraob-
server error on macromorphoscopic trait analysis. Our results
suggest familiarity and training will decrease observer error and
KAMNIKAR ET AL. . INTRAOBSERVER ERROR 9
supports Klales and Kenyherczs (29) recommendation of a
training period prior to macromorphoscopic trait data collection.
As new technologies supplant older approaches, further reduc-
tion in observer error is possible.
Acknowledgments
The authors would like to thank D. Hunt and C.J. Dudar from
the Smithsonian Institution, National Museum of Natural History
for access to the collections, and M. Ancern for special consider-
ations. We would also like to thank Kandus C. Linde for assis-
tance with the collection of the 2016 data. The authors would
also like to thank the two anonymous reviewers, whose com-
ments and insights strengthened the manuscript.
References
1. Ousley SD, Jantz RL. Fordisc 3 and statistical methods for estimating
sex and ancestry. In: Dirkmaat DC, editor. A companion to forensic
anthropology. Oxford, U.K.: Wiley-Blackwell, 2012;31129.
2. Konigsberg L, Algee-Hewitt B, Steadman DW. Estimation and evidence
in forensic anthropology: sex and race. Am J Phys Anthropol
2009;139:7790.
3. Dudzik B, Kolatorowicz A. Craniometric data analysis and estimation of
biodistance. In: Pilloud M, Hefner J, editors. Biological distance analysis:
forensic and bioarchaeological perspectives. New York, NY: Elsevier,
2016;3560.
4. Spradley MK. Ancestry estimation from the postcranial skeleton. In:
Berg G, Taala S, editors. Biological affinity in forensic identification of
human skeletal remains: beyond black and white. Boca Raton, FL: CRC
Press, 2015;8394.
5. Pilloud M, Kenyhercz MW. Dental metrics in biodistance analysis. In:
Pilloud M, Hefner JT, editors. Biological distance analysis: forensic
and bioarchaeological perspectives. New York, NY: Elsevier,
2016;13555.
6. Harris E, Foster C. Size matters: discrimination between American blacks
and whites, males and females, using tooth crown dimensions. In: Berg
G, Taala S, editors. Biological affinity in forensic identification of
human skeletal remains: beyond black and white. Boca Raton, FL: CRC
Press, 2015;20938.
7. Pilloud M, Hefner JT, Hanihara T, Hayashi A. The use of tooth crown
measurements in the assessment of ancestry. J Forensic Sci
2014;59:1493501.
8. Lease L, Sciulli P. Brief communication. discrimination between
European-American and African-American children based on deciduous
dental metrics and morphology. Am J Phys Anthropol 2005;126:
5660.
9. Pilloud M, Edgar H, George RW, Scott G. Dental morphology in biodis-
tance analysis. In: Pilloud M, Hefner JT, editors. Biological distance
analysis: forensic and bioarchaeological perspectives. New York, NY:
Elsevier, 2016;10933.
10. Edgar H. Prediction of race using characteristics of dental morphology.
J Forensic Sci 2005;50:26973.
11. Edgar H. Estimation of ancestry using dental morphological characteris-
tics. J Forensic Sci 2013;58:S38.
12. Scott G. Population variation of Carabellis trait. Hum Biol 1980;52:63
78.
13. Gill G, Rhine S. Skeletal attribution of race: methods for forensic anthro-
pology. Albuquerque, NM: Anthropological Papers of the Maxwell
Museum of Anthropology, 1990.
14. Hefner JT. Cranial nonmetric variation and estimating ancestry. J Foren-
sic Sci 2009;54:98595.
15. Hefner JT, Ousley S. Statistical classification methods for estimating
ancestry using morphoscopic traits. J Forensic Sci 2014;59:88390.
16. Hefner JT. Biological distance analysis, cranial morphoscopic traits, and
ancestry assessment in forensic anthropology. In: Pilloud M, Hefner JT,
editors. Biological distance analysis: forensic and bioarchaeological per-
spectives. New York, NY: Elsevier, 2016;30115.
17. Brues AM. The once and future diagnosis of race. Skeletal attribution of
race: methods for forensic anthropology. Albuquerque, NM: Maxwell
Museum of Anthropology, 1990;19.
18. Plemons AM, Hefner JT. Ancestry estimation using macromorphoscopic
traits. Acad Forensic Pathol 2016;6:40012.
10 JOURNAL OF FORENSIC SCIENCES
19. LAbbe E, Rooyen C, Nawrocki S, Becker P. An evaluation of non-
metric cranial traits used to estimate ancestry in a South African sample.
Forensic Sci Int 2011;209:E17.
20. Hurst C. Morphoscopic trait expressions used to identify Southwest His-
panics. J Forensic Sci 2012;57:85965.
21. Kenyhercz MW, Klales AR, Rainwater CW, Fredette SM. The optimized
summed scored attributes method for the classification of U.S. Blacks
and Whites: a validation study. J Forensic Sci 2017;62:17480.
22. Monsalve T, Hefner JT. Macromorphoscopic trait expression in a cranial
sample from Medellin, Colombia. Forensic Sci Int 2016;266:574.
23. Birkby W, Fenton T, Anderson B. Identifying southwest Hispanics using
nonmetric traits and the cultural profile. J Forensic Sci 2008;53:2933.
24. Utermohle C, Zegura S. Intra and interobserver error in craniometry: a
cautionary tale. Am J Phys Anthropol 1982;57:30310.
25. Utermohle C, Zegura S, Heathcore G. Multiple observers, humidity, and
choice of precision statistics: factors influencing craniometric data qual-
ity. Am J Phys Anthropol 1983;61:8595.
26. Yzerniac S, Lougheed S, Handford P. Measurement error and morpho-
metric studies: statistical power and observer experience. System Biol
1992;41:47182.
27. Perini T, Oliveira GL, Ornellas JS, Oliveira FP. Technical error of mea-
surement in anthropometry. Revista Brasileira de Medicina do Esporte
2005;11:815.
28. Jamison P, Zegura S. A univariate and multivariate examination of
measurement error in anthropometry. Am J Phys Anthropol 1974;40:
197203.
29. Klales AR, Kenyhercz MW. Morphological assessment of ancestry using
cranial macromorphoscopics. J Forensic Sci 2014;60:1320.
30. Usher B. Reference samples: the first step in linking biology and age in
the human skeleton. In: Hoppa R, Vaupel J, editors. Paleodemography:
age distributions from skeletal samples. Cambridge, U.K.: Cambridge
University Press, 2002;2947.
31. Cohen J. A coefficient of agreement for nominal scales. Edu Psychol
Meas 1960;20:3746.
32. Wei T. Corrplot: visualization of a correlation matrix, in R package.
Vienna, Austria: R Foundation for Statistical Computing, 2013.
33. Hefner JT. Assessing nonmetric cranial traits currently used in forensic
determination of ancestry [masters thesis]. Gainesville, FL: The Univer-
sity of Florida, 2003.
34. Hefner JT. The statistical determination of ancestry using cranial non-
metric traits [doctoral dissertation]. Gainesville, FL: The University of
Florida, 2007.
Additional information and reprint requests:
Kelly R. Kamnikar, M.A.
Department of Anthropology
Michigan State University
655 Auditorium Drive
East Lansing
MI 48824
E-mail: kellykamnikar@gmail.com