This article was downloaded by: [Dalhousie University]

On: 24 October 2012, At: 03:37

Publisher: Psychology Press
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Measurement: Interdisciplinary Research

and Perspectives
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/hmes20

Diagnostic Classification Models: Which

One Should I Use?
a
Hong Jiao
a
Department of Measurement, Statistics, and Evaluation, University
of Maryland,

Version of record first published: 05 Mar 2009.

To cite this article: Hong Jiao (2009): Diagnostic Classification Models: Which One Should I Use?,
Measurement: Interdisciplinary Research and Perspectives, 7:1, 65-67

To link to this article: http://dx.doi.org/10.1080/15366360902799869

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation
that the contents will be complete or accurate or up to date. The accuracy of any
instructions, formulae, and drug doses should be independently verified with primary
sources. The publisher shall not be liable for any loss, actions, claims, proceedings,
demand, or costs or damages whatsoever or howsoever caused arising directly or
indirectly in connection with or arising out of the use of this material.
 COMMENTARIES ON ISSUE 6(4) 65

Diagnostic Classification Models: Which One Should I Use?

Hong Jiao
Commentaries On Issue 6(4)

Department of Measurement, Statistics, and Evaluation

University of Maryland

Diagnostic assessment is currently a very active research area in educational measurement.

Literature related to diagnostic modeling has been in existence for several decades, but a great deal
of research has been conducted within the last decade or so, especially within the last five years. The
Downloaded by [Dalhousie University] at 03:37 24 October 2012

dramatic increase of work in this area might be driven by the interplay between assessment and
instruction and by educational policy. The No Child Left Behind Act (NCLB, 2001) requires that all
students make adequate yearly progress and ultimately reach a proficient level. State assessments
thus are constructed to fulfill these educational policy requirements. Current state assessments are
constructed to classify students into different achievement levels; most often an item response the-
ory (IRT) model is adopted for analyzing the test and providing summative information regarding
the proficiency level of each student. With the implementation of such statewide assessments,
achievement gaps were found among different subgroups of the student population as labeled by
ethnicity, socioeconomic status, or English language proficiency status. Current available summa-
tive assessments and prevalent psychometric models can detect the achievement gap between
groups, but are not efficient in extracting diagnostic information from the assessment. Many test
stakeholders call for the assessment community to provide diagnostic information so that crucial
remedial instruction can be directed to those who are diagnosed with a deficit in some skills. Thus,
achievement gaps can be filled through adapted instruction based on the diagnosis by assessment.
Diagnostic models analyze item response data to generate profiles of mastery and nonmastery
of the skills assessed. The cognitive process involved in answering a test item is complicated.
Thus, modeling this process is also very complicated and challenging. Potential users of diagnos-
tic models might be stunned by the large number of available diagnostic models (62 models as
indicated in Fu and Li, 2007). They may ask which diagnostic model should be used to analyze
data to extract the needed diagnostic information. In general, whenever we apply a measurement
model to test data, we need to know the conditions/assumptions in terms of content, statistics, and
practicality (administration settings) to adopt the most appropriate one. This review, along with
several other reviews listed by the authors, is a timely summary for potential users of diagnostic
models and will be a very good starting point to get an overview of diagnostic models.
This review contains five sections. The first three define diagnostic classification models
(DCM) by unifying the currently available diagnostic models by finding the commonality
among those models and setting boundaries for the DCMs in order to distinguish them from
other multidimensional models and among themselves. The clear definitions of the DCMs pre-
sented in this review help the conceptualization of the models from both cognitive and statistical
perspectives. The last two sections summarize the technical details related to model features,

Correspondence should be addressed to Hong Jiao, Department of Measurement, Statistics, and Evaluation, University
of Maryland, 1230B Benjamin Building, College Park, MD 20742-1115. E-mail: hjiao@umd.edu
 66 COMMENTARIES ON ISSUE 6(4)

estimation, and model fit issues. These issues as discussed are properly identified to demonstrate
the complexity of the DCMs, such as the model identification, parameterization, and conver-
gence issues and the availability and validity of software that are very relevant to practitioners
when considering the adoption of a DCM. The review ends with the discussion of possible
research that provides a potential view of DCM development.
This review emphasizes and elaborates the implications of the confirmatory nature of the defined
DCMs. Due to the confirmatory nature of the models, the correct specification of the Q-matrix is a
core component in the model application. But readers are reminded here of the exploratory nature of
the process of constructing and validating the Q-matrix. Most often, the Q-matrix is constructed
based on cognitive theory (Henson and Templin, 2007) to determine the loading of the item on a spe-
cific attribute or skill. However, in analyzing the data using DCMs, some of the attribute loading
Downloaded by [Dalhousie University] at 03:37 24 October 2012

might be adjusted based on initial analysis results. Henson, Templin, and Willse (2007) showed how
to use the loglinear cognitive diagnosis model (LCDM) in an exploratory sense to identify the cogni-
tive structure of an item and give insights to the cognitive process used in answering an item, thus
providing empirical information regarding the appropriate model for each item.
Future research listed in this review around topics such as model performance in complex sam-
pling design, missing data, testlet effects, and differential item and bundle functioning are interest-
ing and important topics to be tapped. If a DCM is applied in educational assessment, testlet-based
assessments such as passage-based reading comprehension tests, graph-based math assessments,
and scenario-based science assessments are common assessment formats. Application of a DCM
to such assessments and comparison among DCMs under such testing conditions will definitely
provide insights into the application of the DCMs. Other possible areas of research might be the
implications of classification errors in estimation and the use of test scores from DCMs. In the
application of the DCMs, false negative errors (students who have mastered a skill but are diag-
nosed as having nonmastery) and false positive errors (students who have not mastered a skill but
diagnosed as having mastery) may have some impact on the utility of the extracted diagnostic
information. The role of classification errors in the DCMs might be a relevant line of research.
At present, most diagnostic analyses are conducted using the test data from assessments
which are originally constructed to provide summative information, not for diagnosis (Hensen &
Templin, 2007; Xu & von Davier, 2006). Providing diagnostic information using summative
assessment is a challenging task since our assessment uses a sample from the content domains to
be assessed. When a sample of the content domain is used to seek diagnostic information over
the whole content domain, the diagnosis based on the sample may not provide a comprehensive
picture regarding the sources of problems in students’ mastery of skills. Whether diagnosis
based on summative assessment is adequate awaits further exploration. Herein, it is recom-
mended that the combination of end-of-unit assessment and end-of-year assessment may be a
possible option to extract accurate diagnostic information. The end-of-unit diagnostic assess-
ment provides timely feedback to students and teachers regarding where remedial work can be
directed. Although end-of-year diagnostic assessment provides diagnostic information regarding
where problems exist, it is more likely a post hoc analysis or summary of the problems. Students
with problems have to wait for the whole year to be assessed and to be provided with informa-
tion regarding where to improve their learning. The measurement community has ample experi-
ence in constructing summative assessment. More guidance is needed on how to construct a test
targeted at providing diagnostic information and a unified system of bridging the end-of-unit
and end-of-year diagnostic assessments (Anozie & Junker, 2006; Junker, 2007).
 COMMENTARIES ON ISSUE 6(4) 67

In general, this review paper summarizes the key components in the application of the
DCMs. It provides practitioners with some important information regarding conceptual frame-
work, utility, application settings, and parameter estimation of the DCMs. This paper is a good
addition to the review literature on diagnostic modeling.

ACKNOWLEDGMENT

The author would like to express his deepest gratitude to his father Jinkuan Jiao for his love,
inspiration, and encouragement.
Downloaded by [Dalhousie University] at 03:37 24 October 2012

REFERENCES

Anozie, N. O., & Junker, B. W. (2006). Predicting end-of-year accountability assessment scores from monthly student
records in an online tutoring system. Presented at the Workshop on Educational Data Mining at the 21st National
Conference on Artificial Intelligence (AAAI 2006). Boston, USA. July 16–17, 2006.
Fu, J., & Li, Y. (2007, April). An integrated review of cognitively diagnostic psychometric models. Paper presented at
the annual meeting of the National Council on Measurement in Education, Chicago, IL.
Henson, R. A., & Templin, J. (2007, April). Large-scale language assessment using cognitive diagnosis models. Paper
presented at the annual meeting of the National Council on Measurement in Education (NCME), Chicago, IL.
Henson, R. A., Templin, J., & Willse, J. (2007, April). Defining a family of cognitive diagnosis models. Paper presented
at the annual meeting of the National Council on Measurement in Education (NCME), Chicago, IL.
Junker, B. W. (2007). Some issues and applications in cognitive diagnosis and educational data mining (PowerPoint).
Keynote presentation to the 2007 Annual Meeting of the Psychometric Society, Tokyo, Japan.
Rupp, A. A. & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive
review of the current state-of-the-art. Measurement, 6(4), 219–262
Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (Research Report No. RR-06-08).
Princeton, NJ: Educational Testing Service.
Measurement, Vol. 7, No. 1, February 2009: pp. 1–14
1536-6359
1536-6367
HMES
Measurement

Some Notes on the Reinvention of Latent Structure

Models as Diagnostic Classification Models
Matthias von Davier
Commentaries On Issue 6(4)

Educational Testing Service

Princeton University, Princeton, New Jersey

INTRODUCTION

Rupp & Templin (2008) have done a remarkable job with their paper. The overview tries and to
a large extent succeeds in being a balanced one—one that does not only include developments

Correspondence should be addressed to Matthias von Davier, Princeton University, ETS, MS 02-T, Princeton, NJ
08541. E-mail: mvondavier@ets.org