Standardized Diagnostic Assessment Design and Analysis: Key Ideas from Modern Measurement Theory

As a response to the ever-increasing demand for diagnostic assessments that can provide more informative feedback about students’ knowledge state, assessment design frameworks are needed that can help designers incorporate relevant cognitive theories into the development, implementation, and analysis process. In this chapter, we describe one prominent framework for principled diagnostic assessment design called evidence-centered design (ECD) (e.g., Mislevy et al. A brief introduction to evidence-centered design. CSE Technical Report 632. Los Angeles: The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, 2004) as well as a class of statistical models called diagnostic classification models (DCMs) (e.g., Rupp et al. Diagnostic assessment methods: theory and application. The Guilford Press, New York, 2010) that can make inferences about student profiles within this framework. With respect to DCMs we describe key terminology, concepts, and a unified estimation framework known as the log-linear cognitive diagnosis model (LCDM) (Henson et al. Psychometrika 74(2):191–210, 2009). We present three examples to illustrate how particular DCMs can be specified to address different cognitive theories concerning the process of knowledge processing. At the end of this chapter, we illustrate the utility of DCMs with a real-data set on arithmetic ability in elementary school to illustrate the type of diagnostic inferences we can make about students’ attribute profiles.

[1]  Ronald H. Stevens,et al.  Measuring Complex Features of Science Instruction: Developing Tools to Investigate the Link Between Teaching and Learning , 2009 .

[2]  Mark J. Gierl,et al.  Identifying Content and Cognitive Dimensions on the SAT , 2005 .

[3]  Alan M. Lesgold,et al.  Diagnostic Monitoring of Skill and Knowledge Acquisition , 2013 .

[4]  R. Hambleton,et al.  Item Response Theory , 1984, The History of Educational Measurement.

[5]  S. Embretson.  A cognitive design system approach to generating valid tests : Application to abstract reasoning , 1998 .

[6]  Robert J. Mislevy,et al.  Specifying and Refining a Measurement Model for a Computer-Based Interactive Assessment , 2004 .

[7]  S. Toulmin.  The uses of argument , 1960 .

[8]  Matthias von Davier,et al.  A GENERAL DIAGNOSTIC MODEL APPLIED TO LANGUAGE TESTING DATA , 2005 .

[9]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[10]  Sophia Rabe-Hesketh,et al.  Generalized latent variable models: multilevel, longitudinal, and structural equation models , 2004 .

[11]  K. Tatsuoka,et al.  Application of the rule-space procedure to language testing: examining attributes of a free response listening test , 1998 .

[12]  André A. Rupp,et al.  A practical illustration of multidimensional diagnostic skills profiling: Comparing results from confirmatory factor analysis and diagnostic classification models , 2009 .

[13]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[14]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[15]  Jianhong Wu,et al.  Data clustering - theory, algorithms, and applications , 2007 .

[16]  R. Glaser,et al.  Knowing What Students Know: The Science and Design of Educational Assessment , 2001 .

[17]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[18]  M. Davier.  Hierarchical mixtures of diagnostic models , 2010 .

[19]  D. Borsboom.  Educational Measurement (4th ed.) , 2009 .

[20]  R. P. McDonald,et al.  Test Theory: A Unified Treatment , 1999 .

[21]  Louis V. DiBello,et al.  31A Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models , 2006 .

[22]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[23]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[24]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[25]  Louis A. Roussos,et al.  The fusion model skills diagnosis system , 2007 .

[26]  Robert J. Mislevy,et al.  A Bayes net approach to modeling learning progressions and task performances , 2009 .

[27]  R. Linn.  Educational Testing and Assessment: Research Needs and Policy Issues. , 1986 .

[28]  J. D. L. Torre,et al.  DINA Model and Parameter Estimation: A Didactic , 2009 .

[29]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[30]  Douglas Steinley,et al.  K-means clustering: a half-century synthesis. , 2006, The British journal of mathematical and statistical psychology.

[31]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[32]  H. Akaike.  A new look at the statistical model identification , 1974 .

[33]  M. Oliveri,et al.  The Learning Sciences in Educational Assessment: The Role of Cognitive Models , 2011, Alberta Journal of Educational Research.

[34]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .

[35]  Robert J. Mislevy,et al.  Putting ECD into Practice: The Interplay of Theory and Data in Evidence Models within a Digital Learning Environment , 2012, EDM 2012.

[36]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[37]  André A. Rupp,et al.  An NCME Instructional Module on Booklet Designs in Large‐Scale Assessments of Student Achievement: Theory and Practice , 2009 .

[38]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[39]  André A. Rupp,et al.  The Impact of Model Misspecification on Parameter Estimation and Item‐Fit Assessment in Log‐Linear Diagnostic Classification Models , 2012 .

[40]  Rebecca Nugent,et al.  Subspace Clustering of Skill Mastery: Identifying Skills that Separate Students , 2009, EDM.

[41]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[42]  Mark J. Gierl,et al.  Cognitive diagnostic assessment for education: Theory and applications. , 2007 .

[43]  Rebecca Nugent,et al.  Skill Set Profile Clustering: The Empty K-Means Algorithm with Automatic Specification of Starting Cluster Centers , 2010, EDM.

[44]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[45]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[46]  A. Agresti.  Analysis of Ordinal Categorical Data: Agresti/Analysis , 2010 .

[47]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[48]  Magdalena Mo Ching 莫慕貞 Mok.  Self-directed learning oriented assessment: Assessment that informs learning & empowers the learner , 2010 .