Application of Conditional Means for Diagnostic Scoring

In educational assessment, demand for diagnostic information from test results has prompted the development of model-based diagnostic assessments. To determine student mastery of specific skills, a number of scoring approaches, including subscore reporting and probabilistic scoring solutions, have been developed to score diagnostic assessments. Although each approach has a unique set of limitations, these approaches are, nevertheless, often used in diagnostic scoring, whereas an alternative approach, Complex Sum Scores (CSS), has not received much attention yet. With the process of developing model-based diagnostic assessments becoming increasingly complex, we revisit the CSS and demonstrate two applications of the CSS in the development of diagnostic assessments. Two applications include: (a) illustrating and validating skills within the model, and (b) partial mastery scoring using model-based distractors. By demonstrating the two applications, we aim to show how model-based diagnostic assessments can be developed and scored using the CSS scoring approach, the results of which can be used by teachers to inform teaching and learning.

[1]  Mark J. Gierl,et al.  Defining and Evaluating Models of Cognition Used in Educational Measurement to Make Inferences about Examinees' Thinking Processes. , 2007 .

[2]  Mark J. Gierl,et al.  Making Diagnostic Inferences about Cognitive Attributes Using the Rule-Space Model and Attribute Hierarchy Method. , 2007 .

[3]  S. Haberman.  WHEN CAN SUBSCORES HAVE VALUE , 2005 .

[4]  Jonathan Templin,et al.  Using Efficient Model Based Sum‐Scores for Conducting Skills Diagnoses , 2007 .

[5]  Howard Wainer,et al.  Augmented Scores-"Borrowing Strength" to Compute Scores Based on Small Numbers ofltems , 2001 .

[6]  Mark Wilson,et al.  Measuring Progressions: Assessment Structures Underlying a Learning Progression , 2009 .

[7]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[8]  Ying Cui,et al.  The Hierarchy Consistency Index: Evaluating Person Fit for Cognitive Diagnostic Assessment , 2009 .

[9]  R. Mislevy.  Evidence and inference in educational assessment , 1994 .

[10]  Sandip Sinharay,et al.  How Often Do Subscores Have Added Value? Results from Operational and Simulated Data , 2010 .

[11]  Richard J. Patz,et al.  Making the Most of What We Have: A Practical Application of Multidimensional Item Response Theory in Test Scoring , 2005 .

[12]  Paul D. Nichols,et al.  A Framework for Developing Cognitively Diagnostic Assessments , 1994 .

[13]  Susan E. Embretson,et al.  Applications of Cognitive Design Systems to Test Development , 1994 .

[14]  Gautam Puhan,et al.  Reporting Diagnostic Scores in Educational Testing: Temptations, Pitfalls, and Some Solutions , 2010, Multivariate behavioral research.

[15]  Matthias von Davier,et al.  A general diagnostic model applied to language testing data. , 2008, The British journal of mathematical and statistical psychology.

[16]  Joanna S. Gorin.  Cognitive Diagnostic Assessment for Education: Test Construction and Diagnostic Testing , 2007 .

[17]  Oksana Babenko,et al.  Comparison and Properties of Correlational and Agreement Methods for Determining Whether or Not to Report Subtest Scores , 2014 .

[18]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[19]  Sarah M. Hartz,et al.  A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. , 2002 .

[20]  Mark Gierl,et al.  Reliability and Attribute-Based Scoring in Cognitive Diagnostic Assessment. , 2009 .