Cognitive Diagnostic Assessment for Education: The Demand for Cognitive Diagnostic Assessment

In this chapter, we explore the nature of the demand for cognitive diagnostic assessment (CDA) in K–12 education and suggest that the demand originates from two sources: assessment developers who are arguing for radical shifts in the way assessments are designed, and the intended users of large-scale assessments who want more instructionally relevant results from these assessments. We first highlight various themes from the literature on CDA that illustrate the demand for CDA among assessment developers. We then outline current demands for diagnostic information from educators in the United States by reviewing results from a recent national survey we conducted on this topic. Finally, we discuss some ways that assessment developers have responded to these demands and outline some issues that, based on the demands discussed here, warrant further attention. THE DEMAND FOR COGNITIVE DIAGNOSTIC ASSESSMENT FROM ASSESSMENT DEVELOPERS To provide the context for assessment developers' call for a revision of contemporary assessment practices that, on the whole, do not operate within a cognitive framework, we offer a perspective on existing CDA literature, and we outline the differences between psychometric and cognitive approaches to assessment design. The phrases working within a cognitive framework, cognitively principled assessment design , and cognitive diagnostic assessment are used interchangeably throughout this chapter. They can be generally defined as the joint practice of using cognitive models of learning as the basis for principled assessment design and reporting assessment results with direct regard to informing learning and instruction.

[1]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[2]  Gerhard H. Fischer,et al.  Some Applications of Logistic Latent Trait Models with Linear Constraints on the Parameters , 1982 .

[3]  S. Whitely.  Construct validity: Construct representation versus nomothetic span. , 1983 .

[4]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[5]  S. Embretson.  CONSTRUCT VALIDITY: CONSTRUCT REPRESENTATION VERSUS NOMOTHETIC SPAN , 1983 .

[6]  R. Sternberg.  Human Abilities: An Information Processing Approach , 1985 .

[7]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[8]  Katherine P. Divine,et al.  Does Interpretive Test Score Information Help Teachers , 1991 .

[9]  P. Nichols.  A Framework for Developing Assessments That Aid Instructional Decisions. , 1993 .

[10]  Paul D. Nichols,et al.  A Framework for Developing Cognitively Diagnostic Assessments , 1994 .

[11]  Robert J. Mislevy,et al.  Test Theory for A New Generation of Tests , 1994 .

[12]  S. Chipman,et al.  Cognitively diagnostic assessment , 1995 .

[13]  Robert J. Mislevy,et al.  Test Theory Reconceived , 1996 .

[14]  Ronald K. Hambleton,et al.  Are NAEP Executive Summary Reports Understandable to Policy Makers and Educators , 1996 .

[15]  Randy Elliot Bennett,et al.  Evaluating an Automatically Scorable, Open-Ended Response Type for Measuring Mathematical Reasoning in Computer-Adaptive Tests. , 1997 .

[16]  Howard Wainer,et al.  Improving Tabular Displays, With NAEP Tables as Examples and Inspirations , 1997 .

[17]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[18]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[19]  Irene Kostin,et al.  DEVELOPMENT, SELECTION AND VALIDATION OF A SET OF COGNITIVE AND LINGUISTIC ATTRIBUTES FOR THE SAT I VERBAL: SENTENCE COMPLETION SECTION1 , 1998 .

[20]  Randy Elliot Bennett,et al.  Validity and Automad Scoring: It's Not Only the Scoring , 1998 .

[21]  Susan E. Embretson,et al.  Generating items during testing: Psychometric issues and models , 1999 .

[22]  James W. Pellegrino,et al.  Chapter 9 : Addressing the “Two Disciplines” Problem: Linking Theories of Cognition and Learning With Assessment and Instructional Practice , 1999 .

[23]  H. Wainer,et al.  Alternative Displays for Communicating NAEP Results: A Redesign and Validity Study , 1999 .

[24]  Russell G. Almond,et al.  Introduction to the Biomass Project: An Illustration of Evidence-Centered Assessment Design and Delivery Capability. CSE Report. , 2003 .

[25]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[26]  Ronald K. Hambleton,et al.  Student Test Score Reports and Interpretive Guides: Review of Current Practices and Suggestions for Future Research , 2004 .

[27]  S. Sireci,et al.  Evaluating the Consistency of Test Content Across Two Successive Administrations of a State-Mandated Science Assessment , 2004 .

[28]  Barbara D. Yunker,et al.  The Table of Specifications: Insuring Accountability in Teacher Made Tests. , 2004 .

[29]  Joanna S. Gorin.  Manipulating Processing Difficulty of Reading Comprehension Questions: The Feasibility of Verbal Item Generation , 2005 .

[30]  Richard J. Stiggins,et al.  The Unfulfilled Promise of Classroom Assessment , 2005 .

[31]  Randy Elliot Bennett.  Using New Technology to Improve Assessment , 2005 .

[32]  S. Sireci,et al.  Validity Issues in Computer-Based Testing , 2005 .

[33]  Steven M. Downing,et al.  Handbook of test development , 2006 .