From Cognitive-Domain Theory to Assessment Practice

This article exemplifies how assessment design might be grounded in theory, thereby helping to strengthen validity claims. Spanning work across multiple related projects, the article first briefly summarizes an assessment system model for the elementary and secondary levels. Next the article describes how cognitive-domain theory and principles are used in the design of a scenario-based summative assessment for argumentation in the English language arts. Finally, results from several psychometric approaches are used to evaluate propositions suggested by the domain theory, including ones related to the use of topical scenarios and learning progressions in assessment design. Although results generally supported these propositions, the work described represents only a small step in a long-term, iterative process of theory development, assessment design, and empirical tryout, which should, in principle, lead to more valid assessments that better inform teaching and learning.

[1]  P. David Pearson.  Research Foundations of the Common Core State Standards in English Language Arts , 2013 .

[2]  J. A. Blair,et al.  Fundamentals of Argumentation Theory : A Handbook of Historical Backgrounds and Contemporary Developments , 1997 .

[3]  John Sabatini,et al.  Key Practices in the English Language Arts (ELA): Linking Learning Theory, Assessment, and Instruction , 2015 .

[4]  P. Deane.  WRITING ASSESSMENT AND COGNITION , 2011 .

[5]  D. Kuhn.  THE SKILLS OF ARGUMENT , 2008, Education for Thinking.

[6]  R. Ferretti,et al.  The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normally achieving peers. , 2000 .

[7]  Kurt VanLehn,et al.  Development of Professional Expertise: Acquiring Conceptual Expertise from Modeling: The Case of Elementary Physics , 2009 .

[8]  Deanna Kuhn,et al.  The Development of Argumentive Discourse Skill , 2001 .

[9]  Maria Bertling,et al.  A CBAL™ SCIENCE MODEL OF COGNITION: DEVELOPING A COMPETENCY MODEL AND LEARNING PROGRESSIONS TO SUPPORT ASSESSMENT DEVELOPMENT , 2013 .

[10]  P. Deane,et al.  Building and Sharing Knowledge Key Practice: What Do You Know, What Don't You Know, What Did You Learn? , 2015 .

[11]  G. Cizek,et al.  Setting performance standards : foundations, methods, and innovations , 2012 .

[12]  S. Graham,et al.  Metacognition and Children’s Writing , 2009 .

[13]  Mark Wilson,et al.  From Principles to Practice: An Embedded Assessment System , 2000 .

[14]  George Kamberelis,et al.  Cultural Artifacts as Scaffolds for Genre Development , 1999 .

[15]  Randy Elliot Bennett,et al.  CBAL: Results from Piloting Innovative K-12 Assessments. Research Report. ETS RR-11-23. , 2011 .

[16]  R. Abbott,et al.  Structural equation modeling of relationships among developmental skills and writing skills in primary- and intermediate-grade writers. , 1993 .

[17]  Mark R. Wilson.  Responding To A Challenge That Learning Progressions Pose To Measurement Practice , 2012 .

[18]  Jerry Andriessen,et al.  From planning to translating: The specificity of argumentative writing. , 1999 .

[19]  Deanna Kuhn,et al.  Dialogic Argumentation as a Vehicle for Developing Young Adolescents’ Thinking , 2011, Psychological science.

[20]  D. Kuhn,et al.  Beyond control of variables: What needs to develop to achieve skilled scientific thinking? , 2008 .

[21]  R. Bennett.  Formative assessment: a critical review , 2011 .

[22]  Sibel Erduran,et al.  Argumentation in Science Education: Perspectives from Classroom-Based Research , 2007 .

[23]  J. Swales,et al.  6. Genre analysis , 2014 .

[24]  Ruth E. Knudson.  The Development of Written Argumentation: An Analysis and Comparison of Argumentative Writing at Four Grade Levels. , 1992 .

[25]  C. Miller,et al.  A theory of argumentative understanding: Relationships among position preference, judgments of goodness, memory and reasoning , 1993 .

[26]  Robert J. Crutcher,et al.  The role of deliberate practice in the acquisition of expert performance. , 1993 .

[27]  Aristotle,et al.  The Rhetoric of Aristotle : an expanded translation with supplementary examples for students of composition and public speaking , 1932 .

[28]  L. Cronbach,et al.  THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY† , 1963 .

[29]  Peggy A. Ertmer,et al.  The expert learner: Strategic, self-regulated, and reflective , 1996 .

[30]  A. Graesser.  An introduction to strategic reading comprehension. , 2007 .

[31]  E. A. Graf.  DEFINING MATHEMATICS COMPETENCY IN THE SERVICE OF COGNITIVELY BASED ASSESSMENT FOR GRADES 6 THROUGH 8 , 2009 .

[32]  John Sabatini,et al.  PRELIMINARY READING LITERACY ASSESSMENT FRAMEWORK: FOUNDATION AND RATIONALE FOR ASSESSMENT AND SYSTEM DESIGN , 2013 .

[33]  J. Delia,et al.  The Development of Functional Persuasive Skills in Childhood and Early Adolescence. , 1976 .

[34]  Ulla Connor,et al.  Contrastive Rhetoric: Cross-Cultural Aspects of Second Language Writing , 1998 .

[35]  Danielle S. McNamara,et al.  The 4-pronged comprehension strategy framework. , 2007 .

[36]  Deanna Kuhn,et al.  The development of argument skills. , 2003, Child development.

[37]  J. B. Wyman,et al.  What is reading ability , 1921 .

[38]  D. McCutchen.  A capacity theory of writing: Working memory in composition , 1996 .

[39]  Robert L. Linn,et al.  Performance-Based Assessment: Implications of Task Specificity , 2005 .

[40]  Ets Rr,et al.  CBAL: Results From Piloting Innovative K-12 Assessments , 2011 .

[41]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[42]  Karin Baier,et al.  The Uses Of Argument , 2016 .

[43]  Susan L. Kline,et al.  The development of persuasive communication strategies in kindergarteners through twelfth‐graders , 1979 .

[44]  Placy Rj,et al.  Setting performance standards. , 1976 .

[45]  D. Kuhn,et al.  Effects of Dyadic Interaction on Argumentive Reasoning , 1997 .

[46]  C. Garvey,et al.  Children's use of verbal strategies in resolving conflicts1 , 1981 .

[47]  Susan R. Goldman,et al.  A Framework for Conceptualizing and Evaluating the Validity of Instructionally Relevant Assessments , 2016 .

[48]  T. Shanahan,et al.  Analysis of Expert Readers in Three Disciplines , 2011 .

[49]  Edward H. Haertel,et al.  4 Reliability Coefficients and Generalizability Theory , 2006 .

[50]  Paul Deane,et al.  A case study in principled assessment design: Designing assessments to measure and support the development of argumentative reading and writing skills , 2014 .

[51]  Paul Deane,et al.  The Key Practice, Discuss and Debate Ideas: Conceptual Framework, Literature Review, and Provisional Learning Progressions for Argumentation , 2015 .

[52]  K. Ann Renninger,et al.  The Centrality of a Distancing Model for the Development of Representational Competence , 2013 .

[53]  Deanna Kuhn,et al.  Coordinating own and other perspectives in argument , 2007 .

[54]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[55]  Selma Leitão,et al.  Evaluating and Selecting Counterarguments , 2003 .

[56]  Deanna Kuhn,et al.  Why Argue? Developing Understanding of the Purposes and Values of Argumentive Discourse , 2010 .

[57]  Laura S. Hamilton,et al.  Testing for Accountability in K-12 , 2006 .

[58]  Jesús Alcolea Banegas.  Argumentation in Mathematics , 2013 .

[59]  Jesse R. Sparks,et al.  Cognitively Based Assessment of Research and Inquiry Skills: Defining a Key Practice in the English Language Arts , 2015 .

[60]  M. Scardamalia,et al.  The psychology of written composition , 1987 .

[61]  Susan R. Goldman,et al.  Adolescent Literacy: Learning and Understanding Content , 2013, The Future of children.

[62]  S. Graham,et al.  Metacognition and strategies instruction in writing , 2010 .

[63]  Douglas J. Hacker,et al.  Handbook of Metacognition in Education , 2009 .

[64]  Scott G. Paris,et al.  Children's reading strategies, metacognition, and motivation , 1986 .

[65]  E. A. Graf,et al.  Empirical recovery of argumentation learning progressions in scenario-based assessments of English language arts , 2014 .

[66]  Catherine E. Overson,et al.  Applying science of learning in education: Infusing psychological science into the curriculum , 2014 .

[67]  K. Bartsch,et al.  Children's Attention to Others' Beliefs during Persuasion: Improvised and Selected Arguments to Puppets and People , 2011 .

[68]  John M. Swales,et al.  Genre Analysis: English in Academic and Research Settings , 1993 .

[69]  Mark Wilson,et al.  Some Links between Large-Scale and Classroom Assessments: The Case of the BEAR Assessment System , 2004, Teachers College Record: The Voice of Scholarship in Education.

[70]  R. Bennett,et al.  Transforming K–12 Assessment: Integrating Accountability Testing, Formative Assessment and Professional Support , 2009 .

[71]  K. Bartsch,et al.  Children's use of mental state information in selecting persuasive arguments. , 2000, Developmental psychology.

[72]  Edith Aurora Graf,et al.  Using Argumentation Learning Progressions to Support Teaching and Assessments of English Language Arts , 2013 .

[73]  Sandip Sinharay,et al.  Assessing Item Fit for Unidimensional Item Response Theory Models Using Residuals from Estimated Item Response Functions , 2013, Psychometrika.

[74]  David N. Perkins,et al.  Everyday reasoning and the roots of intelligence , 1991 .

[75]  Meirav Arieli-Attali,et al.  Expanding the "CBAL"™ Mathematics Assessments to Elementary Grades: The Development of a Competency Model and a Rational Number Learning Progression. Research Report. ETS RR-14-08. , 2014 .

[76]  D. Walton.  Argumentation Schemes for Presumptive Reasoning , 1995 .

[77]  Katherine L. McNeill,et al.  A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts , 2010 .

[78]  B. Zimmerman.  Becoming a Self-Regulated Learner: An Overview , 2002 .

[79]  Deborah McCutchen,et al.  From novice to expert: Implications of language skills and writing-relevant knowledge for memory during the development of writing skill , 2011 .

[80]  Tenaha O'Reilly,et al.  COGNITIVELY BASED ASSESSMENT OF, FOR, AND AS LEARNING: A FRAMEWORK FOR ASSESSING READING COMPETENCY , 2009 .

[81]  Donald A. Rock,et al.  Assessing Writing Skill , 1988 .

[82]  R. T. Kellogg,et al.  Training writing skills: A cognitive developmental perspective , 2008 .

[83]  Mark Wilson,et al.  Measuring Progressions: Assessment Structures Underlying a Learning Progression , 2009 .

[84]  P. Green.  Contrastive Rhetoric: Cross-cultural Aspects of Second-language Writing , 1997 .

[85]  Richard C. Anderson,et al.  Influence of Oral Discussion on Written Argument , 2001 .

[86]  Wolfgang Ziegler,et al.  An Introduction To Reasoning , 2016 .

[87]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[88]  William E. Lewis,et al.  Do Goals Affect the Structure of Students' Argumentative Writing Strategies? , 2009 .

[89]  Shelby J. Haberman,et al.  A GENERAL PROGRAM FOR ITEM‐RESPONSE ANALYSIS THAT EMPLOYS THE STABILIZED NEWTON–RAPHSON ALGORITHM , 2013 .

[90]  S. E. Newman,et al.  Pushing Toulmin Too Far: Learning From an Argument Representation Scheme , 1998 .

[91]  John Sabatini,et al.  Efficiency in Word Reading of Adults: Ability Group Comparisons , 2002 .

[92]  E. Wenger.  Communities of Practice and Social Learning Systems , 2000 .

[93]  John Sabatini,et al.  READING FOR UNDERSTANDING: HOW PERFORMANCE MODERATORS AND SCENARIOS IMPACT ASSESSMENT DESIGN , 2013 .

[94]  F. H. van Eemeren,et al.  Argumentation: Analysis, Evaluation, Presentation , 2002 .

[95]  Ekaterina Midgette,et al.  The effects of content and audience awareness goals for revision on the persuasive essays of fifth- and eighth-grade students , 2008 .

[96]  Douglas Walton,et al.  Plausible argument in everyday conversation , 1992 .

[97]  Randy Elliot Bennett,et al.  Cognitively Based Assessment of, for, and as Learning (CBAL): A Preliminary Theory of Action for Summative and Formative Assessment , 2010 .

[98]  J. Hayes,et al.  A Cognitive Process Theory of Writing , 1981, College Composition & Communication.

[99]  Caroline Golder,et al.  Argumentative text writing: Developmental trends , 1994 .

[100]  George E. Newell,et al.  Teaching and Learning Argumentative Reading and Writing: A Review of Research , 2011, Reading Research Quarterly.

[101]  Micheline B. Chalhoub-Deville.  Validity theory: Reform policies, accountability testing, and consequences , 2016 .

[102]  Janice F. Almasi,et al.  The Impact of a Strategy-Based Intervention on the Comprehension and Strategy Use of Struggling Adolescent Readers. , 2010 .

[103]  Brett van de Sande,et al.  Acquiring Conceptual Expertise from Modeling : Th e Case of Elementary Physics , 2009 .

[104]  C. Kardash,et al.  The Effects of Goal Instructions and Text on the Generation of Counterarguments During Writing. , 2005 .