You Can't Fatten A Hog by Weighing It - Or Can You? Evaluating an Assessment for Learning System Called ACED

The purpose of the study described in this paper was to evaluate the efficacy of an assessment for learning system named ACED (Adaptive Content with Evidence-based Diagnosis). We used an evidencecentered design approach to create an adaptive, diagnostic assessment system which includes five main models: competency, evidence, task, presentation, and assembly. We also included instructional support in the form of elaborated feedback. The key issue we examined was whether the inclusion of the feedback into the system (a) impairs the quality of the assessment (relative to validity, reliability, and efficiency), and (b) does, in fact, enhance student learning. Results from a controlled evaluation testing 268 high-school students showed that the quality of the assessment was unimpaired by the provision of feedback. Moreover, students using the ACED system showed significantly greater learning of the content compared with a control group. These findings suggest that assessments in other settings (e.g. state-mandated tests) might be augmented to support student learning with instructional feedback without jeopardizing the primary purpose of the assessment.

[1]  A. Jameson.  Adaptive interfaces and agents , 2002 .

[2]  Jeffrey A Douglas,et al.  Test Construction for Cognitive Diagnosis , 2005 .

[3]  Valerie J. Shute,et al.  TENSIONS, TRENDS, TOOLS, AND TECHNOLOGIES: TIME FOR AN EDUCATIONAL SEA CHANGE , 2006 .

[4]  Russell G. Almond,et al.  "I Can Name that Bayesian Network in Two Matrixes!" , 2007, BMA.

[5]  Ellen B. Mandinach,et al.  INTEGRATING ASSESSMENT AND INSTRUCTION: A RESEARCH AND DEVELOPMENT AGENDA , 1991 .

[6]  Russell G. Almond,et al.  Graphical Explanation in Belief Networks , 1997 .

[7]  Ding Shuliang,et al.  Test Construction for Cognitive Diagnosis , 2010, 2010 Asia-Pacific Conference on Wearable Computing Systems.

[8]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[9]  Kristin M. Bass,et al.  Developing Assessments to Inform Teaching and Learning: (646292011-001) , 2004 .

[10]  Kikumi K. Tatsuoka,et al.  A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach , 1985 .

[11]  R. Moreno.  Decreasing Cognitive Load for Novice Students: Effects of Explanatory versus Corrective Feedback in Discovery-Based Multimedia , 2004 .

[12]  Chen-Lin C. Kulik,et al.  The Instructional Effect of Feedback in Test-Like Events , 1991 .

[13]  José-Luis Pérez-de-la-Cruz,et al.  A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation , 2002, User Modeling and User-Adapted Interaction.

[14]  H. Penny Nii,et al.  The Handbook of Artificial Intelligence , 1982 .

[15]  Russell G. Almond.  Graphical belief modeling , 1995 .

[16]  Russell G. Almond,et al.  Modeling Diagnostic Assessments with Bayesian Networks , 2007 .

[17]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[18]  L. Vygotsky.  Play and Its Role in the Mental Development of the Child , 1967 .

[19]  Susanne Narciss,et al.  How to design informative tutoring feedback for multi-media learning , 2004 .

[20]  Susan M. Brookhart,et al.  Successful Students' Formative and Summative Uses of Assessment Information , 2001 .

[21]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[22]  Kiley Walsh Symonds,et al.  After the Test: Closing the Achievement Gaps With Data , 2004 .

[23]  Eric G. Hansen,et al.  DESIGNING ADAPTIVE, DIAGNOSTIC MATH ASSESSMENTS FOR INDIVIDUALS WITH AND WITHOUT VISUAL DISABILITIES , 2006 .

[24]  C. Roads,et al.  The Handbook of Artificial Intelligence, Volume 1 , 1982 .

[25]  P. Black,et al.  Teachers developing assessment for learning: impact on student achievement , 2004 .

[26]  Mary Peat,et al.  Supporting student learning: the use of computer-based formative assessment modules , 2002, Br. J. Educ. Technol..

[27]  Avron Barr,et al.  The Handbook of Artificial Intelligence, Volume 1 , 1982 .

[28]  P. Black,et al.  Inside the Black Box: Raising Standards through Classroom Assessment , 2010 .

[29]  Ricardo Conejo,et al.  SIETTE: A Web-Based Tool for Adaptive Testing , 2004, Int. J. Artif. Intell. Educ..

[30]  Kenneth R. Koedinger,et al.  Proceedings of the 2007 conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work , 2007 .

[31]  John R. Anderson,et al.  Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes , 2001, CHI.

[32]  Russell G. Almond,et al.  On the Structure of Educational Assessments, CSE Technical Report. , 2003 .

[33]  I.,et al.  Weight of Evidence : A Brief Survey , 2006 .

[34]  Willem J. van der Linden,et al.  Bayesian item selection criteria for adaptive testing , 1998 .

[35]  D. Watson,et al.  Constructing validity: Basic issues in objective scale development , 1995 .

[36]  J ShuteValerie,et al.  You Can't Fatten A Hog by Weighing It - Or Can You? Evaluating an Assessment for Learning System Called ACED , 2008 .

[37]  Wynne Harlen,et al.  Teachers' summative practices and assessment for learning – tensions and synergies , 2005 .

[38]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[39]  R. Stiggins.  Assessment Crisis: The Absence of Assessment for Learning , 2002 .

[40]  I. Good,et al.  The Diagnostic Process with Special Reference to Errors , 1971, Methods of Information in Medicine.

[41]  Russell G. Almond,et al.  AN ASSESSMENT FOR LEARNING SYSTEM CALLED ACED: DESIGNING FOR LEARNING EFFECTIVENESS AND ACCESSIBILITY , 2007 .

[42]  Roger Azevedo,et al.  A Meta-Analysis of the Effects of Feedback in Computer-Based Instruction , 1995 .

[43]  Robert J. Mislevy,et al.  Accessibility of Computer-Based Testing for Individuals with Disablities and English Language Learners within a Validity Framework , 2008 .

[44]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1998 .

[45]  Dylan Wiliam,et al.  Classroom Assessment: Minute by Minute, Day by Day In classrooms that use assessment to support learning, teachers continually adapt instruction to meet student needs. , 2005 .

[46]  Janet Coffey,et al.  Everyday Assessment in the Science Classroom. , 2003 .

[47]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[48]  Valerie J. Shute,et al.  English ABLE , 2007, AIED.

[49]  V. Shute.  Focus on Formative Feedback , 2007 .

[50]  Russell G. Almond,et al.  AN ILLUSTRATION OF THE USE OF MARKOV DECISION PROCESSES TO REPRESENT STUDENT GROWTH (LEARNING) , 2007 .

[51]  V. Shute,et al.  Adaptive Technologies , 2007 .

[52]  Robert Glaser,et al.  Developing Assessments To Inform Teaching and Learning. CSE Report 628. , 2004 .

[53]  Robert J. Mislevy,et al.  Accessibility of Tests for Individuals with Disabilities within a Validity Framework. , 2005 .

[54]  Grant Wiggins,et al.  Educative Assessment: Designing Assessments to Inform and Improve Student Performance , 1998 .

[55]  M. Lepper,et al.  Intrinsic Motivation and instruction: conflicting Views on the Role of Motivational Processes in Computer-Based Education , 1985 .

[56]  Russell G. Almond,et al.  On Test Selection Strategies for Belief Networks , 1995, AISTATS.

[57]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[58]  Shirley Clarke.  Unlocking formative assessment : practical strategies for enhancing pupils' learning in the primary classroom , 2001 .

[59]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[60]  M. Taras,et al.  Using Assessment for Learning and Learning from Assessment , 2002 .

[61]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[62]  R. Owen,et al.  A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental Testing , 1975 .

[63]  Jay McTighe,et al.  Seven Practices for Effective Learning , 2005 .

[64]  Alan Sutton,et al.  Exploring the role of assessment tasks to promote formative assessment in Key Stage 3 Geography: evidence from twelve teachers , 2006 .