Evidence-centered design for simulation-based assessment.

Simulations provide opportunities for people to learn and to develop skills for situations that are expensive, time-consuming, or dangerous. Careful design can support their learning by tailoring the features of situations to their levels of skill, allowing repeated attempts, and providing timely feedback. The same environments provide opportunities for assessing people's capabilities to act in these situations. This article describes an assessment design framework that can help projects develop effective simulation-based assessments. It reviews the rationale and terminology of the "evidence-centered" assessment design framework, discusses how it aligns with the principles of simulation design, and illustrates ideas with examples from engineering and medicine. Advice is offered for designing a new simulation-based assessment and for adapting an existing simulation system for assessment purposes.

[1]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[2]  S. Toulmin.  The uses of argument , 1960 .

[3]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[4]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[5]  L. Vygotsky.  Mind in Society: The Development of Higher Psychological Processes: Harvard University Press , 1978 .

[6]  Ruth Maciuin,et al.  2 The Experience of the National Board of Medical Examiners , 1983 .

[7]  R. Macklin,et al.  The experience of the National Board of Medical Examiners. , 1983, The Hastings Center report.

[8]  M. Chi,et al.  The Nature of Expertise , 1988 .

[9]  T. Salthouse.  Expertise as the circumvention of human processing limitations. , 1991 .

[10]  P. Chandler,et al.  Cognitive Load Theory and the Format of Instruction , 1991 .

[11]  Carol L. Smith,et al.  Conceptually enhanced simulations: A computer tool for science teaching , 1993 .

[12]  Valerie J. Shute,et al.  Intelligent Tutoring Systems: Past, Present, and Future. , 1994 .

[13]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[14]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[15]  A. Roli.  Artificial Neural Networks , 2012, Lecture Notes in Computer Science.

[16]  Robert J. Mislevy,et al.  The Role of Probability-Based Inference in an Intelligent Tutoring System. , 1995 .

[17]  Linda S. Steinberg,et al.  INTELLIGENT TUTORING AND ASSESSMENT BUILT ON AN UNDERSTANDING OF A TECHNICAL PROBLEM‐SOLVING TASK , 1996 .

[18]  J. Roschelle.  Designing for cognitive communication: epistemic fidelity or mediating collaborative inquiry? , 1997, Computers, Communication and Mental Models.

[19]  J. Wertsch.  Mind as action , 1998 .

[20]  J. Greeno.  THE SITUATIVITY OF KNOWING, LEARNING, AND RESEARCH , 1998 .

[21]  Russell G. Almond,et al.  A cognitive task analysis with implications for designing simulation-based performance assessment☆ , 1999 .

[22]  L A Johnson,et al.  A cognitive task analysis for dental hygiene. , 2000, Journal of dental education.

[23]  Russell G. Almond,et al.  Enhancing the Design and Delivery of Assessment Systems: A Four-Process Architecture , 2002 .

[24]  R. Almond,et al.  Making Sense of Data From Complex Assessments , 2002 .

[25]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[26]  Robert J. Mislevy,et al.  Substance and structure in assessment arguments , 2003 .

[27]  B. Yegnanarayana,et al.  Artificial Neural Networks , 2004 .

[28]  D B Swanson,et al.  Simulations in the United States Medical Licensing Examination™ (USMLE™) , 2004, Quality and Safety in Health Care.

[29]  Mari Pearlman,et al.  Chapter 3 The design architecture of NBPTS certification assessments , 2004 .

[30]  Robert J. Mislevy,et al.  Design Rationale for a Complex Performance Assessment , 2004 .

[31]  David H. Jonassen,et al.  Learning to Troubleshoot: A New Theory-Based Design Architecture , 2006 .

[32]  Bernard R. Gifford,et al.  Computer-Based Assessment in E-Learning: A Framework for Constructing "Intermediate Constraint" Questions and Tasks for Technology Platforms , 2006 .

[33]  Robert J. Mislevy,et al.  26 Bayesian Psychometric Modeling From An Evidence-Centered Design Perspective , 2006 .

[34]  Robert J. Mislevy,et al.  On the roles of external knowledge representations in assessment design , 2007 .

[35]  Kathleen R Rosen,et al.  The history of medical simulation. , 2008, Journal of critical care.

[36]  Eva L. Baker,et al.  Templates and Objects in Authoring Problem-Solving Assessments. CRESST Report 735. , 2008 .

[37]  Yasmin B. Kafai,et al.  How computer games help children learn , 2008 .

[38]  David F. Feldon,et al.  Cognitive task analysis , 2009 .

[39]  Richard M. Luecht.  Adaptive Computer-Based Tasks Under an Assessment Engineering Paradigm , 2009 .

[40]  G. F. Dillon,et al.  Computer-Delivered Patient Simulations in the United States Medical Licensing Examination (USMLE) , 2009, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[41]  Robert J. Mislevy,et al.  Psychometric and Evidentiary Approaches to Simulation Assessment in Packet Tracer Software , 2009, 2009 Fifth International Conference on Networking and Services.

[42]  Theresa J. B. Kline,et al.  Measurement and assessment issues in performance appraisal. , 2009 .

[43]  Dennis C. Frezzo,et al.  Using activity theory to understand the role of a simulation-based interactive learning environment in a computer networking course , 2009 .

[44]  Robert J. Mislevy,et al.  Activity Theory and Assessment Theory in the Design and Understanding of the Packet Tracer Ecosystem , 2009 .

[45]  Robert J. Mislevy,et al.  Evidence-Centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments. , 2010 .

[46]  Richard Wainess,et al.  A Conceptual Framework for Assessing Performance in Games and Simulations. CRESST Report 771. , 2010 .

[47]  Linda S. Steinberg,et al.  The Promises and Challenges of Implementing Evidence-Centered Design in Large-Scale Assessment , 2010 .

[48]  Kristen E. DiCerbo,et al.  An evidence centered design for learning and assessment in the digital world , 2010 .

[49]  David Williamson Shaffer,et al.  Mentor modeling: the internalization of modeled professional thinking in an epistemic game , 2010, J. Comput. Assist. Learn..

[50]  Maureen Ewing,et al.  Representing Targets of Measurement Within Evidence-Centered Design , 2010 .

[51]  Victor Chi,et al.  An Educational Simulator for Training CPR Skills , 2011 .

[52]  R. Mislevy,et al.  Evidence-Centered Design in Assessment Development , 2011 .

[53]  V. Shute.  SteAlth ASSeSSment in computer-BASed GAmeS to Support leArninG , 2011 .

[54]  Robert J. Mislevy,et al.  Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining , 2012, EDM 2012.

[55]  Kristen E. DiCerbo,et al.  Intended and Unintended Deceptions in the Use of Simulations , 2012 .

[56]  Y. Engeström,et al.  Learning by expanding: An activity-theoretical approach to developmental research , 2014 .