How To Create Complex Measurement Models: A Case Study of Principled Assessment Design.

In computer-based simulations, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a simulation system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This study focused on the grounding for a simulation-based assessment of design and troubleshooting in the domain of computer networks. The application was designed as a prototype for assessing these skills in an instructional program, as interim practice tests and as chapter or end-of-course assessments. An evidence-centered assessment design framework was used to guide the work. An important part of this work is a cognitive task analysis, designed to tap the knowledge network engineers and students use when they design and troubleshoot networks and elicit behaviors that manifest this knowledge. There were 24 participants at 3 levels of ability in this early phase of the research. After summarizing the results of the analysis, the paper discusses implications for designing psychometric models, automated scoring algorithms, and task frameworks, and for the capabilities required for the simulation environment. itself. (Contains 6 figures and 24 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document. How To Create Complex Measurement Models: A Case Study of Principled Assessment Design* Malcolm Bauer, David M. Williamson, Linda S. Steinberg Educational Testing Service Robert J. Mislevy University of Maryland

[1]  Stephen G. Clyman,et al.  Scoring a Performance-Based Assessment by Modeling the Judgments of Experts , 1995 .

[2]  R. Mislevy Evidence and inference in educational assessment , 1994 .

[3]  Patrick Tapsfield,et al.  The British Army Recruit Battery Goes Operational: From Theory to Practice in Computer‐Based Testing Using Item‐Generation Techniques , 1995 .

[4]  R. Glaser,et al.  Knowledge-based cognition and performance assessment in the science classroom , 1996 .

[5]  D. Schum The Evidential Foundations of Probabilistic Reasoning , 1994 .

[6]  K. Ericsson,et al.  Prospects and limits of the empirical study of expertise: an introduction , 1991 .

[7]  Russell G. Almond,et al.  A cognitive task analysis with implications for designing simulation-based performance assessment☆ , 1999 .

[8]  Russell G. Almond,et al.  Bayes Nets in Educational Assessment: Where Do the Numbers Come from? CSE Technical Report. , 2000 .

[9]  D. Schum,et al.  A Probabilistic Analysis of the Sacco and Vanzetti Evidence , 1996 .

[10]  R. Macklin,et al.  The experience of the National Board of Medical Examiners. , 1983, The Hastings Center report.

[11]  Linda S. Steinberg,et al.  Cognitive task analysis and interface design in a technical troubleshooting domain , 1993, Knowl. Based Syst..

[12]  Robert J. Mislevy,et al.  Diagnostic Assessment of Troubleshooting Skill in an Intelligent Tutoring System. , 1994 .

[13]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1999 .

[14]  Robert J. Mislevy,et al.  The Role of Probability-Based Inference in an Intelligent Tutoring System. , 1995 .

[15]  Linda S. Steinberg,et al.  INTELLIGENT TUTORING AND ASSESSMENT BUILT ON AN UNDERSTANDING OF A TECHNICAL PROBLEM‐SOLVING TASK , 1996 .

[16]  S. Messick The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .