Putting ECD into Practice: The Interplay of Theory and Data in Evidence Models within a Digital Learning Environment

In this paper we describe the development and refinement of evidence rules and measurement models within the evidence model of the evidence-centered design (ECD) framework in the context of the Packet Tracer digital learning environment of the Cisco Networking Academy. Using Packet Tracer learners design, configure, and troubleshoot computer networks within an interactive interface. This leads to product data, which result from the students' final submitted network configurations, and process data, which are log file entries detailing how they got to the final configurations. We discuss how an iterative cycle of empirical analyses and discussions with subject-matter experts is essential for identifying and accumulating evidence about skill profiles of learners and their development. We present results from descriptive, exploratory, and confirmatory diagnostic modeling analyses for both data types, which required bringing to bear a diversity of tools from multivariate statistics, modern psychometrics, and educational data mining. We close the paper with a discussion of the implications of this work for evidence-based argumentation guided by ECD principles within digital learning environments more generally.

[1]  Robert J. Mislevy,et al.  Evidence-Centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments. , 2010 .

[2]  Gregory K. W. K. Chung,et al.  Identifying Key Features of Student Performance in Educational Video Games and Simulations through Cluster Analysis , 2012, EDM 2012.

[3]  Wendy M. Yen,et al.  Effects of Local Item Dependence on the Fit and Equating Performance of the Three-Parameter Logistic Model , 1984 .

[4]  Robert J. Mislevy,et al.  26 Bayesian Psychometric Modeling From An Evidence-Centered Design Perspective , 2006 .

[5]  Mykola Pechenizkiy,et al.  Handbook of Educational Data Mining , 2010 .

[6]  T. Kline.  Psychological Testing: A Practical Approach to Design and Evaluation , 2005 .

[7]  Robert J. Mislevy,et al.  Specifying and Refining a Measurement Model for a Computer-Based Interactive Assessment , 2004 .

[8]  Roy Levy,et al.  Data-model fit assessment for Bayesian networks for simulation-based assessments , 2011 .

[9]  David M. Williamson,et al.  A Framework for Evaluation and Use of Automated Scoring , 2012 .

[10]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[11]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[12]  J. Templin,et al.  Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art , 2008 .

[13]  P. Green,et al.  Analyzing multivariate data , 1978 .

[14]  G. Box.  Science and Statistics , 1976 .

[15]  Andrew Gelman,et al.  MCMC in Educational Research , 2011 .

[16]  Tena I. Katsaounis,et al.  Analyzing Multivariate Data , 2004, Technometrics.

[17]  Roy Levy,et al.  A generalized dimensionality discrepancy measure for dimensionality assessment in multidimensional item response theory. , 2011, The British journal of mathematical and statistical psychology.

[18]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[19]  Jonathan Templin,et al.  Diagnostic Measurement: Theory, Methods, and Applications , 2010 .

[20]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[21]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[22]  R. Levy.  The Rise of Markov Chain Monte Carlo Estimation for Psychometric Modeling , 2009 .

[23]  Robert J. Mislevy,et al.  Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining , 2012, EDM 2012.

[24]  Jacqueline P. Leighton.  Avoiding Misconception, Misuse, and Missed Opportunities: The Collection of Verbal Reports in Educational Achievement Testing , 2005 .

[25]  K. Petrides.  Introduction to Psychometric Theory , 2011 .

[26]  Russell G. Almond,et al.  On the Structure of Educational Assessments, CSE Technical Report. , 2003 .

[27]  Ryan S. Baker,et al.  The State of Educational Data Mining in 2009: A Review and Future Visions. , 2009, EDM 2009.

[28]  John Scott,et al.  The SAGE Handbook of Social Network Analysis , 2011 .

[29]  Robert J. Mislevy,et al.  Design Patterns for Learning and Assessment: Facilitating the Introduction of a Complex Simulation-Based Learning Environment into a Community of Instructors , 2010 .

[30]  Robert J. Mislevy,et al.  Evidentiary foundations of mixture item response theory models , 2006 .

[31]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[32]  Deniz Senturk-Doganaksoy,et al.  Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach , 2006, Technometrics.

[33]  Janice D. Gobert,et al.  Leveraging Educational Data Mining for Real-time Performance Assessment of Scientific Inquiry Skills within Microworlds , 2012, EDM 2012.

[34]  Wil M. P. van der Aalst,et al.  Process Mining - Discovery, Conformance and Enhancement of Business Processes , 2011 .

[35]  André A. Rupp,et al.  The Impact of Model Misspecification on Parameter Estimation and Item‐Fit Assessment in Log‐Linear Diagnostic Classification Models , 2012 .

[36]  Robert J. Mislevy,et al.  Psychometric and Evidentiary Approaches to Simulation Assessment in Packet Tracer Software , 2009, 2009 Fifth International Conference on Networking and Services.

[37]  I. Jolliffe.  Principal Component Analysis , 2002 .

[38]  Roy Levy.  Posterior predictive model checking for cognitively-based psychometric models , 2005 .

[39]  Russell G. Almond,et al.  Enhancing the Design and Delivery of Assessment Systems: A Four-Process Architecture , 2002 .

[40]  K. Tatsuoka.  Cognitive Assessment: An Introduction to the Rule Space Method , 2009 .

[41]  George A. Marcoulides,et al.  Introduction to Psychometric Theory , 2010 .

[42]  R. Maruyama,et al.  On Test Scoring , 1927 .

[43]  Scott M. Lynch,et al.  Introduction to Applied Bayesian Statistics and Estimation for Social Scientists , 2007 .

[44]  Filiz Garip,et al.  Model Comparison and Simulation for Hierarchical Models: Analyzing Rural-Urban Migration in Thailand , 2011 .

[45]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[46]  Robert J. Mislevy,et al.  Automated scoring of complex tasks in computer-based testing , 2006 .

[47]  Ronen Feldman,et al.  Book Reviews: The Text Mining Handbook: Advanced Approaches to Analyzing Unstructured Data by Ronen Feldman and James Sanger , 2008, CL.