Measurement Challenges of Interactive Educational Assessment

This chapter discusses four measurement challenges of data science in educational assessments that are enabled by technology: (1) Dealing with change over time. (2) How a digital performance space’s relationships interact with learner actions, communications and products. (3) How layers of interpretation are formed from translations of atomistic data into meaningful larger units suitable for making inferences about what someone knows and can do. (4) How to represent the dynamics of interactions between and among learners who are being assessed by their interactions with each other as well as with digital resources and agents in digital performance spaces. Because of the movement from paper-based tests to online learning, and in order to make progress on these challenges, the authors advocate the restructuring of training of the next generation of researchers and psychometricians in technology-enabled assessments.

[1]  J. Michael Spector,et al.  Highly integrated model assessment technology and tools , 2010, CELDA 2008.

[2]  Stephen P. Borgatti,et al.  Network Theory , 2013 .

[3]  O. Sporns.  Networks of the Brain , 2010 .

[4]  S. Messick.  The Interplay of Evidence and Consequences in the Validation of Performance Assessments , 1994 .

[5]  Dirk Ifenthaler,et al.  Assessment for Game-Based Learning , 2012 .

[6]  Athanasios V. Vasilakos,et al.  Big data: From beginning to future , 2016, Int. J. Inf. Manag..

[7]  Mary Webb,et al.  Assessment as, for and of 21st Century learning using information technology : an overview , 2018 .

[8]  Michael J. Timms,et al.  Assessment of Student Learning in Science Simulations and Games , 2009 .

[9]  Birgit Kopainsky,et al.  Automated assessment of learners' understanding in complex dynamic systems , 2012 .

[10]  I. W. Molenaar,et al.  Rasch models: foundations, recent developments and applications , 1995 .

[11]  Jiawei Han,et al.  Frequent pattern mining: current status and future directions , 2007, Data Mining and Knowledge Discovery.

[12]  Michael A. Rodriguez,et al.  Clickstream Data Yields High-Resolution Maps of Science , 2009, PloS one.

[13]  Mary Webb,et al.  Data science in educational assessment , 2015, Education and Information Technologies.

[14]  Michael Quinn Patton,et al.  Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use , 2010 .

[15]  Richard J. Stiggins,et al.  Assessment Literacy for the 21st Century. , 1995 .

[16]  Jody Clarke-Midura,et al.  Some Psychometric and Design Implications of Game Based Learning Analytics , 2015 .

[17]  David Gibson,et al.  Game Changers for Transforming Learning Environments , 2012 .

[18]  Chris Dede,et al.  Assessment, Technology, and Change , 2010 .

[19]  Dirk Ifenthaler,et al.  Preparing the Next Generation of Education Researchers for Big Data in Higher Education , 2017 .

[20]  Ron Stevens,et al.  Design and performance frameworks for constructing problem-solving simulations. , 2003, Cell biology education.

[21]  R. Kass.  Nonlinear Regression Analysis and its Applications , 1990 .

[22]  Erik Duval,et al.  Learning Analytics Dashboard Applications , 2013 .

[23]  Dirk Ifenthaler,et al.  Scope of Graphical Indices in Educational Diagnostics , 2010 .

[24]  John D. Sterman,et al.  Learning in and about complex systems , 1994 .

[25]  Amy Soller,et al.  Probabilities and predictions: modeling the development of scientific problem-solving skills. , 2005, Cell biology education.

[26]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[27]  Helen Margetts,et al.  Addressing the policy challenges and opportunities of “Big data” , 2013 .

[28]  M. Segers,et al.  Effects of problem-based learning: a meta- analysis , 2003 .

[29]  Randy Elliot Bennett,et al.  Cognitively Based Assessment of, for, and as Learning (CBAL): A Preliminary Theory of Action for Summative and Formative Assessment , 2010 .

[30]  Dirk Ifenthaler,et al.  Comparison of Two Analysis Approaches for Measuring Externalized Mental Models , 2011, J. Educ. Technol. Soc..

[31]  P. Kline.  The New Psychometrics: Science, Psychology and Measurement , 1998 .

[32]  David H. Jonassen,et al.  Instructional design models for well-structured and III-structured problem-solving learning outcomes , 1997 .

[33]  Dirk Ifenthaler,et al.  Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives , 2012 .

[34]  Mary Webb,et al.  Challenges for information technology supporting educational assessment , 2013, J. Comput. Assist. Learn..

[35]  Dirk Ifenthaler.  AKOVIA: Automated Knowledge Visualization and Assessment , 2014, Technol. Knowl. Learn..

[36]  G. Wiggins.  Teaching to the (Authentic) Test. , 1989 .

[37]  Panagiotis Zervas,et al.  Supporting Reflective Lesson Planning Based on Inquiry Learning Analytics for Facilitating Students’ Problem Solving Competence Development: The Inspiring Science Education Tools , 2018 .

[38]  Yaneer Bar-Yam,et al.  Dynamics Of Complex Systems , 2019 .

[39]  Hedeker Donald,et al.  Longitudinal Data Analysis: Hedeker/Longitudinal , 2006 .

[40]  George Sugihara,et al.  Detecting Causality in Complex Ecosystems , 2012, Science.

[41]  Jodi L. Davenport,et al.  21st Century Dynamic Assessment , 2012 .

[42]  Mary Webb,et al.  Technology enhanced assessment in complex collaborative settings , 2015, Education and Information Technologies.

[43]  Donald Hedeker,et al.  Longitudinal Data Analysis , 2006 .

[44]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[45]  Paul Thagard,et al.  How Brains Make Mental Models , 2010 .

[46]  Elizabeth Bagley,et al.  Epistemic network analysis : a Prototype for 21 st Century assessment of Learning , 2009 .

[47]  Carlos Delgado Kloos,et al.  Key Action Extraction for Learning Analytics , 2012, EC-TEL.

[48]  Ulrik Brandes,et al.  Network Analysis: Methodological Foundations (Lecture Notes in Computer Science) , 2005 .

[49]  Marc Prensky,et al.  Digital game-based learning , 2000, CIE.

[50]  C. Teddlie,et al.  SAGE Handbook of Mixed Methods in Social & Behavioral Research , 2010 .

[51]  Jeremy Hodgen,et al.  Validity in teachers’ summative assessments , 2010 .

[52]  David Gibson,et al.  Game Changers for Teacher Education , 2011 .