Assessment With Microworlds using MicroDYN: Measurement Invariance and Latent Mean Comparisons

Computer-simulated microworlds have witnessed significant international interest over the last decades as assessment vehicles for complex mental skills. This interest strongly contrasts to what is currently known about measurement characteristics of microworlds. In this study measurement invariance and latent means of the MicroDYN measure, a computer-based assessment instrument containing an entire set of dynamic microworlds, were examined in four German subsamples of junior high school students in 8th–10th grade (n = 309), senior high school students in 11th–13th grade (n = 484), university students (n = 222), and blue-collar workers (n = 181). The findings support satisfactory measurement invariance of a two-dimensional structure of the MicroDYN measure with the dimensions knowledge acquisition and knowledge application across all samples, and yield meaningful comparisons between latent means with university students performing best. It is suggested to further explore measurement characteristics of comp...

[1]  Oscar Valiente,et al.  PISA 2009 Results: Students On Line Digital Technologies and Performance (Volume VI) , 2011 .

[2]  Samuel Greiff,et al.  Complex problem solving : More than reasoning ? , 2017 .

[3]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[4]  Annette Kluge,et al.  Performance Assessments With Microworlds and Their Difficulty , 2008 .

[5]  Sylvia Green,et al.  On-line Mathematics Assessment: The Impact of Mode on Performance and Question Answering Strategies , 2006 .

[6]  Markus Bühner,et al.  Working memory, visual–spatial-intelligence and their relationship to problem-solving , 2008 .

[7]  Samuel Greiff,et al.  Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity , 2013 .

[8]  Gordon W. Cheung,et al.  Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance , 2002 .

[9]  Samuel Greiff,et al.  The Genetics Lab: Acceptance and psychometric characteristics of a computer-based microworld assessing complex problem solving , 2012 .

[10]  Jan J. Elshout,et al.  Differential effects of instructional support on learning in simultation environments , 1994 .

[11]  J. Funke.  Dynamic systems as tools for analysing human judgement , 2001 .

[12]  Samuel Greiff,et al.  Complex Problem Solving in Educational Contexts—Something Beyond g: Concept, Assessment, Measurement Invariance, and Construct Validity , 2013 .

[13]  S. Ian Robertson,et al.  Problem-solving , 2001, Human Thinking.

[14]  Jan L. Plass,et al.  Intelligence Assessment with Computer Simulations. , 2005 .

[15]  Barbara M. Byrne,et al.  The MACS approach to testing for multigroup invariance of a second-order structure , 2006 .

[16]  Samuel Greiff,et al.  Dynamic Problem Solving: A new measurement perspective , 2012 .

[17]  Beatriz Luna,et al.  Cognitive processes in the development of TOL performance , 2006, Neuropsychologia.

[18]  K. Holyoak,et al.  The Cambridge handbook of thinking and reasoning , 2005 .

[19]  D. D. Bickerstaff,et al.  Computerized adaptive testing , 2015 .

[20]  Dianne C. Berry,et al.  The effect of different forms of advice on the control of a simulated complex system , 1995 .

[21]  Berndt Brehmer,et al.  Reliability and validity of performance measures in microworlds , 2002 .

[22]  Cleotilde Gonzalez,et al.  The Relationships between Cognitive Ability and Dynamic Decision Making. , 2005 .

[23]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[24]  Roy B. Clariana,et al.  Paper-based versus computer-based assessment: key factors associated with the test mode effect , 2002, Br. J. Educ. Technol..

[25]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[26]  W. Holmes Finch,et al.  Confirmatory Factor Analytic Procedures for the Determination of Measurement Invariance , 2006 .

[27]  S. Gregorich.  Do Self-Report Instruments Allow Meaningful Comparisons Across Diverse Population Groups?: Testing Measurement Invariance Using the Confirmatory Factor Analysis Framework , 2006, Medical care.

[28]  Joachim Funke,et al.  Complex problem solving: The European perspective - 10 years after , 2007 .

[29]  Samuel Greiff,et al.  Dynamic Problem Solving , 2012 .

[30]  B. Byrne,et al.  TEACHER'S CORNER: The MACS Approach to Testing for Multigroup Invariance of a Second-Order Structure: A Walk Through the Process , 2006 .

[31]  David Klahr,et al.  Dual Space Search During Scientific Reasoning , 1988, Cogn. Sci..

[32]  L. Thompson,et al.  Predicting academic achievement with cognitive ability , 2007 .

[33]  J. Funke,et al.  Beyond IQ: A Latent State-Trait Analysis of General Intelligence, Dynamic Decision Making, and Implicit Learning , 2011 .

[34]  Samuel Greiff,et al.  Cross-national gender differences in complex problem solving and their determinants , 2014 .