First among equals: Log data indicates ability differences despite equal scores

Abstract Analyzing test-taking behavior allows researchers to investigate the steps and actions resulting in the specific test outcome. The underlying assumption is that test-taking behavior is a valid indicator of the tested ability. The aim of this paper is to scrutinize this assumption in the context of complex problem solving (CPS) by analyzing individual differences in test-taking behavior and their relation to individual differences in established correlates of CPS ability. We investigated a sample of Finnish students who achieved the maximum score on five CPS tasks and worked on an additional intelligence measure. We logged the number of interactions with the CPS tasks and time-on-task. Students showed significant variance in both time-on-task (s2 = 260.09, p = .005) and the number of interactions (s2 = 0.381, p

[1]  Frank Goldhammer,et al.  Validating Test Score Interpretations Using Time Information , 2019, Front. Psychol..

[2]  Matthias Stadler,et al.  Complex problem solving and intelligence. A meta-analysis , 2015 .

[3]  Samuel Greiff,et al.  Sometimes less is more: Comparing the validity of complex problem solving measures ☆ , 2015 .

[4]  Samuel Greiff,et al.  Assessing complex problem-solving skills with multiple complex systems , 2015 .

[5]  Michael J Schwartz,et al.  Eye Metrics as an Objective Assessment of Surgical Skill , 2010, Annals of surgery.

[6]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[7]  Timo Ahonen,et al.  Test-taking behaviors in a neurocognitive assessment: associations with school-age outcomes in a Finnish longitudinal follow-up. , 2011, Psychological assessment.

[8]  Claire Scoular,et al.  Monitoring patterns of social and cognitive student behaviors in online collaborative problem solving assessments , 2020, Comput. Hum. Behav..

[9]  Johannes Naumann,et al.  The Skilled, the Knowledgeable, and the Motivated: Investigating the Strategic Allocation of Time on Task in a Computer-Based Assessment , 2019, Front. Psychol..

[10]  Frank Goldhammer,et al.  Validating process variables of sourcing in an assessment of multiple document comprehension , 2019, The British journal of educational psychology.

[11]  Johannes Naumann,et al.  Modelling individual response time effects between and within experimental speed conditions: A GLMM approach for speeded tests , 2017, The British journal of mathematical and statistical psychology.

[12]  R. M. Elliott,et al.  Behavior of Organisms , 1991 .

[13]  Johannes Naumann,et al.  More is not Always Better: The Relation between Item Response and Item Response Time in Raven's Matrices , 2015 .

[14]  Gregory K. W. K. Chung,et al.  Issues in the Computer-based Assessment of Collaborative Problem Solving , 2003 .

[15]  Walter Schneider,et al.  Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. , 1977 .

[16]  Johannes Naumann,et al.  Time-on-task effects in digital reading are non-linear and moderated by persons' skills and tasks' demands , 2017 .

[17]  A. Tomarken,et al.  Structural equation modeling: strengths, limitations, and misconceptions. , 2005, Annual review of clinical psychology.

[18]  Florian Domnick,et al.  Intelligence and school grades: A meta-analysis , 2015 .

[19]  Dragan Gasevic,et al.  Learning analytics to unveil learning strategies in a flipped classroom , 2017, Internet High. Educ..

[20]  Fabian Zehner,et al.  What to Make Of and How to Interpret Process Data , 2017 .

[21]  Samuel Greiff,et al.  The Role of Strategy Knowledge for the Application of Strategies in Complex Problem Solving Tasks , 2014, Technol. Knowl. Learn..

[22]  A. Jensen,et al.  What is a good g , 1994 .

[23]  Nancy Eisenberg,et al.  The relation of preterm and full-term infants' temperament to test-taking behaviors and developmental status , 1984 .

[24]  Samuel Greiff,et al.  Taking a Closer Look: An Exploratory Analysis of Successful and Unsuccessful Strategy Use in Complex Problems , 2019, Front. Psychol..

[25]  Walter Schneider,et al.  Controlled and Automatic Human Information Processing: 1. Detection, Search, and Attention. , 1977 .

[26]  Elena L. Grigorenko,et al.  Confirmatory Factor Analysis of the Sternberg Triarchic Abilities Test in Three International Samples , 2001 .

[27]  Fatos Xhafa,et al.  A parallel grid-based implementation for real-time processing of event log data of collaborative applications , 2010, Int. J. Web Grid Serv..

[28]  Samuel Greiff,et al.  Construct validity of complex problem solving: A comprehensive view on different facets of intelligence and school grades , 2016 .

[29]  Lei Zhang,et al.  Online Learners’ Reading Ability Detection Based on Eye-Tracking Sensors , 2016, Sensors.

[30]  Samuel Greiff,et al.  Teacher evaluation of student ability: what roles do teacher gender, student gender, and their interaction play? , 2014 .

[31]  Robert M. McFatter,et al.  The Use of Structural Equation Models in Interpreting Regression Equations Including Suppressor and Enhancer Variables , 1979 .

[32]  A. J. Conger.  A Revised Definition for Suppressor Variables: a Guide To Their Identification and Interpretation , 1974 .

[33]  Yan Ren,et al.  Exploring Multiple Goals Balancing in Complex Problem Solving Based on Log Data , 2019, Front. Psychol..

[34]  Samuel Greiff,et al.  Intelligence in action. Effective strategic behaviors while solving complex problems. , 2017 .

[35]  Samuel Greiff,et al.  Easily too difficult: Estimating item difficulty in computer simulated microworlds , 2016, Comput. Hum. Behav..

[36]  P. Frensch,et al.  Definitions, traditions, and a general framework for understanding complex problem solving , 1995 .

[37]  R. Azevedo.  Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues , 2015 .

[38]  J. Kranzler,et al.  Meta-analysis of the relationship between intelligence and inspection time , 2001 .

[39]  Frank Goldhammer,et al.  How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items , 2018, Behaviormetrika.

[40]  Samuel Greiff,et al.  Differentiating between static and complex problems: A theoretical framework and its empirical validation , 2019, Intelligence.

[41]  J. B. Olsen,et al.  THE FOUR GENERATIONS OF COMPUTERIZED EDUCATIONAL MEASUREMENT , 1988 .

[42]  Dirk Ifenthaler,et al.  Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives , 2012 .

[43]  Ulf Kroehne,et al.  (Why) Should We Use SEM? Pros and Cons of Structural Equation Modeling , 2003 .

[44]  D. Borsboom,et al.  The Theoretical Status of Latent Variables , 2003 .

[45]  Patrick Griffin,et al.  Assessment and Teaching of 21st Century Skills , 2011 .

[46]  Samuel Greiff,et al.  The logic of success: the relation between complex problem-solving skills and university achievement , 2018 .