More is not Always Better: The Relation between Item Response and Item Response Time in Raven's Matrices

The role of response time in completing an item can have very different interpretations. Responding more slowly could be positively related to success as the item is answered more carefully. However, the association may be negative if working faster indicates higher ability. The objective of this study was to clarify the validity of each assumption for reasoning items considering the mode of processing. A total of 230 persons completed a computerized version of Raven’s Advanced Progressive Matrices test. Results revealed that response time overall had a negative effect. However, this effect was moderated by items and persons. For easy items and able persons the effect was strongly negative, for difficult items and less able persons it was less negative or even positive. The number of rules involved in a matrix problem proved to explain item difficulty significantly. Most importantly, a positive interaction effect between the number of rules and item response time indicated that the response time effect became less negative with an increasing number of rules. Moreover, exploratory analyses suggested that the error type influenced the response time effect.

[1]  Bradley A. Hanson,et al.  Development and Calibration of an Item Response Model That Incorporates Response Time , 2005 .

[2]  Edward E. Roskam,et al.  Models for Speed and Time-Limit Tests , 1997 .

[3]  S. Greven,et al.  On the behaviour of marginal and conditional AIC in linear mixed models , 2010 .

[4]  The Use of Item Response Time Measurements in the Construction of Educational Achievement Tests , 1953 .

[5]  Wendy M. Yen,et al.  Scaling Performance Assessments: Strategies for Managing Local Item Dependence , 1993 .

[6]  Wayne A. Wickelgren,et al.  Speed-accuracy tradeoff and information processing dynamics , 1977 .

[7]  R. H. Klein Entink,et al.  A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers , 2008, Psychometrika.

[8]  Alistair Sutcliffe,et al.  A Taxonomy of Error Types for Failure Analysis and Risk Assessment , 1998, Int. J. Hum. Comput. Interact..

[9]  Wendy M. Yen,et al.  Effects of Local Item Dependence on the Fit and Equating Performance of the Three-Parameter Logistic Model , 1984 .

[10]  Paul De Boeck,et al.  Random Item IRT Models , 2008 .

[11]  Frank Goldhammer,et al.  Speed of reasoning and its relation to reasoning ability , 2011 .

[12]  Heiko Rölke,et al.  The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. , 2014 .

[13]  F. Vaida,et al.  Conditional Akaike information for mixed-effects models , 2005 .

[14]  Samuel Greiff,et al.  Exploring the relation between speed and ability in complex problem solving , 2015 .

[15]  M. W. Molen,et al.  Error analysis of raven test performance , 1994 .

[16]  Nathaniel Lasry,et al.  Response times to conceptual questions , 2013 .

[17]  P. Ackerman.  Predicting individual differences in complex skill acquisition: dynamics of ability determinants. , 1992, The Journal of applied psychology.

[18]  D. Bates,et al.  Linear Mixed-Effects Models using 'Eigen' and S4 , 2015 .

[19]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[20]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[21]  A. Neubauer.  Speed of information processing in the hick paradigm and response latencies in a psychometric intelligence test , 1990 .

[22]  D. Mewhort,et al.  Analysis of Response Time Distributions: An Example Using the Stroop Task , 1991 .

[23]  Walter Schneider,et al.  Controlled & automatic processing: behavior, theory, and biological mechanisms , 2003, Cogn. Sci..

[24]  K. McGrew.  CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research , 2009 .

[26]  Samuel Greiff,et al.  Exploring the Relation between Time on Task and Ability in Complex Problem Solving , 2015 .

[27]  Mollie E. Brooks,et al.  Generalized linear mixed models: a practical guide for ecology and evolution. , 2009, Trends in ecology & evolution.

[28]  Douglas M. Bates,et al.  Estimating the Multilevel Rasch Model: With the lme4 Package , 2007 .

[29]  R. Shiffrin,et al.  Controlled and automatic human information processing: I , 1977 .

[30]  M. Meo,et al.  Element salience as a predictor of item difficulty for Raven's Progressive Matrices , 2007 .

[31]  Johannes Naumann,et al.  Assessing Individual Differences in Basic Computer Skills , 2013 .

[32]  Patrick C. Kyllonen,et al.  Human Cognitive Abilities , 2015 .

[33]  M. J. Emerson,et al.  The Unity and Diversity of Executive Functions and Their Contributions to Complex “Frontal Lobe” Tasks: A Latent Variable Analysis , 2000, Cognitive Psychology.

[34]  W. D. Linden,et al.  Conceptual Issues in Response-Time Modeling. , 2009 .

[35]  Steven L. Wise,et al.  An Application of Item Response Time: The Effort‐Moderated IRT Model , 2006 .

[36]  Nadin Beckmann,et al.  Effects of feedback on performance and response latencies in untimed reasoning tests , 2005 .

[37]  R. L. Babcock.  Analysis of Age Differences in Types of Errors on the Raven's Advanced Progressive Matrices. , 2002 .

[38]  S. L. Sporer,et al.  Eyewitness identification accuracy, confidence, and decision times in simultaneous and sequential lineups , 1993 .

[39]  J. Carroll.  Human Cognitive Abilities-a sur-vey of factor-analytic studies , 1993 .

[40]  P. Johnson-Laird.  A model theory of induction , 1994 .

[41]  P. Boeck,et al.  Generation speed in Raven's progressive matrices test , 1999 .

[42]  Yulia Dodonova,et al.  Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty , 2013 .

[43]  Lutz F. Hornke,et al.  Response Time in Computer-Aided Testing: A "Verbal Memory" Test for Routes and Maps , 2005 .

[44]  M A Just,et al.  From the SelectedWorks of Marcel Adam Just 1990 What one intelligence test measures : A theoretical account of the processing in the Raven Progressive Matrices Test , 2016 .

[45]  Abe D. Hofman,et al.  The estimation of item response models with the lmer function from the lme4 package in R , 2011 .

[46]  P. Johnson-Laird,et al.  Reasoning from inconsistency to consistency. , 2004, Psychological review.

[47]  Wim J. van der Linden,et al.  Using Response-Time Constraints to Control for Differential Speededness in Computerized Adaptive Testing , 1999 .

[48]  Walter Schneider,et al.  Controlled and Automatic Human Information Processing: 1. Detection, Search, and Attention. , 1977 .

[49]  J. Beckmann.  Differentielle Latenzzeiteffekte bei der Bearbeitung von Reasoning-Items , 2000 .

[50]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[51]  Lutz F. Hornke,et al.  Item Response Times in Computerized Adaptive Testing , 2000 .

[52]  J WIM,et al.  A HIERARCHICAL FRAMEWORK FOR MODELING SPEED AND ACCURACY ON TEST ITEMS , 2007 .