Measuring Ability, Speed, or Both? Challenges, Psychometric Solutions, and What Can Be Gained From Experimental Control

The main challenge of ability tests relates to the difficulty of items, whereas speed tests demand that test takers complete very easy items quickly. This article proposes a conceptual framework to represent how performance depends on both between-person differences in speed and ability and the speed-ability compromise within persons. Related measurement challenges and psychometric models that have been proposed to deal with the challenges are discussed. It is argued that addressing individual differences in the speed-ability trade-off requires the control of item response times. In this way, response behavior can be captured exclusively with the response variable remedying problems in traditional measurement approaches.

[1]  J. Nietfeld,et al.  Examining the self-regulation of impulsive and reflective response styles on academic tasks , 2003 .

[2]  B. Baxter.  An experimental analysis of the contributions of speed and level in an intelligence test. , 1941 .

[3]  L. L. Thurstone,et al.  Ability, motivation, and speed , 1937 .

[4]  R. H. Klein Entink,et al.  A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers , 2008, Psychometrika.

[5]  Abe D. Hofman,et al.  The estimation of item response models with the lmer function from the lme4 package in R , 2011 .

[6]  G. Maris,et al.  Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy , 2012 .

[7]  Jeffrey J. Walczyk,et al.  Time Restriction and the Linkage between Subcomponent Efficiency and Algebraic Inequality Success. , 2006 .

[8]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[9]  Wayne A. Wickelgren,et al.  Speed-accuracy tradeoff and information processing dynamics , 1977 .

[10]  D. Woltz,et al.  Transforming response time and errors to address tradeoffs in complex measures of processing speed , 2015 .

[11]  Frank S. Freeman,et al.  Interpretation of educational measurements , 1928 .

[12]  G. Lienert,et al.  Ein Index zur numerischen Bestimmung der Niveau-Eigenschaft eines psychologischen Tests , 1960 .

[13]  Heiko Rölke,et al.  The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. , 2014 .

[14]  Douglas M. Bates,et al.  Estimating the Multilevel Rasch Model: With the lme4 Package , 2007 .

[15]  Steven L. Wise,et al.  An Application of Item Response Time: The Effort‐Moderated IRT Model , 2006 .

[16]  L. M. Kendall.  The Effects of Varying Time Limits on Test Validity , 1964 .

[17]  Frank M. Spinath,et al.  The interrelationship between speeded and unspeeded divergent thinking and reasoning, and the role of mental speed , 2011 .

[18]  A. Sias.  Interpretation of educational measurements. , 1928 .

[19]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[20]  M. Davison,et al.  On the Reliability and Validity of a Numerical Reasoning Speed Dimension Derived From Response Times Collected in Computerized Testing , 2012 .

[21]  Edward E. Roskam,et al.  Models for Speed and Time-Limit Tests , 1997 .

[22]  W. Furneaux,et al.  Intellectual Abilities and Problem-solving Behaviour , 1973 .

[23]  M. Davison,et al.  Modeling Individual Differences in Numerical Reasoning Speed as a Random Effect of Response Time Limits , 2011 .

[24]  Johannes Naumann,et al.  More is not Always Better: The Relation between Item Response and Item Response Time in Raven's Matrices , 2015 .

[25]  L. Cronbach,et al.  Time-limit tests: Estimating their reliability and degree of speeding , 1951, Psychometrika.

[26]  C. SPEARMAN,et al.  The Measurement of Intelligence , 1927, Nature.

[27]  A V Reed,et al.  Speed-Accuracy Trade-Off in Recognition Memory , 1973, Science.

[28]  Z. Breznitz,et al.  The Underlying Factors of Word Reading Rate , 2003 .

[29]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[30]  M. Kane.  Validating the Interpretations and Uses of Test Scores , 2013 .

[31]  O. Wilhelm,et al.  The relation of speeded and unspeeded reasoning with mental speed , 2002 .

[32]  J. N. Bassili,et al.  Response Latency and the Accessibility of Voting Intentions: What Contributes to Accessibility and How it Affects Vote Choice , 1995 .

[33]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[34]  David M. Riefer,et al.  Theoretical and empirical review of multinomial process tree modeling , 1999, Psychonomic bulletin & review.

[35]  Hartmut Leuthold,et al.  Short article: Control over speeded actions: A common processing locus for micro- and macro-trade-offs? , 2006, Quarterly journal of experimental psychology.

[36]  H. Gulliksen.  Theory of mental tests , 1952 .

[37]  Kristof Baten,et al.  A Joint Modeling Approach for Reaction Time and Accuracy in Psycholinguistic Experiments , 2011 .

[38]  J. N. Bassili,et al.  Response-time measurement in survey research: A method for CATI and a new look at nonattitudes. , 1991 .

[39]  Cornelis A.W. Glas,et al.  Modeling Nonignorable Missing Data in Speeded Tests , 2008 .

[40]  Yi-Hsuan Lee,et al.  A review of recent response-time analyses in educational testing , 2011 .

[41]  David F Lohman,et al.  Spatial Ability: Individual Differences in Speed and Level. Technical Report No. 9. Aptitude Research Project. , 1979 .

[42]  Michael C. Pyryt.  Human cognitive abilities: A survey of factor analytic studies , 1998 .

[43]  C Kaernbach,et al.  Simple adaptive testing with the weighted up-down method , 1991, Perception & psychophysics.

[44]  Andrew Heathcote,et al.  The Lognormal Race: A Cognitive-Process Model of Choice and Latency with Desirable Psychometric Properties , 2015, Psychometrika.

[45]  Bradley A. Hanson,et al.  Development and Calibration of an Item Response Model That Incorporates Response Time , 2005 .

[46]  Jochen Ranger,et al.  Improving Item Response Theory Model Calibration by Considering Response Times in Psychological Tests , 2012 .

[47]  S. Messick.  The matter of style: Manifestations of personality in cognition, learning, and teaching , 1994 .

[48]  Jochen Ranger,et al.  The case of dependency of responses and response times: A modeling approach based on standard latent trait models , 2012 .

[49]  Wim J. van der Linden,et al.  Using Response-Time Constraints to Control for Differential Speededness in Computerized Adaptive Testing , 1999 .

[50]  P. O. White.  Individual Differences in Speed, Accuracy and Persistence: a Mathematical Model for Problem Solving1,2 , 1973 .

[51]  Richard E. Stafford,et al.  The Speededness Quotient: A New Descriptive Statistic for Tests. , 1971 .

[52]  E. Roskam,et al.  A Rasch Model for the Speed-Accuracy Tradeoff In Time limited Tests , 1991 .

[53]  J. C. Johnston,et al.  On the limits of advance preparation for a task switch: do people prepare all the task some of the time or some of the task all the time? , 2005, Journal of experimental psychology. Human perception and performance.

[54]  Gerard J. P. Van Breukelen,et al.  Psychometric Modeling of response speed and accuracy with mixed and conditional regression , 2005 .

[55]  Richard P. Heitz,et al.  The speed-accuracy tradeoff: history, physiology, methodology, and behavior , 2014, Front. Neurosci..

[56]  Pere J. Ferrando,et al.  An Item Response Theory Model for Incorporating Response Time Data in Binary Personality Items , 2007 .

[57]  D. Lohman,et al.  The effect of speed-accuracy tradeoff on sex differences in mental rotation , 1986, Perception & psychophysics.

[58]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[59]  J. F. Schouten,et al.  Reaction time and accuracy. , 1967, Acta psychologica.

[60]  Zvia Breznitz,et al.  Increasing first graders' reading accuracy and comprehension by accelerating their reading rates. , 1987 .

[61]  A. Onwuegbuzie,et al.  The Effect of Time Constraints and Statistics Test Anxiety on Test Performance in a Statistics Course , 1995 .

[62]  James B. Rowe,et al.  Dissociable mechanisms of speed-accuracy tradeoff during visual perceptual learning are revealed by a hierarchical drift-diffusion model , 2014, Front. Neurosci..

[63]  Paul De Boeck,et al.  IRTrees: Tree-Based Item Response Models of the GLMM Family , 2012 .

[64]  Willem J. van der Linden,et al.  Linear Models for Optimal Test Design , 2005 .

[65]  Francis Tuerlinckx,et al.  Two interpretations of the discrimination parameter , 2005 .

[66]  W. D. Linden,et al.  Conceptual Issues in Response-Time Modeling. , 2009 .

[67]  Jochen Ranger,et al.  A Race Model for Responses and Response Times in Tests , 2015, Psychometrika.

[68]  David Wright,et al.  Exploiting the speed-accuracy trade-off. , 1999 .

[69]  P. Vernon,et al.  Reaction times and speed-of-processing: Their relationship to timed and untimed measures of intelligence , 1985 .

[70]  Francis Tuerlinckx,et al.  A Bivariate Generalized Linear Item Response Theory Modeling Framework to the Analysis of Responses and Response Times , 2015, Multivariate behavioral research.

[71]  Frank Goldhammer,et al.  Controlling Individuals’ Time Spent on Task in Speeded Performance Measures , 2014 .

[72]  A. Wesman,et al.  Consistency in response and logical interpretation of psychoneurotic inventory items. , 1941 .

[73]  Frank Goldhammer,et al.  FACT-2—The Frankfurt Adaptive Concentration Test: Convergent validity with self-reported cognitive failures. , 2009 .

[74]  David F. Lohman,et al.  Individual differences in errors and latencies on cognitive tasks , 1989 .

[75]  Stephen G. Sireci,et al.  Validity Issues in Test Speededness , 2007 .

[76]  R. Duncan Luce,et al.  Response Times: Their Role in Inferring Elementary Mental Organization , 1986 .

[77]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[78]  Ian Dennis,et al.  The speed–error trade‐off problem in psychometric testing , 1996 .

[79]  Deborah L. Schnipke,et al.  Modeling Item Response Times With a Two-State Mixture Model: A New Method of Measuring , 1997 .

[80]  Jean-Paul Fox,et al.  Evaluating cognitive theory: a joint modeling approach using responses and response times. , 2009, Psychological methods.

[81]  G. Salomon,et al.  Skill may not be enough: The role of mindfulness in learning and transfer , 1987 .

[82]  J. Beishuizen,et al.  Intellectual and metacognitive skills of novices while studying texts under conditions of text difficulty and time constraint , 2004 .

[83]  W. D. Linden.  Predictive Control of Speededness in Adaptive Testing , 2009 .

[84]  Yi-Hsuan Lee,et al.  Using response time to investigate students' test-taking behaviors in a NAEP computer-based study , 2014, Large-scale Assessments in Education.

[85]  Cornelis A.W. Glas,et al.  Statistical Tests of Conditional Independence Between Responses and/or Response Times on Test Items , 2010 .

[86]  Walczyk,et al.  Time Limitations Enhance Reading Comprehension. , 1999, Contemporary educational psychology.

[87]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[88]  R. Drechsler,et al.  The impact of instruction and response cost on the modulation of response-style in children with ADHD , 2010, Behavioral and Brain Functions.

[89]  Adam Chuderski,et al.  When are fluid intelligence and working memory isomorphic and when are they not , 2013 .

[90]  Paul De Boeck,et al.  Can fast and slow intelligence be differentiated , 2012 .

[91]  Denny Borsboom,et al.  Cognitive psychology meets psychometric theory: on the relation between process models for decision making and latent variable models for individual differences. , 2011, Psychological review.

[92]  J. Rost,et al.  Applications of Latent Trait and Latent Class Models in the Social Sciences , 1998 .

[93]  Susan Ellerin Rindler,et al.  PITFALLS IN ASSESSING TEST SPEEDEDNESS , 1979 .

[94]  N. D. Verhelst,et al.  A Logistic Model for Time-Limit Tests , 1992 .

[95]  Paul De Boeck,et al.  How Much Power and Speed Is Measured in This Test? , 2013, Assessment.