Item Position Effects Are Moderated by Changes in Test-Taking Effort

This article examines the interdependency of two context effects that are known to occur regularly in large-scale assessments: item position effects and effects of test-taking effort on the probability of correctly answering an item. A microlongitudinal design was used to measure test-taking effort over the course of a large-scale assessment of 60 min. Two components of test-taking effort were investigated: initial effort and change in effort. Both components of test-taking effort significantly affected the probability to solve an item. In addition, it was found that participants’ current test-taking effort diminished considerably across the course of the test. Furthermore, a substantial linear position effect was found, which indicated that item difficulty increased during the test. This position effect varied considerably across persons. Concerning the interplay of position effects and test-taking effort, it was found that only the change in effort moderates the position effect and that persons differ with respect to this moderation effect. The consequences of these results concerning the reliability and validity of large-scale assessments are discussed.

[1]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[2]  Samuel Messick,et al.  The Psychology of Educational Measurement. , 1984 .

[3]  Neal M. Kingston,et al.  Item Location Effects and Their Implications for IRT Equating and Adaptive Testing , 1984 .

[4]  Linda L. Cook,et al.  Problems Related to the Use of Conventional and Item Response Theory Equating Methods in Less Than Optimal Circumstances , 1987 .

[5]  Howard Wainer,et al.  Item Clusters and Computerized Adaptive Testing: A Case for Testlets , 1987 .

[6]  Robert L. Brennan.  The Context of Context Effects , 1992 .

[7]  J. Marks.  Performance and Competence in Second Language Acquisition , 1998 .

[8]  Richard J. Patz,et al.  A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models , 1999 .

[9]  N. G. Best,et al.  WinBUGS User Manual: Version 1.4 , 2001 .

[10]  Mark Wilson,et al.  A framework for item response models , 2004 .

[11]  Francis Tuerlinckx,et al.  Models for residual dependencies , 2004 .

[12]  R. Brennan,et al.  Test Equating, Scaling, and Linking , 2004 .

[13]  J. Schepers,et al.  Models with item and item group predictors , 2004 .

[14]  Christine E. DeMars,et al.  Low Examinee Effort in Low-Stakes Assessment: Problems and Potential Solutions , 2005 .

[15]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[16]  Lale Khorramdel,et al.  Examining item-position effects in large-scale assessment using the Linear Logistic Test Model , 2008 .

[17]  Analyzing position effects within reasoning items using the LLTM for structurally incomplete data , 2008 .

[18]  Walter D. Way,et al.  Item Position and Item Difficulty Change in an IRT-Based Common Item Equating Design , 2008 .

[19]  Andreas Gold,et al.  The confirmatory investigation of APM items with loadings as a function of the position and easiness of items: A two-dimensional model of APM , 2009 .

[20]  André A. Rupp,et al.  An NCME Instructional Module on Booklet Designs in Large‐Scale Assessments of Student Achievement: Theory and Practice , 2009 .

[21]  Jean-Paul Fox,et al.  Bayesian Item Response Modeling , 2010 .

[22]  Student Motivation and Effort in the Swedish TIMSS Advanced Field Study , 2010 .

[23]  J. Kruschke.  Doing Bayesian Data Analysis: A Tutorial with R and BUGS , 2010 .

[24]  J. Fox.  Bayesian Item Response Modeling: Theory and Applications , 2010 .

[25]  Eugenio Gonzalez,et al.  principles of multiple matrix booklet designs and parameter recovery in large-scale assessments , 2010 .

[26]  Lale Khorramdel,et al.  Analysing item position effects due to test booklet design within large-scale assessment , 2011 .

[27]  Steven L. Wise,et al.  A model of examinee test-taking effort. , 2011 .

[28]  J Christine Harmes,et al.  Two Approaches for Identifying Low-Motivated Students in a Low-Stakes Assessment Context , 2011 .

[29]  Abe D. Hofman,et al.  The estimation of item response models with the lmer function from the lme4 package in R , 2011 .

[30]  Heinz Holling,et al.  Measuring current achievement motivation with the QCM: Short form development and investigation of measurement invariance , 2011 .

[31]  High-Stakes Testing in Education: Science and Practice in K–12 Settings , 2012 .

[32]  Johannes Hartig,et al.  A multilevel item response model for item position effects and individual persistence , 2012 .

[33]  Andreas Frey,et al.  On the Importance of Using Balanced Booklet Designs in PISA , 2012 .

[34]  Martyn Plummer,et al.  JAGS: Just Another Gibbs Sampler , 2012 .

[35]  Rianne Janssen,et al.  Modeling Item-Position Effects Within an IRT Framework , 2012 .

[36]  H. A. Pant.  IQB-Ländervergleich 2012 Mathematische und naturwissenschaftliche Kompetenzen am Ende der Sekundarstufe I , 2013 .

[37]  Multilevel Modeling of Item Position Effects , 2013 .

[38]  Johannes Hartig,et al.  Student, School, and Country Differences in Sustained Test-Taking Effort in the 2009 PISA Reading Assessment , 2014 .

[39]  Sebastian Weirich,et al.  Modeling Item Position Effects Using Generalized Linear Mixed Models , 2014 .

[40]  An Investigation of Position Effects in Large-Scale Writing Assessments , 2014 .

[41]  The role of test-taking motivation for students’ performance in low-stakes assessments: an investigation of school-track-specific differences , 2014 .

[42]  Sebastian Weirich,et al.  Effects of Design Properties on Parameter Estimation in Large-Scale Assessments , 2015, Educational and psychological measurement.

[43]  Christiane Penk,et al.  Change in test-taking motivation and its relationship to test performance in low-stakes assessments , 2017 .