A multilevel item response model for item position effects and individual persistence

The paper presents a multilevel item response model for item position effects. It includes individual differences regarding the position effect to which we refer to as the persistence of the test-takers. The model is applied to published data from the PISA 2006 science assessment. We analyzed responses to 103 science test items from N = 64.251 students from 10 countries selected to cover a wide range of national performance levels. All effects of interest were analyzed separately for each country. A significant negative effect of item position on performance was found in all countries, which is more prominent in countries with a lower national performance level. The individual differences in persistence were relatively small in all countries, but more pronounced in countries with lower performance levels. Students’ performance level is practically uncorrelated with persistence in high performing countries, while it is negatively correlated within low performing countries.

[1]  Andreas Gold,et al.  The confirmatory investigation of APM items with loadings as a function of the position and easiness of items: A two-dimensional model of APM , 2009 .

[2]  Lale Khorramdel,et al.  Analysing item position effects due to test booklet design within large-scale assessment , 2011 .

[3]  Neal M. Kingston,et al.  THE EFFECT OF THE POSITION OF AN ITEM WITHIN A TEST ON ITEM RESPONDING BEHAVIOR: AN ANALYSIS BASED ON ITEM RESPONSE THEORY , 1982 .

[4]  Johannes Hartig,et al.  A Confirmatory Analysis of Item Reliability Trends (CAIRT): Differentiating True Score and Error Variance in the Analysis of Item Context Effects , 2007, Multivariate behavioral research.

[5]  J. Hox.  Multilevel analysis: Techniques and applications, 2nd ed. , 2010 .

[6]  Mark Wilson,et al.  Constructing Measures: An Item Response Modeling Approach , 2004 .

[7]  Neal M. Kingston,et al.  Item Location Effects and Their Implications for IRT Equating and Adaptive Testing , 1984 .

[8]  Herbert Matschinger,et al.  Estimation of item location effects by means of the generalized logistic regression model: a simulation study and an application , 2008 .

[9]  Lale Khorramdel,et al.  Examining item-position effects in large-scale assessment using the Linear Logistic Test Model , 2008 .

[10]  Andreas Schleicher,et al.  PISA 2006: Science Competencies for Tomorrow's World , 2007 .

[11]  R. Dawis,et al.  The Influence of Test Context On Item Difficulty , 1976 .

[12]  S. Troche,et al.  On the special relationship between fluid and general intelligence: New evidence obtained by considering the position effect , 2011 .

[13]  Walter D. Way,et al.  Item Position and Item Difficulty Change in an IRT-Based Common Item Equating Design , 2008 .

[14]  Tom Verguts,et al.  A Rasch Model for Detecting Learning While Solving an Intelligence Test , 2000 .

[15]  E. S. Knowles,et al.  Item context effects on personality scales: Measuring changes the measure. , 1988 .

[16]  L. Corrado.  Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models , 2005 .

[17]  Cornelis A.W. Glas,et al.  A dynamic generalization of the Rasch model , 1993 .

[18]  Akihito Kamata,et al.  Multilevel Rasch Models , 2007 .