Conditioning factors of test-taking engagement in PIAAC: an exploratory IRT modelling approach considering person and item characteristics

BackgroundA potential problem of low-stakes large-scale assessments such as the Programme for the International Assessment of Adult Competencies (PIAAC) is low test-taking engagement. The present study pursued two goals in order to better understand conditioning factors of test-taking disengagement: First, a model-based approach was used to investigate whether item indicators of disengagement constitute a continuous latent person variable by domain. Second, the effects of person and item characteristics were jointly tested using explanatory item response models.MethodsAnalyses were based on the Canadian sample of Round 1 of the PIAAC, with N = 26,683 participants completing test items in the domains of literacy, numeracy, and problem solving. Binary item disengagement indicators were created by means of item response time thresholds.ResultsThe results showed that disengagement indicators define a latent dimension by domain. Disengagement increased with lower educational attainment, lower cognitive skills, and when the test language was not the participant’s native language. Gender did not exert any effect on disengagement, while age had a positive effect for problem solving only. An item’s location in the second of two assessment modules was positively related to disengagement, as was item difficulty. The latter effect was negatively moderated by cognitive skill, suggesting that poor test-takers are especially likely to disengage with more difficult items.ConclusionsThe negative effect of cognitive skill, the positive effect of item difficulty, and their negative interaction effect support the assumption that disengagement is the outcome of individual expectations about success (informed disengagement).

[1]  J. Rost,et al.  Lehrbuch Testtheorie, Testkonstruktion , 1999 .

[2]  J. Spence,et al.  Achievement and achievement motives : psychological and sociological approaches , 1984 .

[3]  Steven L. Wise,et al.  An Investigation of the Differential Effort Received by Items on a Low-Stakes Computer-Based Test , 2006 .

[4]  Yi-Hsuan Lee,et al.  Using response time to investigate students' test-taking behaviors in a NAEP computer-based study , 2014, Large-scale Assessments in Education.

[5]  J. C. Setzer,et al.  An Investigation of Examinee Test-Taking Effort on a Large-Scale Assessment , 2013 .

[6]  Steven L. Wise,et al.  Rapid-Guessing Behavior: Its Identification, Interpretation, and Implications , 2017 .

[7]  Abe D. Hofman,et al.  The estimation of item response models with the lmer function from the lme4 package in R , 2011 .

[8]  P. Gollwitzer.  The Volitional Benefits of Planning , 1996 .

[9]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[10]  Frances B. Stancavage,et al.  NAEP Validity Studies: An Investigation of Why Students Do Not Respond to Questions. Working Paper No. 2003-12. , 2003 .

[11]  Ou Lydia Liu,et al.  Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not? , 2017 .

[12]  Kentaro Yamamoto,et al.  ESTIMATING THE EFFECTS OF TEST LENGTH AND TEST TIME ON PARAMETER ESTIMATION USING THE HYBRID MODEL , 1995 .

[13]  S. Finney,et al.  Low-Stakes Testing and Psychological Reactance: Using the Hong Psychological Reactance Scale to Better Understand Compliant and Non-Compliant Examinees , 2011 .

[14]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[15]  Wim J. van der Linden,et al.  Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing , 2008 .

[16]  Steven L. Wise,et al.  An Application of Item Response Time: The Effort‐Moderated IRT Model , 2006 .

[17]  B. Finn.  Measuring Motivation in Low-Stakes Assessments , 2015 .

[18]  Rebecca Holman,et al.  Modelling non-ignorable missing-data mechanisms with item response theory models. , 2005, The British journal of mathematical and statistical psychology.

[19]  J. Kuhl.  Chapter 5 – A Functional-Design Approach to Motivation and Self-Regulation: The Dynamics of Personality Systems Interactions , 2000 .

[20]  R. Desjardins,et al.  OECD Skills Outlook 2013: First Results from the Survey of Adult Skills , 2013 .

[21]  Robin D. Anderson,et al.  Proctors Matter: Strategies for Increasing Examinee Effort on General Education Program Assessments , 2009, The Journal of General Education.

[22]  J. Eccles.  Expectancies, values and academic behaviors , 1983 .

[23]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[24]  Steffi Pohl,et al.  Dealing With Omitted and Not-Reached Items in Competence Tests , 2014 .

[25]  Jean-Paul Fox,et al.  Person-Fit Statistics for Joint Models for Accuracy and Speed , 2017 .

[26]  Christine E. DeMars,et al.  Low Examinee Effort in Low-Stakes Assessment: Problems and Potential Solutions , 2005 .

[27]  David A. Bergin,et al.  Predicting student achievement for low stakes tests with effort and task value , 2008 .

[28]  Jeffrey Douglas,et al.  Nonparametric Item Response Function Estimation for Assessing Parametric Model Fit , 2001 .

[29]  T. Haladyna,et al.  Construct-Irrelevant Variance in High-Stakes Testing. , 2005 .

[30]  Claus H. Carstensen,et al.  Investigating Mechanisms for Missing Responses in Competence Tests , 2015 .

[31]  S. Wise.  Effort Analysis: Individual Score Validation of Achievement Test Data , 2015 .

[32]  J. Bargh,et al.  The psychology of action : linking cognition and motivation to behavior , 1999 .

[33]  Ulrich Trautwein,et al.  Probing for the Multiplicative Term in Modern Expectancy-Value Theory: A Latent Interaction Modeling Study. , 2012 .

[34]  Douglas M. Bates,et al.  Estimating the Multilevel Rasch Model: With the lme4 Package , 2007 .

[35]  Andreas Frey,et al.  Too hard, too easy, or just right? The relationship between effort or boredom and ability-difficulty fit , 2013 .

[36]  Christine E. DeMars,et al.  The Role of Gender in Test-Taking Motivation under Low-Stakes Conditions. , 2013 .

[37]  S. Wise,et al.  A General Approach to Measuring Test-Taking Effort on Computer-Based Tests , 2017 .

[38]  Oliver Lüdtke,et al.  Test-taking engagement in PIAAC , 2016 .

[39]  Johannes Hartig,et al.  Student, School, and Country Differences in Sustained Test-Taking Effort in the 2009 PISA Reading Assessment , 2014 .

[40]  J. P. Meyer,et al.  A Mixture Rasch Model With Item Response Time Components , 2010 .

[41]  J. Eccles,et al.  Motivational beliefs, values, and goals. , 2002, Annual review of psychology.

[42]  Henry Braun,et al.  An Experimental Study of the Effects of Monetary Incentives on Performance on the 12th-Grade NAEP Reading Assessment , 2011, Teachers College Record: The Voice of Scholarship in Education.

[43]  Donna L. Sundre,et al.  An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? , 2004 .

[44]  Deborah L. Schnipke,et al.  Modeling Item Response Times With a Two-State Mixture Model: A New Method of Measuring , 1997 .

[45]  Francesco Bartolucci,et al.  A class of multidimensional IRT models for testing unidimensionality and clustering items , 2007 .

[46]  S. Urbina.  Essentials of Psychological Testing , 2005, PsyPag Quarterly.

[47]  Steven L. Wise,et al.  Strategies for Managing the Problem of Unmotivated Examinees in Low-Stakes Testing Programs , 2009, The Journal of General Education.

[48]  B. Wright.  Reasonable mean-square fit values , 1994 .

[49]  Pao-Kuei Wu,et al.  MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING , 1996 .

[50]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[51]  The role of test-taking motivation for students’ performance in low-stakes assessments: an investigation of school-track-specific differences , 2014 .

[52]  Sun Hee Kim,et al.  An item response theory approach to longitudinal analysis with application to summer setback in preschool language/literacy , 2013 .

[53]  Steven L. Wise,et al.  Setting the Response Time Threshold Parameter to Differentiate Solution Behavior From Rapid-Guessing Behavior , 2007 .

[54]  Heiko Rölke,et al.  The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. , 2014 .

[55]  H. Marsh,et al.  Reciprocal Effects of Self-Concept and Performance From a Multidimensional Perspective: Beyond Seductive Pleasure and Unidimensional Perspectives , 2006, Perspectives on psychological science : a journal of the Association for Psychological Science.

[56]  Jeffrey K. Smith,et al.  Consequence of Performance, Test, Motivation, and Mentally Taxing Items , 1995 .

[57]  S. Wise,et al.  Setting Response Time Thresholds for a CAT Item Pool: The Normative Threshold Method , 2012 .