Grade of Membership Response Time Model for Detecting Guessing Behaviors

A response model that is able to detect guessing behaviors and produce unbiased estimates in low-stake conditions using timing information is proposed. The model is a special case of the grade of membership model in which responses are modeled as partial members of a class that is affected by motivation and a class that responds only according to the level of ability. Monte Carlo simulations were conducted to compare the proposed model with an approach that ignored guessing and an approach based on item filtering. In each simulated condition, the proposed model outperformed the other approaches by showing the lowest level of bias and the highest precision of item and persons estimates. Finally, the model was estimated using real life data from Programme for the International Assessment of Adult Competencies research (PIAAC). The results showed slight but expected corrections for the levels of proficiency in all countries.

[1]  E. Erosheva.  Comparing Latent Structures of the Grade of Membership, Rasch, and Latent Class Models , 2005 .

[2]  Harold F. O'Neil,et al.  Effects of Motivational Interventions on the National Assessment of Educational Progress Mathematics Performance , 1995 .

[3]  Multilevel Mixture Models , 2006 .

[4]  Donna L. Sundre,et al.  An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? , 2004 .

[5]  Elena A. Erosheva,et al.  2 A Tale of Two ( Types of ) Memberships : Comparing Mixed and Partial Membership with a Continuous Data Example , 2014 .

[6]  Elena A. Erosheva,et al.  Grade of membership and latent structure models with application to disability survey data , 2002 .

[7]  George B. Macready,et al.  Concomitant-Variable Latent-Class Models , 1988 .

[8]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[9]  Edoardo M. Airoldi,et al.  Introduction to Mixed Membership Models and Methods , 2014, Handbook of Mixed Membership Models and Their Applications.

[10]  Robert J. Mislevy,et al.  Modeling item responses when different subjects employ different solution strategies , 1990 .

[11]  Steven L. Wise,et al.  Taking the Time to Improve the Validity of Low-Stakes Tests: The Effort-Monitoring CBT. , 2006 .

[12]  Jeroen K. Vermunt,et al.  Factor Analysis with Categorical Indicators: A Comparison Between Traditional and Latent Class Approaches , 2005 .

[13]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[14]  Deborah L. Schnipke,et al.  Modeling Item Response Times With a Two-State Mixture Model: A New Method of Measuring , 1997 .

[15]  Kentaro Yamamoto,et al.  HYBRID MODEL OF IRT AND LATENT CLASS MODELS , 1982 .

[16]  Donna L. Sundre,et al.  Motivation Matters: Using the Student Opinion Scale to Make Valid Inferences About Student Performance , 2009, The Journal of General Education.

[17]  Christine E. DeMars,et al.  Low Examinee Effort in Low-Stakes Assessment: Problems and Potential Solutions , 2005 .

[18]  Henk Kelderman,et al.  The Mixed Birnbaum Model: Estimation using Collateral Information , 2000 .

[19]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[20]  H. van der Flier,et al.  Collateral information and Mixed Rasch models , 1999 .

[21]  Mark Wilson,et al.  A framework for item response models , 2004 .

[22]  Jeffrey K. Smith,et al.  Consequence of Performance, Test, Motivation, and Mentally Taxing Items , 1995 .

[23]  Matthias von Davier,et al.  Mixture-Distribution and HYBRID Rasch Models , 2007 .

[24]  Jürgen Rost,et al.  A logistic mixture distribution model for polychotomous item responses , 1991 .

[25]  Matthias von Davier,et al.  Multivariate and Mixture Distribution Rasch Models , 2007 .

[26]  R. Desjardins,et al.  OECD Skills Outlook 2013: First Results from the Survey of Adult Skills , 2013 .