Identifying Disengaged Survey Responses: New Evidence Using Response Time Metadata

ABSTRACT Disengaged responding is a phenomenon that often biases observed scores from achievement tests and surveys in practically and statistically significant ways. This problem has led to the development of methods to detect and correct for disengaged responses on both achievement test and survey scores. One major disadvantage when trying to detect disengaged responses on surveys is that, unlike on achievement tests, there are no correct answers. As a result, validating decision rules for detection methods is problematic. In this study, we condition results from a variety of detection methods used to identify disengaged survey responses on response times. We then show how this conditional approach may be useful in identifying where to set response time thresholds for survey items, as well as in avoiding misclassification when using other detection methods.

[1]  P. Pintrich,et al.  Motivation in Education: Theory, Research, and Applications , 1995 .

[2]  Christine E. DeMars.  Test Stakes and Item Format Interactions , 2000 .

[3]  Steven L. Wise,et al.  Setting the Response Time Threshold Parameter to Differentiate Solution Behavior From Rapid-Guessing Behavior , 2007 .

[4]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[5]  John A. Johnson.  Ascertaining the validity of individual protocols from Web-based personality inventories. , 2005 .

[6]  Steven L. Wise,et al.  An Investigation of the Differential Effort Received by Items on a Low-Stakes Computer-Based Test , 2006 .

[7]  Christine E. DeMars.  Changes in Rapid-Guessing Behavior Over a Series of Assessments , 2007 .

[8]  R. Tourangeau,et al.  Fast times and easy questions: the effects of age, experience and question complexity on web survey response times , 2008 .

[9]  Dena A. Pastor,et al.  Correlates of Rapid-Guessing Behavior in Low-Stakes Testing: Implications for Test Development and Measurement Practice , 2009 .

[10]  J Christine Harmes,et al.  Two Approaches for Identifying Low-Motivated Students in a Low-Stakes Assessment Context , 2011 .

[11]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[12]  S. Wise,et al.  Setting Response Time Thresholds for a CAT Item Pool: The Normative Threshold Method , 2012 .

[13]  Elizabeth M. Poposki,et al.  Detecting and Deterring Insufficient Effort Responding to Surveys , 2012 .

[14]  J. C. Setzer,et al.  An Investigation of Examinee Test-Taking Effort on a Large-Scale Assessment , 2013 .

[15]  Yi-Hsuan Lee,et al.  Using response time to investigate students' test-taking behaviors in a NAEP computer-based study , 2014, Large-scale Assessments in Education.

[16]  Frederick G. Conrad,et al.  Speeding in Web Surveys: The tendency to answer very fast and its association with straightlining , 2014 .

[17]  Joseph A. Rios,et al.  Identifying Low-Effort Examinees on Student Learning Outcomes Assessment: A Comparison of Two Approaches , 2014 .

[18]  K. W. Mossholder,et al.  Are we on the same page? The performance effects of congruence between supervisor and group trust. , 2015, The Journal of applied psychology.

[19]  Yigal Attali,et al.  Gamification in assessment: Do points affect test performance? , 2015, Comput. Educ..

[20]  Jason L. Huang,et al.  Insufficient effort responding: examining an insidious confound in survey data. , 2015, The Journal of applied psychology.

[21]  S. Wise.  Effort Analysis: Individual Score Validation of Achievement Test Data , 2015 .

[22]  Shelby J. Haberman,et al.  A New Procedure for Detection of Students’ Rapid Guessing Responses Using Response Time , 2016 .

[23]  Oliver Lüdtke,et al.  Test-taking engagement in PIAAC , 2016 .

[24]  S. Wise,et al.  Modeling Student Test-Taking Motivation in the Context of an Adaptive Achievement Test. , 2016 .

[25]  P. Curran.  Methods for the detection of carelessly invalid responses in survey data , 2016 .

[26]  Frank Goldhammer,et al.  Test-Taking Engagement in PIAAC. OECD Education Working Papers, No. 133. , 2016 .

[27]  Ou Lydia Liu,et al.  Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not? , 2017 .

[28]  Steven L. Wise,et al.  Rapid-Guessing Behavior: Its Identification, Interpretation, and Implications , 2017 .

[29]  J. Soland.  Are Achievement Gap Estimates Biased by Differential Student Test Effort? Putting an Important Policy Metric to the Test , 2018 .