Too Fast, too Straight, too Weird: Non-Reactive Indicators for Meaningless Data in Internet Surveys

Practitioners use various indicators to screen for meaningless, careless, or fraudulent responses in Internet surveys. This study employs an experimental-like design to empirically test the ability of non-reactive indicators to identify records with low data quality. Findings suggest that careless responses are most reliably identified by questionnaire completion time, but the tested indicators do not allow for detecting intended faking. The article introduces various indicators, their benefits and drawbacks, proposes a completion speed index for common application in data cleaning, and discusses whether to remove meaningless records at all.

[1]  Nicholas W. Jankowski,et al.  Conducting Online Surveys , 2006 .

[2]  Stanley Presser,et al.  Public Opinion and Public Ignorance: The Fine Line Between Attitudes and Nonattitudes , 1980, American Journal of Sociology.

[3]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[4]  I. Ajzen Attitudes, Personality and Behavior , 1988 .

[5]  Stefan Stieger,et al.  What are participants doing while filling in an online questionnaire: A paradata collection tool and an empirical study , 2010, Comput. Hum. Behav..

[6]  W. Revelle,et al.  Predicting variability from perceived situational similarity , 1986 .

[7]  Joseph A. Konstan,et al.  The Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research , 2006, J. Comput. Mediat. Commun..

[8]  Christian Fg Schendera,et al.  Datenqualität mit SPSS , 2007 .

[9]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[10]  W. Dijkstra,et al.  Response latencies and (para)linguistic expressions as indicators of response error , 2004 .

[11]  J. E. Kurtz,et al.  Semantic Response Consistency and Protocol Validity in Structured Personality Assessment: The Case of the NEO-PI-R , 2001, Journal of personality assessment.

[12]  Birk Diedenhofen,et al.  PageFocus: Using paradata to detect and prevent cheating on online achievement tests , 2017, Behavior research methods.

[13]  Karen Cortell Reisman "The art of asking questions". , 2008, Texas dental journal.

[14]  Hunter Gehlbach,et al.  Using the Theory of Satisficing to Evaluate the Quality of Survey Data , 2011, Research in Higher Education.

[15]  J. Krosnick Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[16]  R. Tourangeau,et al.  Fast times and easy questions: the effects of age, experience and question complexity on web survey response times , 2008 .

[17]  Jason W. Osborne,et al.  Best Practices in Data Cleaning: A Complete Guide to Everything You Need to Do Before and After Collecting Your Data , 2012 .

[18]  R L Greene,et al.  Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials. , 1989, Journal of clinical psychology.

[19]  Anja S. Göritz,et al.  The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels , 2004 .

[20]  Christoph J. Kemper,et al.  How Do Real and Falsified Data Differ? Psychology of Survey Response as a Source of Falsification Indicators in Face-to-Face Surveys , 2014 .

[21]  K. Slomczynski,et al.  Non-unique Records in International Survey Projects: The Need for Extending Data Quality Control , 2017 .

[22]  Carol M. Woods Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis , 2006 .

[23]  Melanie Revilla,et al.  What are the Links in a Web Survey Among Response Time, Quality, and Auto-Evaluation of the Efforts Done? , 2015 .

[24]  Neil Malhotra,et al.  Completion Time and Response Order Effects in Web Surveys , 2008 .

[25]  A. Furnham,et al.  On-line questionnaire completion time and personality test scores , 2013 .

[26]  Eszter Hargittai,et al.  An Update on Survey Measures of Web-Oriented Digital Literacy , 2009 .

[27]  M. Birnbaum Methodological and Ethical Issues in Conducting Social Psychology Research via the Internet. , 2004 .

[28]  Ulf-Dietrich Reips,et al.  A Brief History of Web Experimenting , 2000 .

[29]  Kristen M Olson,et al.  Collecting Paradata for Measurement Error Evaluations , 2013 .

[30]  James D. Wright,et al.  Handbook of Survey Research. , 1985 .

[31]  Kumar Nagaraja Rao,et al.  Survey Mode Effects on Data Quality: Comparison of Web and Mail Modes in a U.S. National Panel Survey , 2012 .

[32]  S. Day,et al.  Survey Questions: Handcrafting the Standardized Questionnaire. , 1987 .

[33]  V. Toepoel,et al.  Straightlining in Web survey panels over time , 2015 .

[34]  Justin A. DeSimone,et al.  Best practice recommendations for data screening , 2015 .

[35]  Nonattitudes / pseudo-opinions : definitional problems, critical variables, cognitive components and solutions , 2001 .

[36]  J. Butcher,et al.  Detection of faking on the MMPI-2: differentiation among faking-bad, denial, and claiming extreme virtue. , 1996, Journal of personality assessment.

[37]  H. Breitsohl,et al.  The Impact of Insufficient Effort Responding Detection Methods on Substantive Responses: Results from an Experiment Testing Parameter Invariance , 2018 .

[38]  H. Weisberg The Total Survey Error Approach: A Guide to the New Science of Survey Research , 2005 .

[39]  I. Allen Detecting respondents who fake and confuse information about question areas on surveys. , 1966, Journal of Applied Psychology.

[40]  Vivek Bhaskaran,et al.  Online Surveys For Dummies , 2010 .

[41]  Birk Diedenhofen,et al.  Seriousness checks are useful to improve data validity in online research , 2012, Behavior Research Methods.

[42]  J. Krosnick,et al.  Survey research. , 1999, Annual review of psychology.

[43]  Troy Devon Thomas,et al.  Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies , 2013 .

[44]  Hans Baumgartner,et al.  Response Styles in Marketing Research: A Cross-National Investigation , 2001 .

[45]  S. L. Payne Thoughts About Meaningless Questions , 1950 .

[46]  Christoph J. Kemper,et al.  Nuisance or Remedy? The Utility of Stylistic Responding as an Indicator of Data Fabrication in Surveys , 2014 .

[47]  John A. Johnson Ascertaining the validity of individual protocols from Web-based personality inventories. , 2005 .

[48]  Jean-Paul Fox,et al.  Using Item Response Theory to Measure Extreme Response Style in Marketing Research: A Global Investigation , 2008 .

[49]  Uninformed response rates in survey research: New evidence , 1985 .

[50]  O. Azfar,et al.  Identifying Reticent Respondents: Assessing the Quality of Survey Data on Corruption and Values , 2005, Economic Development and Cultural Change.

[51]  Sjoerd Hardeman Organization level research in scientometrics: a plea for an explicit pragmatic approach , 2012, Scientometrics.

[52]  G. Moran,et al.  Bogus Publicity Items and the Contingency Between Awareness and Media-Induced Pretrial Prejudice , 1997 .

[53]  Bengü Börkan,et al.  The Mode Effect in Mixed-Mode Surveys , 2010 .

[54]  D. Kuhn THE SKILLS OF ARGUMENT , 2008, Education for Thinking.

[55]  Elizabeth A. Skewes,et al.  Item Nonresponse: Distinguishing between don't Know and Refuse , 2002 .

[56]  Dominik J. Leiner Our research’s breadth lives on convenience samples A case study of the online respondent pool “SoSci Panel” , 2016 .

[57]  P. Curran Methods for the detection of carelessly invalid responses in survey data , 2016 .

[58]  Victor J. Strecher,et al.  Data Quality in web-based HIV/AIDS research: Handling Invalid and Suspicious Data. , 2012, Field methods.

[59]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[60]  J. Cappella,et al.  Argument Repertoire as a Reliable and Valid Measure of Opinion Quality: Electronic Dialogue During Campaign 2000 , 2002 .

[61]  D. Dillman Mail and internet surveys , 1999 .

[62]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[63]  Candice M. Daniel,et al.  Identifying Multiple Submissions in Internet Research: Preserving Data Integrity , 2008, AIDS and Behavior.

[64]  Harald Schoen,et al.  Exploring the Effects of Removing “Too Fast” Responses and Respondents from Web Surveys , 2015 .

[65]  P. Mahalanobis On the generalized distance in statistics , 1936 .

[66]  Mario Callegaro,et al.  Paradata in Web Surveys , 2013 .

[67]  Douglas E. Pine Assessing the Validity of Job Ratings: An Empirical Study of False Reporting in Task Inventories , 1995 .

[68]  P. Sniderman,et al.  CHAPTER 12: A Consistency Theory of Public Opinion and Political Choice: The Hypothesis of Menu Dependence , 2004 .

[69]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[70]  Russell H. Fazio,et al.  A practical guide to the use of response latency in social psychological research. , 1990 .

[71]  Jon A. Krosnick,et al.  A TEST OF THE FORM-RESISTANT CORRELATION HYPOTHESIS RATINGS, RANKINGS, AND THE MEASUREMENT OF VALUES , 1988 .

[72]  Floyd J. Fowler,et al.  Survey Research Methods , 1984 .

[73]  Lois A. Ritter,et al.  Conducting Online Surveys , 2011 .

[74]  R. Goldsmith Reducing Spurious Response in a Field Survey. , 1989 .