Preventing Satisficing in Online Surveys: A "Kapcha" to Ensure Higher Quality Data

Researchers are increasingly using online labor markets such as Amazon’s Mechanical Turk (MTurk) as a source of inexpensive data. One of the most popular tasks is answering surveys. However, without adequate controls, researchers should be concerned that respondents may fill out surveys haphazardly in the unsupervised environment of the Internet. Social scientists refer to mental shortcuts that people take as “satisficing” and this concept has been applied to how respondents take surveys. We examine the prevalence of survey satisficing on MTurk. We present a questionpresentation method, called Kapcha, which we believe reduces satisficing, thereby improving the quality of survey results. We also present an open-source platform for further survey experimentation on MTurk.

[1]  Lydia B. Chilton,et al.  Task search in a human computation market , 2010, HCOMP '10.

[2]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[3]  Stanford E. Taylor.  Eye Movements in Reading: Facts and Fallacies , 1965 .

[4]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[5]  J. Hoffman,et al.  The role of visual attention in saccadic eye movements , 1995, Perception & psychophysics.

[6]  Roger Tourangeau,et al.  Taking the Audio Out of Audio-CASI , 2009 .

[7]  C. F. Kao,et al.  The efficient assessment of need for cognition. , 1984, Journal of personality assessment.

[8]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[9]  Ananthanarayanan Parasuraman,et al.  Assessing response quality. A self‐disclosure approach to assessing response quality in mall intercept and telephone interviews , 1984 .

[10]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[11]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[12]  John Langford,et al.  CAPTCHA: Using Hard AI Problems for Security , 2003, EUROCRYPT.

[13]  J. Chall,et al.  Readability revisited : the new Dale-Chall readability formula , 1995 .

[14]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[15]  H. Simon,et al.  A Behavioral Model of Rational Choice , 1955 .

[16]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[17]  Richard H. Thaler,et al.  Mental Accounting and Consumer Choice , 1985, Mark. Sci..

[18]  Jon A. Krosnick,et al.  Satisficing in surveys: Initial evidence , 1996 .