Crowdsourcing GO: Effect of Worker Situation on Mobile Crowdsourcing Performance

The increasing popularity of mobile crowdsourcing platforms has enabled crowd workers to accept jobs wherever/whenever they are, and also provides opportunity for task requesters to order time/location specific tasks to workers. Since workers on mobile platforms are working on the go, the situation of the workers is expected to influence their performance. However, the effects of mobile worker situations to task performance is an uninvestigated area. In this paper, our research question is, "do worker situations affect task completion, price and quality on mobile crowdsourcing platforms?" We draw on economics and psychology research to examine whether worker situations such as busyness, fatigue and presence of companions affect their performance. Our three-week between-subjects field experiment revealed that worker busyness caused 30.1% relative decrease of task completion rate. Mean accepted task price increased by 7.6% when workers are with companions. Worker fatigue caused 37.4% relative decrease of task quality.

[1]  Michael S. Bernstein,et al.  Twitch crowdsourcing: crowd contributions in short bursts of time , 2014, CHI.

[2]  Maarten A. S. Boksem,et al.  Effects of mental fatigue on attention: an ERP study. , 2005, Brain research. Cognitive brain research.

[3]  Jaime Teevan,et al.  Chain Reactions: The Impact of Order on Microtask Chains , 2016, CHI.

[4]  Deepak Ganesan,et al.  Labor dynamics in a mobile micro-task market , 2013, CHI.

[5]  H. Marmorstein,et al.  The Value of Time Spent in Price-Comparison Shopping: Survey and Experimental Evidence , 1992 .

[6]  Jorge Gonçalves,et al.  Situated crowdsourcing using a market model , 2014, UIST.

[7]  Aniket Kittur,et al.  A comparison of social, learning, and financial strategies on crowd engagement and output quality , 2014, CSCW.

[8]  M. B. Johnson.  TRAVEL TIME AND THE PRICE OF LEISURE , 1966 .

[9]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[10]  Phuoc Tran-Gia,et al.  Quantification of YouTube QoE via Crowdsourcing , 2011, 2011 IEEE International Symposium on Multimedia.

[11]  Maarten A. S. Boksem,et al.  Mental fatigue, motivation and action monitoring , 2006, Biological Psychology.

[12]  Enrico Gregori,et al.  Smartphone-based crowdsourcing for network monitoring: Opportunities, challenges, and a case study , 2014, IEEE Communications Magazine.

[13]  A. Muzet,et al.  Correlation between driving errors and vigilance level: influence of the driver's age , 2004, Physiology & Behavior.

[14]  Shalini Misra,et al.  The iPhone Effect , 2016 .

[15]  Robert E. Kraut,et al.  Experiment 1 : Motivating Conversational Contributions Through Group Homogeneity and Individual Uniqueness , 2010 .

[16]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[17]  Michael S. Bernstein,et al.  Embracing Error to Enable Rapid Crowdsourcing , 2016, CHI.

[18]  Natasha M. Maurits,et al.  Mental Fatigue Affects Visual Selective Attention , 2012, PloS one.

[19]  Mor Naaman,et al.  The motivations and experiences of the on-demand mobile workforce , 2014, CSCW.

[20]  Mary Czerwinski,et al.  Instant Messaging: Effects of Relevance and Timing , 2000 .

[21]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[22]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[23]  Aaron D. Shaw,et al.  Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India , 2012, CHI.

[24]  Brian P. Bailey,et al.  If not now, when?: the effects of interruption at different moments within task execution , 2004, CHI.

[25]  Geri Gay,et al.  The laptop and the lecture: The effects of multitasking in learning environments , 2003, J. Comput. High. Educ..

[26]  Víctor M. González,et al.  Multitasking and interruptions: a SIG on bridging the gap between research on the micro and macro worlds , 2012, CHI Extended Abstracts.

[27]  Alireza Sahami Shirazi,et al.  Location-based crowdsourcing: extending crowdsourcing to the real world , 2010, NordiCHI.

[28]  Pradeep Sopory,et al.  Effects of Background Television on Phonological and Visuo-Spatial Working Memory , 1997 .

[29]  Demetrios Zeinalipour-Yazti,et al.  Crowdsourcing with Smartphones , 2012, IEEE Internet Computing.

[30]  Reuben Gronau,et al.  The Intrafamily Allocation of Time: The Value of the Housewives' Time , 1973 .

[31]  Krzysztof Z. Gajos,et al.  Curiosity Killed the Cat, but Makes Crowdwork Better , 2016, CHI.

[32]  C. Zick,et al.  Alternative Strategies For Pricing Home Work Time , 1983 .

[33]  John Le,et al.  Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution , 2010 .

[34]  Brian P. Bailey,et al.  On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state , 2006, Comput. Hum. Behav..

[35]  J. G. Kreifeldt,et al.  Interruption as a test of the user-computer interface , 1981 .

[36]  Tadashi Okoshi,et al.  Attelia: Reducing user's cognitive load due to interruptive notifications on smart phones , 2015, 2015 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[37]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[38]  Joyce Ho,et al.  Using context-aware computing to reduce the perceived burden of interruptions from mobile devices , 2005, CHI.

[39]  Michael S. Bernstein,et al.  Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity , 2016, CHI.

[40]  Heli Väätäjä,et al.  Crowdsourced news reporting: supporting news content creation with mobile phones , 2011, Mobile HCI.

[41]  Tatsuo Nakajima,et al.  Drawing on mobile crowds via social media , 2012, Multimedia Systems.

[42]  Steven A. Lippman,et al.  The Economics of Job Search: A Survey: Part I , 1976 .

[43]  C. M. Koolstra,et al.  Distraction Effects of Background Soap Operas on Homework Performance: An experimental study enriched with observational data , 2003 .

[44]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[45]  Eric Horvitz,et al.  Notifications and awareness: a field study of alert usage and preferences , 2010, CSCW '10.

[46]  David Hausheer,et al.  Demonstration of NetworkCoverage – A Mobile Network Performance Measurement App , 2015 .

[47]  Michael S. Bernstein,et al.  Expert crowdsourcing with flash teams , 2014, UIST.

[48]  Edward Cutrell,et al.  mClerk: enabling mobile crowdsourcing in developing regions , 2012, CHI.

[49]  S. Lippman,et al.  THE ECONOMICS OF JOB SEARCH: A SURVEY* , 1976 .

[50]  Joseph S. Valacich,et al.  The Influence of Task Interruption on Individual Decision Making: An Information Overload Perspective , 1999 .

[51]  F. Zijlstra,et al.  Temporal factors in mental work: Effects of interrupted activities , 1999 .

[52]  J. Johansson,et al.  Determinants of Search for Lower Prices: An Empirical Assessment of the Economics of Information Theory , 1978 .

[53]  Georg Groh,et al.  Conversational context helps improve mobile notification management , 2016, MobileHCI.

[54]  Mark W. Newman,et al.  A field study comparing approaches to collecting annotated activity data in real-world settings , 2015, UbiComp.

[55]  Russell Roberts.  Getting the Most Out of Life: The Concept of Opportunity Cost , 2007 .