Improving learning through achievement priming in crowdsourced information finding microtasks

Crowdsourcing has become an increasingly popular means to acquire human input on demand. Microtask crowdsourcing market-places facilitate the access to millions of people (called workers) who are willing to participate in tasks in return for monetary rewards or other forms of compensation. This paradigm presents a unique learning context where workers have to learn to complete tasks on-the-fly by applying their learning immediately through the course of tasks. However, most workers typically dropout early in large batches of tasks, depriving themselves of the opportunity to learn on-the-fly through the course of batch completion. By doing so workers squander a potential chance at improving their performance and completing tasks effectively. In this paper, we propose a novel method to engage and retain workers, to improve their learning in crowdsourced information finding tasks by using achievement priming. Through rigorous experimental findings, we show that it is possible to retain workers in long batches of tasks by triggering their inherent motivation to achieve and excel. As a consequence of increased worker retention, we find that workers learn to perform more effectively, depicting relatively more stable accuracy and lower task completion times in comparison to workers who drop out early.

[1]  Elizabeth Gerber,et al.  Affect and Creative Performance on Crowdsourcing Platforms , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[2]  D. Mcclelland,et al.  TOWARD A THEORY OF MOTIVE ACQUISITION. , 1965, The American psychologist.

[3]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[4]  Bipin Indurkhya,et al.  Effect of Task Presentation on the Performance of Crowd Workers - A Cognitive Study , 2013, HCOMP.

[5]  Stefan Engeser,et al.  Nonconscious Activation of Achievement Goals Moderated by Word Class and the Explicit Achievement Motive , 2009 .

[6]  M. Six Silberman,et al.  Turkopticon: interrupting worker invisibility in amazon mechanical turk , 2013, CHI.

[7]  Ricardo Kawase,et al.  Training Workers for Improving Performance in Crowdsourcing Microtasks , 2015, EC-TEL.

[8]  J. Bargh,et al.  The automated will: nonconscious activation and pursuit of behavioral goals. , 2001, Journal of personality and social psychology.

[9]  Stefan Dietze,et al.  A taxonomy of microtasks on the web , 2014, HT.

[10]  Judith Redi,et al.  Modeling Task Complexity in Crowdsourcing , 2016, HCOMP.

[11]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[12]  Eric Horvitz,et al.  Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing , 2013, HCOMP.

[13]  Peng Dai,et al.  Inserting Micro-Breaks into Crowdsourcing Workflows , 2013, HCOMP.

[14]  S. Fiske Social cognition and social perception. , 1993, Annual review of psychology.

[15]  Gianluca Demartini,et al.  Scaling-Up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement , 2014, HCOMP.

[16]  Jacki O'Neill,et al.  Turk-Life in India , 2014, GROUP.

[17]  Derek Ruths,et al.  How One Microtask Affects Another , 2016, CHI.

[18]  D. Albarracín,et al.  The effects of chronic achievement motivation and achievement primes on the activation of achievement and fun goals. , 2009, Journal of personality and social psychology.

[19]  Elena Paslaru Bontas Simperl,et al.  Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives , 2015, WWW.

[20]  Steven Franconeri,et al.  Influencing visual judgment through affective priming , 2013, CHI.

[21]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[22]  Elizabeth Gerber,et al.  Affective computational priming and creativity , 2011, CHI.

[23]  Panagiotis G. Ipeirotis,et al.  The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk , 2015, WWW.

[24]  Walter S. Lasecki,et al.  Crowd Memory: Learning in the Collective , 2012, ArXiv.

[25]  Peng Dai,et al.  And Now for Something Completely Different: Improving Crowdsourcing Workflows with Micro-Diversions , 2015, CSCW.

[26]  Lane Harrison,et al.  Exploring the impact of emotion on visual judgement , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[27]  Daren C. Brabham MOVING THE CROWD AT THREADLESS , 2010 .

[28]  Stefan Dietze,et al.  Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys , 2015, CHI.

[29]  Daren C. Brabham Moving the crowd at iStockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application , 2008, First Monday.

[30]  Elizabeth Gerber,et al.  A pilot study of using crowds in the classroom , 2013, CHI.

[31]  Richard Lynn,et al.  A multifactorial approach to achievement motivation: The development of a comprehensive measure , 1989, Journal of Occupational Psychology.

[32]  Elizabeth Gerber,et al.  Priming for Better Performance in Microtask Crowdsourcing Environments , 2012, IEEE Internet Computing.

[33]  Patrick Jermann,et al.  Task Performance vs. Learning Outcomes: A Study of a Tangible User Interface in the Classroom , 2010, EC-TEL.

[34]  Gary P. Latham,et al.  The effect of primed goals on employee performance: Implications for human resource management , 2011 .

[35]  Jaime Teevan,et al.  Chain Reactions: The Impact of Order on Microtask Chains , 2016, CHI.