Enhancing Crowdworkers' Vigilance

This paper presents methods for improving the attention span of workers in tasks that heavily rely on their attention to the occurrence of rare events. The underlying idea in our approach is to dynamically augment the task with some dummy (artificial) events at different times throughout the task, rewarding the worker upon identifying and reporting them. The proposed approach is an alternative to the traditional approach of exclusively relying on rewarding the worker for successfully identifying the event of interest itself. We propose three methods for timing the dummy events throughout the task. Two of these methods are static and determine the timing of the dummy events at random or uniformly throughout the task. The third method is dynamic and uses the identification (or misidentification) of dummy events as a signal for the worker’s attention to the task, adjusting the rate of dummy events generation accordingly.

[1]  David Sarne,et al.  Improving comparison shopping agents' competence through selective price disclosure , 2015, Electron. Commer. Res. Appl..

[2]  Ming Yin,et al.  Bonus or Not? Learn to Reward in Crowdsourcing , 2015, IJCAI.

[3]  Avshalom Elmalech,et al.  Agent development as a strategy shaper , 2015, Autonomous Agents and Multi-Agent Systems.

[4]  Avshalom Elmalech,et al.  Problem restructuring for better decision making in recurring decision situations , 2014, Autonomous Agents and Multi-Agent Systems.

[5]  Rada Mihalcea,et al.  Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation , 2010, Mturk@HLT-NAACL.

[6]  David M. Rahman.  But Who Will Monitor the Monitor , 2012 .

[7]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[8]  Avshalom Elmalech,et al.  When Suboptimal Rules , 2015, AAAI.

[9]  Michael Wooldridge,et al.  Autonomous agents and multi-agent systems , 2014 .

[10]  Thore Graepel,et al.  Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests , 2012, AAAI.

[11]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[12]  Avshalom Elmalech,et al.  Extending Workers' Attention Span Through Dummy Events , 2016, HCOMP.

[13]  Krzysztof Z. Gajos,et al.  Curiosity Killed the Cat, but Makes Crowdwork Better , 2016, CHI.

[14]  Parham Aarabi,et al.  The art of lecturing : a practical guide to successful university lectures and business presentations , 2007 .

[15]  Athanasios V. Vasilakos,et al.  TRAC: Truthful auction for location-aware collaborative sensing in mobile crowdsourcing , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[16]  Noah A. Smith,et al.  Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2014 .

[17]  Hinrich Schütze,et al.  Active Learning with Amazon Mechanical Turk , 2011, EMNLP.

[18]  David Sarne,et al.  Enhancing comparison shopping agents through ordering and gradual information disclosure , 2017, Autonomous Agents and Multi-Agent Systems.