Bonus or Not? Learn to Reward in Crowdsourcing

Recent work has shown that the quality of work produced in a crowdsourcing working session can be influenced by the presence of performance-contingent financial incentives, such as bonuses for exceptional performance, in the session. We take an algorithmic approach to decide when to offer bonuses in a working session to improve the overall utility that a requester derives from the session. Specifically, we propose and train an input-output hidden Markov model to learn the impact of bonuses on work quality and then use this model to dynamically decide whether to offer a bonus on each task in a working session to maximize a requester's utility. Experiments on Amazon Mechanical Turk show that our approach leads to higher utility for the requester than fixed and random bonus schemes do. Simulations on synthesized data sets further demonstrate the robustness of our approach against different worker population and worker behavior in improving requester utility.

[1]  Krzysztof Z. Gajos,et al.  Toward automatic task design: a progress report , 2010, HCOMP '10.

[2]  Aditya G. Parameswaran,et al.  Finish Them!: Pricing Algorithms for Human Computation , 2014, Proc. VLDB Endow..

[3]  Janet L. Yellen,et al.  The Fair Wage-Effort Hypothesis and Unemployment , 1990 .

[4]  Ioana Popescu,et al.  Dynamic Pricing Strategies with Reference Effects , 2007, Oper. Res..

[5]  David Baker,et al.  Algorithm discovery by protein folding game players , 2011, Proceedings of the National Academy of Sciences.

[6]  Yaron Singer,et al.  Pricing mechanisms for crowdsourcing markets , 2013, WWW.

[7]  Gerardo Hermosillo,et al.  Supervised learning from multiple experts: whom to trust when everyone lies a bit , 2009, ICML '09.

[8]  Yu-An Sun,et al.  Task Sequence Design: Evidence on Price and Difficulty , 2013, HCOMP.

[9]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[10]  J. Tobin,et al.  ハンセンと公共政策 (ハンセンとその業績(The Quarterly Journal of Economics,1976年2月号)) , 1976 .

[11]  Yu-An Sun,et al.  The Effects of Performance-Contingent Financial Incentives in Online Labor Markets , 2013, AAAI.

[12]  Xi Chen,et al.  Optimistic Knowledge Gradient Policy for Optimal Budget Allocation in Crowdsourcing , 2013, ICML.

[13]  L. Goddard,et al.  Operations Research (OR) , 2007 .

[14]  Kate Saenko,et al.  Proceedings of the 27th AAAI Conference on Artificial Intelligence, AAAI 2013 , 2013, AAAI 2013.

[15]  原田 秀逸.  私の computer 環境 , 1998 .

[16]  Yoshua Bengio,et al.  Input-output HMMs for sequence processing , 1996, IEEE Trans. Neural Networks.

[17]  Leslie Pack Kaelbling,et al.  Learning Policies for Partially Observable Environments: Scaling Up , 1997, ICML.

[18]  Virgílio A. F. Almeida,et al.  Proceedings of the 22nd international conference on World Wide Web , 2013, WWW 2013.

[19]  Andreas Krause,et al.  Truthful incentives in crowdsourcing tasks using regret minimization mechanisms , 2013, WWW.

[20]  O. Bagasra,et al.  Proceedings of the National Academy of Sciences , 1914, Science.

[21]  Peng Dai,et al.  Decision-Theoretic Control of Crowd-Sourced Workflows , 2010, AAAI.

[22]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[23]  Aleksandrs Slivkins,et al.  Incentivizing high quality crowdwork , 2015, SECO.

[24]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[25]  Eric Horvitz,et al.  Combining human and machine intelligence in large-scale crowdsourcing , 2012, AAMAS.

[26]  Christopher G. Harris.  You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks , 2011 .

[27]  Aleksandrs Slivkins,et al.  Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problems , 2016, J. Artif. Intell. Res..

[28]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[29]  Panagiotis G. Ipeirotis,et al.  Quality-Based Pricing for Crowdsourced Workers , 2013 .

[30]  Pietro Perona,et al.  The Caltech-UCSD Birds-200-2011 Dataset , 2011 .

[31]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[32]  Mausam,et al.  Dynamically Switching between Synergistic Workflows for Crowdsourcing , 2012, AAAI.

[33]  Andrew J. Viterbi,et al.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm , 1967, IEEE Trans. Inf. Theory.

[34]  S. Gorman,et al.  Volunteered Geographic Information and Crowdsourcing Disaster Relief: A Case Study of the Haitian Earthquake , 2010 .

[35]  Robert M Thrall,et al.  Mathematics of Operations Research. , 1978 .

[36]  Neil Savage,et al.  Gaining wisdom from crowds , 2012, Commun. ACM.

[37]  Daniel Gooch,et al.  Communications of the ACM , 2011, XRDS.

[38]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..