On the Posted Pricing in Crowdsourcing: Power of Bonus

In practical crowdsourcing systems such as Amazon Mechanical Turk, posted pricing is widely used due to its simplicity, where a task requester publishes a pricing rule a priori, on which workers decide whether to accept and perform the task or not, and are often paid according to the quality of their effort. One of the key ingredients of a good posted pricing lies in how to recruit more high-quality workers with less budget, for which the following two schemes are considered: (i) personalized pricing by profiling users in terms of their quality and cost, and (ii) additional bonus payment offered for more qualified task completion. Despite their potential benefits in crowdsourced pricing, it has been under-explored how much gain each or both of personalization and bonus payment actually provides to the requester. In this paper, we study four possible combinations of posted pricing made by pricing with/without personalization and bonus. We aim at analytically quantifying when and how much such two ideas contribute to the requester's utility. To this end, we first derive the optimal personalized and common pricing schemes and analyze their computational tractability. Next, we quantify the gap in the utility between with and without bonus payment in both pricing schemes. We analytically prove that the impact of bonus is negligible significantly marginal in personalized pricing, whereas crucial in common pricing. Finally, we study the notion of Price of Agnosticity that quantifies the utility gap between personalized and common pricing policies. This implies that a complex personalized pricing with privacy concerns can be replaced by a simple common pricing with bonus. We validate our analytical findings through extensive simulations and real experiments done in Amazon Mechanical Turk, and provide additional implications that are useful in designing a pricing policy.

[1]  Leib Litman,et al.  The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk , 2014, Behavior Research Methods.

[2]  Bo Jiang,et al.  Opportunistic Flooding in Low-Duty-Cycle Wireless Sensor Networks with Unreliable Links , 2009, IEEE Transactions on Computers.

[3]  T. Klastorin On a discrete nonlinear and nonseparable knapsack problem , 1990 .

[4]  Yang Liu,et al.  Learning to Incentivize: Eliciting Effort via Output Agreement , 2016, IJCAI.

[5]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[6]  Xiaoying Gan,et al.  Incentivize Multi-Class Crowd Labeling Under Budget Constraint , 2017, IEEE Journal on Selected Areas in Communications.

[7]  Thibaut Horel Notes on Greedy Algorithms for Submodular Maximization Thibaut , 2015 .

[8]  Jie Zhang,et al.  Optimal Posted-Price Mechanism in Microtask Crowdsourcing , 2017, IJCAI.

[9]  Li-zhen Cui,et al.  Optimal Crowds Contest Model for Crowdsourcing , 2017, ICCSE'17.

[10]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[11]  Shaojie Tang,et al.  Online Pricing for Mobile Crowdsourcing with Multi-Minded Users , 2017, MobiHoc.

[12]  Zongpeng Li,et al.  Optimal Posted Prices for Online Cloud Resource Allocation , 2017, Proc. ACM Meas. Anal. Comput. Syst..

[13]  Shaojie Tang,et al.  A Budget Feasible Incentive Mechanism for Weighted Coverage Maximization in Mobile Crowdsensing , 2017, IEEE Transactions on Mobile Computing.

[14]  Balasubramanian Sivan,et al.  Optimal Crowdsourcing Contests , 2011, Encyclopedia of Algorithms.

[15]  Yu-An Sun,et al.  Monetary Interventions in Crowdsourcing Task Switching , 2014, HCOMP.

[16]  G. Dantzig Discrete-Variable Extremum Problems , 1957 .

[17]  Nihar B. Shah,et al.  Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing , 2014, J. Mach. Learn. Res..

[18]  V. Wortman Making better use of the crowd , 2017 .

[19]  Eric Balkanski,et al.  Bayesian Budget Feasibility with Posted Pricing , 2015, WWW.

[20]  Aleksandrs Slivkins,et al.  Incentivizing high quality crowdwork , 2015, SECO.

[21]  Henryk Wozniakowski,et al.  Complexity of linear programming , 1982, Oper. Res. Lett..

[22]  K. J. Ray Liu,et al.  On Cost-Effective Incentive Mechanisms in Microtask Crowdsourcing , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[23]  Boi Faltings,et al.  Incentives for Effort in Crowdsourcing Using the Peer Truth Serum , 2016, ACM Trans. Intell. Syst. Technol..

[24]  Ming Yin,et al.  Bonus or Not? Learn to Reward in Crowdsourcing , 2015, IJCAI.

[25]  Gagan Goel,et al.  Mechanism Design for Crowdsourcing: An Optimal 1-1/e Competitive Budget-Feasible Mechanism for Large Markets , 2014, 2014 IEEE 55th Annual Symposium on Foundations of Computer Science.

[26]  Milan Vojnovic,et al.  Crowdsourcing and all-pay auctions , 2009, EC '09.

[27]  Yuval Peres,et al.  Approval Voting and Incentives in Crowdsourcing , 2015, ICML.

[28]  Xiaoying Gan,et al.  Incentivize crowd labeling under budget constraint , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[29]  Mihaela van der Schaar,et al.  Reputation-based incentive protocols in crowdsourcing applications , 2011, 2012 Proceedings IEEE INFOCOM.

[30]  Huadong Ma,et al.  Collection-behavior based multi-parameter posted pricing mechanism for crowd sensing , 2013, 2014 IEEE International Conference on Communications (ICC).

[31]  Chao Liu,et al.  TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings , 2012, ICML.

[32]  Kai Han,et al.  Posted pricing for robust crowdsensing , 2016, MobiHoc.

[33]  Andreas Krause,et al.  Truthful incentives in crowdsourcing tasks using regret minimization mechanisms , 2013, WWW.

[34]  Roger B. Myerson,et al.  Optimal Auction Design , 1981, Math. Oper. Res..

[35]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[36]  Eugene L. Lawler,et al.  Fast approximation algorithms for knapsack problems , 1977, 18th Annual Symposium on Foundations of Computer Science (sfcs 1977).

[37]  Ruggiero Cavallo,et al.  Efficient crowdsourcing contests , 2012, AAMAS.

[38]  Christopher G. Harris You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks , 2011 .

[39]  Philippe Michelon,et al.  0-1 Quadratic Knapsack Problems: An Exact Approach Based on a t-Linearization , 2012, SIAM J. Optim..

[40]  Christos H. Papadimitriou,et al.  Free-riding and whitewashing in peer-to-peer systems , 2004, IEEE Journal on Selected Areas in Communications.

[41]  Jinwoo Shin,et al.  Optimality of Belief Propagation for Crowdsourced Classification , 2016, ICML.

[42]  Yaron Singer,et al.  Pricing mechanisms for crowdsourcing markets , 2013, WWW.

[43]  Oscar H. Ibarra,et al.  Fast Approximation Algorithms for the Knapsack and Sum of Subset Problems , 1975, JACM.

[44]  Maxim Sviridenko,et al.  A note on maximizing a submodular set function subject to a knapsack constraint , 2004, Oper. Res. Lett..

[45]  Teck-Hua Ho,et al.  Knowledge Market Design: A Field Experiment at Google Answers , 2010 .