Aggregating Crowd Opinions Using Shapley Value Regression

Crowdsourcing is becoming increasingly popular in various tasks. Aggregating answers from workers in crowdsouring has been a widely used technique for providing many applications and services. To aggregate these answers, fair evaluation of workers is important to motivate them to give high quality answers. However, it is difficult to fairly evaluate workers if their answers show a high degree of correlation. In this paper, we propose to use the Shapley value regression as a means to address this problem. The regression technique is based on ideas developed from cooperative game theory to evaluate the relative importance of explanatory variables in reducing the error. We also exploit sparseness of worker collaboration graph to effectively calculate the Shapley value, since it requires an exponential computation time to calculate the Shapley value.

[1]  Sheldon B. Akers,et al.  Binary Decision Diagrams , 1978, IEEE Transactions on Computers.

[2]  Makoto Yokoo,et al.  Robust Combinatorial Auction Protocol against False-Name Bids , 2000, AAAI/IAAI.

[3]  Shin-ichi Minato,et al.  Zero-Suppressed BDDs for Set Manipulation in Combinatorial Problems , 1993, 30th ACM/IEEE Design Automation Conference.

[4]  Aaron D. Shaw,et al.  Designing incentives for inexpert human raters , 2011, CSCW.

[5]  S. Lipovetsky,et al.  Analysis of regression in game theory approach , 2001 .

[6]  Hisashi Kashima,et al.  Accurate Integration of Crowdsourced Labels Using Workers' Self-reported Confidence Scores , 2013, IJCAI.

[7]  Hisashi Kashima,et al.  Statistical quality estimation for general crowdsourcing tasks , 2013, HCOMP.

[8]  David Maxwell Chickering,et al.  Approximating the shapley value via multi-issue decompositions , 2014, AAMAS.

[9]  L. Shapley.  A Value for n-person Games , 1988 .

[10]  L. S. Shapley,et al.  17. A Value for n-Person Games , 1953 .

[11]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[12]  Makoto Yokoo,et al.  Ability Grouping of Crowd Workers via Reward Discrimination , 2013, HCOMP.

[13]  Lei Duan,et al.  Separate or joint? Estimation of multiple labels from crowdsourced annotations , 2014, Expert Syst. Appl..

[14]  John C. Platt,et al.  Learning from the Wisdom of Crowds by Minimax Entropy , 2012, NIPS.

[15]  Takeru Inoue,et al.  Frontier-Based Search for Enumerating All Constrained Subgraphs with Compressed Representation , 2017, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[16]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.