Challenges and Opportunities for Trust Management in Crowdsourcing

Crowd sourcing (CS) systems offer a new way for businesses and individuals to leverage on the power of mass collaboration to accomplish complex tasks in a divide-and-conquer manner. In existing CS systems, no facility has been provided for analyzing the trustworthiness of workers and providing decision support for allocating tasks to workers, which leads to high dependency of the quality of work on the behavior of workers in CS systems as shown in this paper. To address this problem, trust management mechanisms are urgently needed. Traditional trust management techniques are focused on identifying the most trustworthy service providers (SPs) as accurately as possible. Little thoughts were given to the question of how to utilize these SPs due to two common assumptions: 1) an SP can serve an unlimited number of requests in one time unit, and 2) a service consumer (SC) only needs to select one SP for interaction to complete a task. However, in CS systems, these two assumptions are no longer valid. Thus, existing models cannot be directly used for trust management in CS systems. This paper takes the first step towards a systematic investigation of trust management in CS systems by extending existing trust management models for CS trust management and conducting extensive experiments to study and analyze the performance of various trust management models in crowd sourcing. In this paper, the following key contributions are made. We 1) propose extensions to existing trust management approaches to enable them to operate in CS systems, 2) design a simulation test-bed based on the system characteristics of Amazon's Mechanical Turk (AMT) to make evaluation close to practical CS systems, 3) discuss the effect of incorporating trust management into CS system on the overall social welfare, and 4) identify the challenges and opportunities for future trust management research in CS systems.

[1]  Mihaela Ulieru,et al.  The State of the Art in Trust and Reputation Systems: A Framework for Comparison , 2010, J. Theor. Appl. Electron. Commer. Res..

[2]  Nicholas R. Jennings,et al.  A Hierarchical Bayesian Trust Model based on Reputation and Group Behaviour , 2008 .

[3]  Bill Tomlinson,et al.  Who are the Turkers? Worker Demographics in Amazon Mechanical Turk , 2009 .

[4]  Chunyan Miao,et al.  Credibility: How Agents Can Handle Unfair Third-Party Testimonies in Computational Trust Models , 2010, IEEE Transactions on Knowledge and Data Engineering.

[5]  Chunyan Miao,et al.  A Survey of Trust and Reputation Management Systems in Wireless Communications , 2010, Proceedings of the IEEE.

[6]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[7]  Mark Hoogendoorn,et al.  Exploration and Exploitation in Adaptive Trust-Based Decision Making in Dynamic Environments , 2010, 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.

[8]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[9]  Daniel G. Goldstein,et al.  Honesty in an Online Labor Market , 2011, Human Computation.

[10]  Panagiotis G. Ipeirotis.  Analyzing the Amazon Mechanical Turk marketplace , 2010, XRDS.

[11]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[12]  Nicholas R. Jennings,et al.  Coping with inaccurate reputation sources: experimental analysis of a probabilistic trust model , 2005, AAMAS '05.

[13]  David G. Rand,et al.  The online laboratory: conducting experiments in a real labor market , 2010, ArXiv.

[14]  Raj Jain,et al.  A Quantitative Measure Of Fairness And Discrimination For Resource Allocation In Shared Computer Systems , 1998, ArXiv.

[15]  Chunyan Miao,et al.  Trust-based web service selection in virtual communities , 2011, Web Intell. Agent Syst..

[16]  Dídac Busquets,et al.  Strategies for Exploiting Trust Models in Competitive Multi-Agent Systems , 2009, MATES.

[17]  Audun Jøsang,et al.  A survey of trust and reputation systems for online service provision , 2007, Decis. Support Syst..

[18]  Audun Jøsang,et al.  AIS Electronic Library (AISeL) , 2017 .

[19]  Duncan J. Watts,et al.  Cooperation and Contagion in Web-Based, Networked Public Goods Experiments , 2010, SECO.

[20]  Munindar P. Singh,et al.  Detecting deception in reputation management , 2003, AAMAS '03.