Understanding Crowdsourcing Systems from a Multiagent Perspective and Approach

Crowdsourcing has recently been significantly explored. Although related surveys have been conducted regarding this subject, each has mainly consisted of a review of a single aspect of crowdsourcing systems or on the application of crowdsourcing in a specific application domain. A crowdsourcing system is a comprehensive set of multiple entities, including various elements and processes. Multiagent computing has already been widely envisioned as a powerful paradigm for modeling autonomous multi-entity systems with adaptation to dynamic environments. Therefore, this article presents a novel multiagent perspective and approach to understanding crowdsourcing systems, which can be used to correlate the research on crowdsourcing and multiagent systems and inspire possible interdisciplinary research between the two areas. This article mainly discusses the following two aspects: (1) The multiagent perspective can be used for conducting a comprehensive survey on the state of the art of crowdsourcing, and (2) the multiagent approach can bring about concrete enhancements for crowdsourcing technology and inspire future research directions that enable crowdsourcing research to overcome the typical challenges in crowdsourcing technology. Finally, this article discusses the advantages and disadvantages of the multiagent perspective by comparing it with two other popular perspectives on crowdsourcing: the business perspective and the technical perspective.

[1]  Mausam,et al.  Dynamically Switching between Synergistic Workflows for Crowdsourcing , 2012, AAAI.

[2]  Nicholas R. Jennings,et al.  Efficient Crowdsourcing of Unknown Experts using Multi-Armed Bandits , 2012, ECAI.

[3]  Mark Harman,et al.  A survey of the use of crowdsourcing in software engineering , 2017, J. Syst. Softw..

[4]  Nuno Silva,et al.  A survey of task-oriented crowdsourcing , 2015, Artificial Intelligence Review.

[5]  Reza Olfati-Saber,et al.  Flocking for multi-agent dynamic systems: algorithms and theory , 2006, IEEE Transactions on Automatic Control.

[6]  P. Tichy,et al.  Multiagent technology for fault tolerance and flexible control , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[7]  Chunyan Miao,et al.  A reputation-aware decision-making approach for improving the efficiency of crowdsourcing systems , 2013, AAMAS.

[8]  Gagan Goel,et al.  Mechanism Design for Crowdsourcing Markets with Heterogeneous Tasks , 2014, HCOMP.

[9]  Bo An,et al.  Context-Aware Reliable Crowdsourcing in Social Networks , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[10]  Xi Fang,et al.  Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing , 2012, Mobicom '12.

[11]  Jon Chamberlain,et al.  Groupsourcing: Distributed Problem Solving Using Social Networks , 2014, HCOMP.

[12]  Devavrat Shah,et al.  Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems , 2011, Oper. Res..

[13]  Yichuan Jiang,et al.  The Rich Get Richer: Preferential Attachment in the Task Allocation of Cooperative Networked Multiagent Systems With Resource Caching , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[14]  Sepehr Assadi,et al.  Online Assignment of Heterogeneous Tasks in Crowdsourcing Markets , 2015, HCOMP.

[15]  Nicholas R. Jennings,et al.  An integrated trust and reputation model for open multi-agent systems , 2006, Autonomous Agents and Multi-Agent Systems.

[16]  Anthony Stentz,et al.  Market-based Multirobot Coordination for Complex Tasks , 2006, Int. J. Robotics Res..

[17]  Nicholas R. Jennings,et al.  Efficient crowdsourcing of unknown experts using bounded multi-armed bandits , 2014, Artif. Intell..

[18]  Nicholas R. Jennings,et al.  Trust-based fusion of untrustworthy information in crowdsourcing applications , 2013, AAMAS.

[19]  Hisashi Kashima,et al.  Leveraging non-expert crowdsourcing workers for improper task detection in crowdsourcing marketplaces , 2014, Expert Syst. Appl..

[20]  Peng Dai,et al.  POMDP-based control of workflows for crowdsourcing , 2013, Artif. Intell..

[21]  Ari Kobren,et al.  Getting More for Less: Optimized Crowdsourcing with Dynamic Tasks and Goals , 2015, WWW.

[22]  Milan Vojnovic,et al.  Crowdsourcing and all-pay auctions , 2009, EC '09.

[23]  Kwong-Sak Leung,et al.  A Survey of Crowdsourcing Systems , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[24]  Victor R. Lesser,et al.  Cooperative Multiagent Systems: A Personal View of the State of the Art , 1999, IEEE Trans. Knowl. Data Eng..

[25]  WeynsDanny,et al.  The MACODO middleware for context-driven dynamic agent organizations , 2010 .

[26]  Jacques Ferber,et al.  A meta-model for the analysis and design of organizations in multi-agent systems , 1998, Proceedings International Conference on Multi Agent Systems (Cat. No.98EX160).

[27]  Julie A. Adams,et al.  Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence , 2001, AI Mag..

[28]  Adil Baykasoglu,et al.  A multi-agent based approach to dynamic scheduling with flexible processing capabilities , 2017, J. Intell. Manuf..

[29]  PANOS IPEIROTIS,et al.  Composing and Analyzing Crowdsourcing Workflows , 2014 .

[30]  Rajkumar Buyya,et al.  A Taxonomy of Workflow Management Systems for Grid Computing , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[31]  Luca Becchetti,et al.  The Importance of Being Expert: Efficient Max-Finding in Crowdsourcing , 2015, SIGMOD Conference.

[32]  Hengrun Zhang,et al.  A Survey on Security, Privacy, and Trust in Mobile Crowdsourcing , 2018, IEEE Internet of Things Journal.

[33]  Robbie T. Nakatsu,et al.  A taxonomy of crowdsourcing based on task complexity , 2014, J. Inf. Sci..

[34]  Gita Sukthankar,et al.  AmalgaCloud: Social Network Adaptation for Human and Computational Agent Team Formation , 2012 .

[35]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[36]  Andreas Krause,et al.  Building Collaborative Teams: A Local Navigation-based Approach , 2014 .

[37]  Yichuan Jiang,et al.  Understanding Social Networks From a Multiagent Perspective , 2014, IEEE Transactions on Parallel and Distributed Systems.

[38]  Y. Narahari,et al.  A Multiarmed Bandit Incentive Mechanism for Crowdsourcing Demand Response in Smart Grids , 2014, AAAI.

[39]  Bin Bi,et al.  Iterative Learning for Reliable Crowdsourcing Systems , 2012 .

[40]  Aijun An,et al.  Efficient Bi-objective Team Formation in Social Networks , 2012, ECML/PKDD.

[41]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[42]  Franco Zambonelli.  Pervasive urban crowdsourcing: Visions and challenges , 2011, 2011 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[43]  Tianqing Zhu,et al.  Reward-based spatial crowdsourcing with differential privacy preservation , 2017, Enterp. Inf. Syst..

[44]  Sihem Amer-Yahia,et al.  Crowds, not Drones: Modeling Human Factors in Interactive Crowdsourcing , 2013, DBCrowd.

[45]  Gang Yang,et al.  Structure-based control of complex networks with nonlinear dynamics , 2016, Proceedings of the National Academy of Sciences.

[46]  Theodoros Lappas,et al.  Finding a team of experts in social networks , 2009, KDD.

[47]  Jaime G. Carbonell,et al.  Collaborative workflow for crowdsourcing translation , 2012, CSCW.

[48]  Mihai Georgescu,et al.  Aggregation of Crowdsourced Labels Based on Worker History , 2014, WIMS '14.

[49]  Samik Datta,et al.  Capacitated team formation problem on social networks , 2012, KDD.

[50]  Klemens Böhm,et al.  High-throughput crowdsourcing mechanisms for complex tasks , 2013, Social Network Analysis and Mining.

[51]  Dídac Busquets,et al.  Electronic Social Capital for Self-Organising Multi-Agent Systems , 2017, ACM Trans. Auton. Adapt. Syst..

[52]  Jean C. Walrand,et al.  Incentive mechanisms for smartphone collaboration in data acquisition and distributed computing , 2012, 2012 Proceedings IEEE INFOCOM.

[53]  Gang Feng,et al.  Output Consensus of Heterogeneous Linear Multi-Agent Systems by Distributed Event-Triggered/Self-Triggered Strategy , 2017, IEEE Transactions on Cybernetics.

[54]  Mark S. Ackerman,et al.  Crowdsourcing and knowledge sharing: strategic user behavior on taskcn , 2008, EC '08.

[55]  Xiang-Yang Li,et al.  How to crowdsource tasks truthfully without sacrificing utility: Online incentive mechanisms with budget constraint , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[56]  Wanyuan Wang,et al.  Task Allocation for Undependable Multiagent Systems in Social Networks , 2013, IEEE Transactions on Parallel and Distributed Systems.

[57]  Ming Yin,et al.  Bonus or Not? Learn to Reward in Crowdsourcing , 2015, IJCAI.

[58]  Jonathan Lee,et al.  A high-level Petri nets-based approach to verifying task structures , 2002 .

[59]  Devavrat Shah,et al.  Budget-optimal crowdsourcing using low-rank matrix approximations , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[60]  Davide Martinenghi,et al.  A Framework for Crowdsourced Multimedia Processing and Querying , 2012, CrowdSearch.

[61]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[62]  Jiming Liu,et al.  Agent-based load balancing on homogeneous minigrids: macroscopic modeling and characterization , 2005, IEEE Transactions on Parallel and Distributed Systems.

[63]  Nicholas R. Jennings,et al.  Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks , 2013, AAMAS.

[64]  Dimos V. Dimarogonas,et al.  Task and Motion Coordination for Heterogeneous Multiagent Systems With Loosely Coupled Local Tasks , 2017, IEEE Transactions on Automation Science and Engineering.

[65]  Jing Hu,et al.  Decision Making of Networked Multiagent Systems for Interaction Structures , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[66]  Shigeo Matsubara,et al.  Efficient Task Decomposition in Crowdsourcing , 2014, PRIMA.

[67]  Boi Faltings,et al.  Incentive Schemes for Participatory Sensing , 2015, AAMAS.

[68]  Wouter Joosen,et al.  The MACODO middleware for context-driven dynamic agent organizations , 2010, TAAS.

[69]  Sarvapali D. Ramchurn,et al.  BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees , 2014, AAMAS.

[70]  Victor R. Lesser,et al.  Task Interdependencies in Design-to-time Real-time Scheduling , 1993, AAAI.

[71]  Eric Horvitz,et al.  Incentives for truthful reporting in crowdsourcing , 2012, AAMAS.

[72]  Marie desJardins,et al.  Local strategy learning in networked multi-agent team formation , 2006, Autonomous Agents and Multi-Agent Systems.

[73]  Chien-Ju Ho,et al.  Online Task Assignment in Crowdsourcing Markets , 2012, AAAI.

[74]  Guangyu Zou,et al.  An agent-based model for crowdsourcing systems , 2014, Proceedings of the Winter Simulation Conference 2014.

[75]  Panagiotis G. Ipeirotis,et al.  Have you done anything like that?: predicting performance using inter-category reputation , 2013, WSDM.

[76]  Fan Ye,et al.  Mobile crowdsensing: current state and future challenges , 2011, IEEE Communications Magazine.

[77]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2015, Commun. ACM.

[78]  Jiming Chen,et al.  Toward optimal allocation of location dependent tasks in crowdsensing , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[79]  Schahram Dustdar,et al.  Auction-based crowdsourcing supporting skill management , 2013, Inf. Syst..

[80]  David C. Parkes,et al.  Optimal Coordinated Planning Amongst Self-Interested Agents with Private State , 2006, UAI.

[81]  Agostino Poggi,et al.  Agent-based interpretations of classic network models , 2013, Comput. Math. Organ. Theory.

[82]  Miguel A. Labrador,et al.  A location-based incentive mechanism for participatory sensing systems with budget constraints , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications.

[83]  Thomas Kohler,et al.  Crowdsourcing business models that last , 2017 .

[84]  Ge Yu,et al.  A Weighted Aggregation Rule in Crowdsourcing Systems for High Result Accuracy , 2014, 2014 IEEE 12th International Conference on Dependable, Autonomic and Secure Computing.

[85]  Elizabeth Gerber,et al.  Using Anonymity and Communal Efforts to Improve Quality of Crowdsourced Feedback , 2015, HCOMP.

[86]  Luca de Alfaro,et al.  Reliable Aggregation of Boolean Crowdsourced Tasks , 2015, HCOMP.

[87]  Lars Hetmank,et al.  Components and Functions of Crowdsourcing Systems - A Systematic Literature Review , 2013, Wirtschaftsinformatik.

[88]  Sarit Kraus,et al.  Methods for Task Allocation via Agent Coalition Formation , 1998, Artif. Intell..

[89]  Sarvapali D. Ramchurn,et al.  Trust-Based Mechanisms for Robust and Efficient Task Allocation in the Presence of Execution Uncertainty , 2009, J. Artif. Intell. Res..

[90]  Eli Blevis,et al.  A survey of crowdsourcing as a means of collaboration and the implications of crowdsourcing for interaction design , 2011, 2011 International Conference on Collaboration Technologies and Systems (CTS).

[91]  Pramod K. Varshney,et al.  Multi-Object Classification via Crowdsourcing With a Reject Option , 2016, IEEE Transactions on Signal Processing.

[92]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[93]  Sihem Amer-Yahia,et al.  Task Assignment Optimization in Collaborative Crowdsourcing , 2015, 2015 IEEE International Conference on Data Mining.

[94]  Hisashi Kashima,et al.  Crowdsourcing Quality Control for Item Ordering Tasks , 2013, HCOMP.

[95]  Joon Hyub Lee,et al.  Crowd vs. crowd: large-scale cooperative design through open team competition , 2013, CSCW.

[96]  Nidhi Kalra,et al.  Market-Based Multirobot Coordination: A Survey and Analysis , 2006, Proceedings of the IEEE.

[97]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[98]  Reynold Cheng,et al.  Optimizing plurality for human intelligence tasks , 2013, CIKM.

[99]  Marc Cavazza,et al.  Learning Behaviors in Agents Systems with Interactive Dynamic Influence Diagrams , 2015, IJCAI.

[100]  Athanasios V. Vasilakos,et al.  A Survey of Self-Organization Mechanisms in Multiagent Systems , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[101]  Yichuan Jiang,et al.  A Survey of Task Allocation and Load Balancing in Distributed Systems , 2016, IEEE Transactions on Parallel and Distributed Systems.

[102]  Toru Ishida,et al.  Paper Special Section on Knowledge-based Software Engineering Coordination of Local Process Views in Interorganizational Business Process , 2022 .

[103]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[104]  Yunpeng Li,et al.  Reliable Task Allocation with Load Balancing in Multiplex Networks , 2015, TAAS.

[105]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[106]  Archan Misra,et al.  Multi-Agent Task Assignment for Mobile Crowdsourcing under Trajectory Uncertainties , 2015, AAMAS.

[107]  Hongzhi Wang,et al.  Brief survey of crowdsourcing for data mining , 2014, Expert Syst. Appl..

[108]  Abbas Heydarnoori,et al.  CrowdSummarizer: Automated Generation of Code Summaries for Java Programs through Crowdsourcing , 2017, IEEE Software.

[109]  Anthony T. Chronopoulos,et al.  Algorithmic mechanism design for load balancing in distributed systems , 2002, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[110]  Stefan Siersdorfer,et al.  Groupsourcing: Team Competition Designs for Crowdsourcing , 2015, WWW.

[111]  Per Ola Kristensson,et al.  Crowdsourcing a HIT: Measuring Workers' Pre-Task Interactions on Microtask Markets , 2013, HCOMP.

[112]  Alessandro Bozzon,et al.  Choosing the right crowd: expert finding in social networks , 2013, EDBT '13.

[113]  Jiuchuan Jiang,et al.  Contextual Resource Negotiation-Based Task Allocation and Load Balancing in Complex Software Systems , 2009, IEEE Transactions on Parallel and Distributed Systems.

[114]  Bastin Tony Roy Savarimuthu,et al.  Monitoring and controlling of a multi - agent based workflow system , 2004, ACSW.

[115]  Anand Kulkarni,et al.  Wish: Amplifying Creative Ability with Expert Crowds , 2014, HCOMP.

[116]  Stéphane Bressan,et al.  An efficient and truthful pricing mechanism for team formation in crowdsourcing markets , 2015, 2015 IEEE International Conference on Communications (ICC).

[117]  Dorian Kodelja,et al.  Multiagent cooperation and competition with deep reinforcement learning , 2015, PloS one.

[118]  Tridib Mukherjee,et al.  PISCES: Participatory Incentive Strategies for Effective Community Engagement in Smart Cities , 2015, HCOMP.

[119]  Sarvapali D. Ramchurn,et al.  Trust in multi-agent systems , 2004, The Knowledge Engineering Review.

[120]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[121]  Mihaela van der Schaar,et al.  Reputation-based incentive protocols in crowdsourcing applications , 2011, 2012 Proceedings IEEE INFOCOM.

[122]  Gagan Goel,et al.  Allocating tasks to workers with matching constraints: truthful mechanisms for crowdsourcing markets , 2014, WWW.

[123]  Luca de Alfaro,et al.  WorkerRank: Using Employer Implicit Judgements to Infer Worker Reputation , 2015, WSDM.

[124]  Manuela M. Veloso,et al.  Task Decomposition, Dynamic Role Assignment, and Low-Bandwidth Communication for Real-Time Strategic Teamwork , 1999, Artif. Intell..