Components and Functions of Crowdsourcing Systems - A Systematic Literature Review

Many organizations are now starting to introduce crowdsourcing as a new model of business to outsource tasks, which are traditionally performed by a small group of people, to an undefined large workforce. While the utilization of crowdsourcing offers a lot of advantages, the development of the required system carries some risks, which are reduced by establishing a profound theo- retical foundation. Thus, this article strives to gain a better understanding of what crowdsourcing systems are and what typical design aspects are considered in the development of such systems. In this paper, the author conducted a sys- tematic literature review in the domain of crowdsourcing systems. As a result, 17 definitions of crowdsourcing systems were found and categorized into four perspectives: the organizational, the technical, the functional, and the human- centric. In the second part of the results, the author derived and presented com- ponents and functions that are implemented in a crowdsourcing system.

[1]  Marco Brambilla,et al.  A revenue sharing mechanism for federated search and advertising , 2012, WWW.

[2]  J. Bortz,et al.  Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler , 2006 .

[3]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[4]  Arjen P. de Vries,et al.  Increasing cheat robustness of crowdsourcing tasks , 2013, Information Retrieval.

[5]  Samuel Greengard,et al.  Following the crowd , 2011, Commun. ACM.

[6]  Anne C. Rouse,et al.  A Preliminary Taxonomy of Crowdsourcing , 2010 .

[7]  Daren C. Brabham.  Crowdsourcing as a Model for Problem Solving , 2008 .

[8]  Pearl Brereton,et al.  Protocol for a Tertiary study of Systematic Literature Reviews and Evidence-based Guidelines in IT and Software Engineering , 2009 .

[9]  Douglas Thain,et al.  Expert-Citizen Engineering: "Crowdsourcing" Skilled Citizens , 2011, 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing.

[10]  Aniket Kittur,et al.  CrowdWeaver: visually managing complex crowd work , 2012, CSCW.

[11]  Reza Zafarani,et al.  Maximizing benefits from crowdsourced data , 2012, Computational and Mathematical Organization Theory.

[12]  Wolf-Tilo Balke,et al.  Information Extraction Meets Crowdsourcing: A Promising Couple , 2012, Datenbank-Spektrum.

[13]  Phuoc Tran-Gia,et al.  Modeling of crowdsourcing platforms and granularity of work organization in Future Internet , 2011, 2011 23rd International Teletraffic Congress (ITC).

[14]  Jan Marco Leimeister,et al.  Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition , 2009, J. Manag. Inf. Syst..

[15]  Bill Tomlinson,et al.  Who are the crowdworkers?: shifting demographics in mechanical turk , 2010, CHI Extended Abstracts.

[16]  Huiji Gao,et al.  Harnessing the Crowdsourcing Power of Social Media for Disaster Relief , 2011, IEEE Intelligent Systems.

[17]  David R. Karger,et al.  Human-powered Sorts and Joins , 2011, Proc. VLDB Endow..

[18]  Rick Kazman,et al.  The metropolis model a new logic for development of crowdsourced systems , 2009, CACM.

[19]  Claudio Bartolini,et al.  Towards a Research Agenda for Enterprise Crowdsourcing , 2010, ISoLA.

[20]  Carla Simone,et al.  Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work Companion , 2012, CSCW 2012.

[21]  Christian Heipke,et al.  Crowdsourcing geospatial data , 2010 .

[22]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[23]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[24]  William C. Regli,et al.  Putting the crowd to work in a knowledge-based factory , 2010, Adv. Eng. Informatics.

[25]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[26]  Hajo Hippner,et al.  Crowdsourcing , 2012, Business & Information Systems Engineering.

[27]  Mihaela van der Schaar,et al.  Reputation-based incentive protocols in crowdsourcing applications , 2011, 2012 Proceedings IEEE INFOCOM.

[28]  Tim Kraska,et al.  CrowdDB: answering queries with crowdsourcing , 2011, SIGMOD '11.

[29]  Carlos J. Costa,et al.  Proposing a system to support crowdsourcing , 2012, OSDOC '12.

[30]  Andrea Castelletti,et al.  Putting humans in the loop: Social computing for Water Resources Management , 2012, Environ. Model. Softw..

[31]  Viswanath Poosala,et al.  MoneyBee: Towards enabling a ubiquitous, efficient, and easy-to-use mobile crowdsourcing service in the emerging market , 2011, Bell Labs Technical Journal.

[32]  Scott R. Klemmer,et al.  Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology , 2011, UIST 2011.

[33]  Devavrat Shah,et al.  Budget-optimal crowdsourcing using low-rank matrix approximations , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[34]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[35]  Martin Schader,et al.  Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes , 2011, AMCIS.

[36]  Tobias Hoßfeld,et al.  Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms , 2013, Math. Comput. Model..

[37]  Li Liu,et al.  Factors affecting participation of solvers in crowdsourcing: an empirical study from China , 2012, Electron. Mark..

[38]  Neoklis Polyzotis,et al.  Max algorithms in crowdsourcing environments , 2012, WWW.

[39]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[40]  Daniel Schall,et al.  Expertise ranking using activity and contextual link measures , 2012, Data Knowl. Eng..

[41]  Soundar Kumara,et al.  Proceedings of the 2nd international workshop on Ubiquitous crowdsouring , 2011, UbiComp 2011.

[42]  Licia Capra,et al.  Quality control for real-time ubiquitous crowdsourcing , 2011, UbiCrowd '11.

[43]  Tatsuo Nakajima,et al.  Drawing on mobile crowds via social media , 2012, Multimedia Systems.

[44]  Phuoc Tran-Gia,et al.  Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com , 2011, 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing.

[45]  T. J. Watson.  Some Thoughts on a Framework for Crowdsourcing A Position Paper for the CHI 2011 Workshop on Crowdsourcing and Human Computation , 2011 .

[46]  Milan Vojnovic,et al.  Crowdsourcing and all-pay auctions , 2009, EC '09.

[47]  Kristina Höök,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 2012 .

[48]  Kwong-Sak Leung,et al.  A Survey of Crowdsourcing Systems , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[49]  Eric Schenk,et al.  Towards a characterization of crowdsourcing practices , 2011 .

[50]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[51]  Abdulsalam Yassine,et al.  Proceedings of the 2011 international ACM workshop on Ubiquitous meta user interfaces , 2011, MM 2011.

[52]  Hiroaki Kimura,et al.  A crowdsourcing based mobile image translation and knowledge sharing service , 2010, MUM.

[53]  SchallDaniel.  Expertise ranking using activity and contextual link measures , 2012, ICDKE 2012.

[54]  A. Meystel,et al.  Intelligent Systems , 2001 .

[55]  Deepak Sundararajan,et al.  Enterprise crowdsourcing solutions for software development and ideation , 2011, UbiCrowd '11.

[56]  Maja Vukovic,et al.  Crowdsourcing for Enterprises , 2009, 2009 Congress on Services - I.

[57]  Schahram Dustdar,et al.  Tweetflows: flexible workflows with twitter , 2011, PESOS '11.

[58]  Panagiotis G. Ipeirotis,et al.  Quality management on Amazon Mechanical Turk , 2010, HCOMP '10.

[59]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[60]  Tatsuo Nakajima,et al.  Gamifying intelligent environments , 2011, Ubi-MUI '11.

[61]  Michael S. Bernstein,et al.  Crowds in two seconds: enabling realtime crowd-powered interfaces , 2011, UIST.

[62]  Krzysztof Z. Gajos,et al.  Human computation tasks with global constraints , 2012, CHI.

[63]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[64]  Matthew Lease,et al.  Enabling trust in crowd labor relations through identity sharing , 2011, ASIST.

[65]  Gabriella Kazai,et al.  Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking , 2011, SIGIR.

[66]  Nikolay Archak,et al.  Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com , 2010, WWW '10.