Crowdsourcing: a comprehensive literature review

– The purpose of this paper is to explore the development of crowdsourcing literature. , – This study is a comprehensive review of 346 articles on crowdsourcing. Both statistical and contents analyses were conducted in this paper. , – ISI listed journal articles, non-ISI listed journal articles and conference articles have had nearly the same contribution in crowdsourcing literature. Articles published in non-ISI listed journals have had an essential role in the initial theory development related to crowdsourcing. Scholars from the USA have authored approximately the same number of articles as scholars from all the European countries combined. Scholars from developing countries have been more relatively active in authoring conference articles than journal articles. Only very recently, top-tier journals have engaged in publishing on crowdsourcing. Crowdsourcing has proven to be beneficial in many tasks, but the extant literature does not give much help to practitioners in capturing value from crowdsourcing. Despite understanding that the motivations of crowds are crucial when planning crowdsourcing activities, the various motivations in different contexts have not been explored sufficiently. A major concern has been the quality and accuracy of information that has been gathered through crowdsourcing. Crowdsourcing bears a lot of unused potential. For example, it can increase employment opportunities to low-income people in developing countries. On the other hand, more should be known of fair ways to organize crowdsourcing so that solution seekers do not get a chance to exploit individuals committing to provide solutions. , – The literature included in the study is extensive, but an all-inclusive search for articles was limited to only nine selected publishers. However, in addition to the articles retrieved from the nine selected publishers, 52 highly cited articles were also included from other publishers. , – Crowdsourcing has much unused potential, and the use of crowdsourcing is increasing rapidly. The study provides a thematic review of various applications of crowdsourcing. , – The study is the first of its kind to explore the development of crowdsourcing literature, discussing the loci and foci of extant articles and listing applications of crowdsourcing. Successful applications of crowdsourcing include idea generation, microtasking, citizen science, public participation, wikies, open source software and citizen journalism.

[1]  Michael F. Goodchild,et al.  Please Scroll down for Article International Journal of Digital Earth Crowdsourcing Geographic Information for Disaster Response: a Research Frontier Crowdsourcing Geographic Information for Disaster Response: a Research Frontier , 2022 .

[2]  Jan Marco Leimeister,et al.  Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition , 2009, J. Manag. Inf. Syst..

[3]  Christopher L. Tucci,et al.  Crowdsourcing As a Solution to Distant Search , 2012 .

[4]  Sascha Friesike,et al.  The crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation , 2011 .

[5]  Mokter Hossain,et al.  Generating Ideas on Online Platforms: A Case Study of “My Starbucks Idea” , 2015 .

[6]  Franco Zambonelli Pervasive urban crowdsourcing: Visions and challenges , 2011, 2011 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[7]  Daren C. Brabham MOVING THE CROWD AT THREADLESS , 2010 .

[8]  Nathan Eagle,et al.  txteagle: Mobile Crowdsourcing , 2009, HCI.

[9]  M. Hossain,et al.  Users' motivation to participate in online crowdsourcing platforms , 2012, 2012 International Conference on Innovation Management and Technology Research.

[10]  Karthika Muthukumaraswamy,et al.  WHEN THE MEDIA MEET CROWDS OF WISDOM , 2010 .

[11]  R. Lutz Marketing Scholarship 2.0 , 2011 .

[12]  Fang Wu,et al.  Feedback Loops of Attention in Peer Production , 2009, 2009 International Conference on Computational Science and Engineering.

[13]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[14]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[15]  Saif Mohammad,et al.  CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON , 2013, Comput. Intell..

[16]  Carlos Granell,et al.  An assessment of Public Participation GIS and Web 2.0 technologies in urban planning practice in Canela, Brazil , 2010 .

[17]  Henry Chesbrough,et al.  Open Innovation: The New Imperative for Creating and Profiting from Technology , 2003 .

[18]  Mokter Hossain Performance and Potential of Open Innovation Intermediaries , 2012 .

[19]  Sonja Marjanovic,et al.  Crowdsourcing based business models: In search of evidence for innovation 2.0 , 2012 .

[20]  Daren C. Brabham Crowdsourcing as a Model for Problem Solving , 2008 .

[21]  Taewoo Nam,et al.  Suggesting frameworks of citizen-sourcing via Government 2.0 , 2012, Gov. Inf. Q..

[22]  Huiji Gao,et al.  Harnessing the Crowdsourcing Power of Social Media for Disaster Relief , 2011, IEEE Intelligent Systems.

[23]  Barry L. Bayus,et al.  Crowdsourcing New Product Ideas over Time: An Analysis of the Dell IdeaStorm Community , 2013, Manag. Sci..

[24]  Ricardo Baeza-Yates,et al.  Design and Implementation of Relevance Assessments Using Crowdsourcing , 2011, ECIR.

[25]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[26]  Anne-Wil Harzing,et al.  A preliminary test of Google Scholar as a source for citation data: a longitudinal study of Nobel prize winners , 2013, Scientometrics.

[27]  Yi Xu,et al.  Innovation Contests, Open Innovation, and Multiagent Problem Solving , 2008, Manag. Sci..

[28]  Massimo Franceschet,et al.  A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar , 2010, Scientometrics.

[29]  Sayan Chatterjee,et al.  Democratizing Strategy: How Crowdsourcing Can Be Used for Strategy Dialogues , 2012 .

[30]  Eric Schenk,et al.  Towards a characterization of crowdsourcing practices , 2011 .

[31]  D. Gann,et al.  How open is innovation , 2010 .

[32]  Erran Carmel,et al.  The process of atomization of business tasks for crowdsourcing , 2013 .

[33]  Mark Wexler Reconfiguring the sociology of the crowd: exploring crowdsourcing , 2011 .

[34]  M. Shafique Thinking inside the box? Intellectual structure of the knowledge base of innovation research (1988–2008) , 2013 .

[35]  Serena Carpenter,et al.  A study of content diversity in online citizen journalism and online newspaper articles , 2010, New Media Soc..

[36]  Qinghua Zhu,et al.  Evaluation on crowdsourcing research: Current status and future direction , 2012, Information Systems Frontiers.

[37]  J. Silvertown A new dawn for citizen science. , 2009, Trends in ecology & evolution.

[38]  S. Gorman,et al.  Volunteered Geographic Information and Crowdsourcing Disaster Relief: A Case Study of the Haitian Earthquake , 2010 .

[39]  Daren C. Brabham Crowdsourcing the Public Participation Process for Planning Projects , 2009 .

[40]  Melanie Swan,et al.  Citizen science genomics as a model for crowdsourced preventive medicine research , 2010 .

[41]  Dana Chandler,et al.  Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2012, ArXiv.

[42]  Antony J. Williams,et al.  The Spectral Game: leveraging Open Data and crowdsourcing for education , 2009, J. Cheminformatics.

[43]  O. Bjelland,et al.  An Inside View of IBM's 'Innovation Jam' , 2008 .

[44]  Phuoc Tran-Gia,et al.  Modeling of crowdsourcing platforms and granularity of work organization in Future Internet , 2011, 2011 23rd International Teletraffic Congress (ITC).

[45]  Chris Callison-Burch,et al.  Crowdsourcing Translation: Professional Quality from Non-Professionals , 2011, ACL.

[46]  Tanja Aitamurto THE IMPACT OF CROWDFUNDING ON JOURNALISM , 2011 .

[47]  Kevin Crowston,et al.  From Conservation to Crowdsourcing: A Typology of Citizen Science , 2011, 2011 44th Hawaii International Conference on System Sciences.

[48]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[49]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[50]  Frank Kleemann,et al.  Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing , 2008 .

[51]  Leping Liu,et al.  Web 2.0 Articles: Content Analysis and a Statistical Model to Predict Recognition of the Need for New Instructional Design Strategies , 2008 .

[52]  Steffen Fritz,et al.  Geo-Wiki.Org: The Use of Crowdsourcing to Improve Global Land Cover , 2009, Remote. Sens..

[53]  Andrew Rice,et al.  Open source software and crowdsourcing for energy analysis , 2012 .

[54]  Björn Remneland Wikhamn,et al.  Structuring of the Open Innovation Field , 2013 .

[55]  Rose Holley,et al.  Crowdsourcing: How and Why Should Libraries Do It? , 2010, D Lib Mag..

[56]  Mokter Hossain Crowdsourcing in business and management disciplines: an integrative literature review , 2015 .

[57]  E. Seltzer,et al.  Citizen Participation, Open Innovation, and Crowdsourcing , 2013 .

[58]  Michael F. Goodchild,et al.  NeoGeography and the nature of geographic expertise , 2009, J. Locat. Based Serv..

[59]  J. Crampton Cartography: maps 2.0 , 2009 .

[60]  Jennifer Preece,et al.  Dynamic changes in motivation in collaborative citizen-science projects , 2012, CSCW.

[61]  H. Krcmar,et al.  Community Engineering for Innovations: The Ideas Competition as a Method to Nurture a Virtual Community for Innovations , 2009 .

[62]  Francis D. Tuggle,et al.  Fostering innovation with KM 2.0 , 2010 .

[63]  Karim R. Lakhani,et al.  Marginality and Problem-Solving Effectiveness in Broadcast Search , 2010, Organ. Sci..

[64]  J. Albors,et al.  New learning network paradigms: Communities of objectives, crowdsourcing, wikis and open source , 2008 .

[65]  James Surowiecki The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[66]  Samantha A Adams,et al.  Sourcing the crowd for health services improvement: The reflexive patient and "share-your-experience" websites. , 2011, Social science & medicine.

[67]  Christoph Ihl,et al.  Citizensourcing: Applying the Concept of Open Innovation to the Public Sector , 2010 .

[68]  Fang Wu,et al.  Crowdsourcing, attention and productivity , 2008, J. Inf. Sci..

[69]  Ann Majchrzak,et al.  Towards an Information Systems Perspective and Research Agenda on Crowdsourcing for Innovation , 2013, J. Strateg. Inf. Syst..

[70]  Steffen Roth,et al.  The Diaspora as a Nation’S Capital: Crowdsourcing Strategies for the Caucasus , 2010 .

[71]  A. Hermida TWITTERING THE NEWS , 2010 .

[72]  Anne C. Rouse,et al.  A Preliminary Taxonomy of Crowdsourcing , 2010 .

[73]  Andrew Crooks,et al.  Mapping for the Masses , 2009 .

[74]  Christian Heipke,et al.  Crowdsourcing geospatial data , 2010 .

[75]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[76]  George Ghinea,et al.  Why do commercial companies contribute to open source software? , 2012, Int. J. Inf. Manag..

[77]  Mark Dredze,et al.  Annotating Named Entities in Twitter Data with Crowdsourcing , 2010, Mturk@HLT-NAACL.

[78]  E. Stern,et al.  Web-Based and Traditional Public Participation in Comprehensive Planning: A Comparative Study , 2009 .

[79]  Tara S. Behrend,et al.  The viability of crowdsourcing for survey research , 2011, Behavior research methods.

[80]  Pietro Perona,et al.  Online crowdsourcing: Rating annotators and obtaining cost-effective labels , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[81]  Jared L. Howland,et al.  How Scholarly Is Google Scholar? A Comparison to Library Databases , 2009 .

[82]  Sudhir Agarwal,et al.  Managing Quality of Human-Based eServices , 2008, ICSOC Workshops.

[83]  Sean Ekins,et al.  Reaching Out to Collaborators: Crowdsourcing for Pharmaceutical Research , 2010, Pharmaceutical Research.

[84]  Luke Goode,et al.  Social news, citizen journalism and democracy , 2009, New Media Soc..

[85]  Lydia B. Chilton,et al.  The labor economics of paid crowdsourcing , 2010, EC '10.

[86]  Schahram Dustdar,et al.  Modeling and mining of dynamic trust in complex service-oriented systems , 2010, Inf. Syst..

[87]  Maja Vukovic,et al.  Crowdsourcing for Enterprises , 2009, 2009 Congress on Services - I.

[88]  Martin Schreier,et al.  The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas? , 2009 .