Crowdsourcing: A Snapshot of Published Research

Crowdsourcing, originally defined as “taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in an open call,” is a distributed, collaborative, crossorganizational process seeing increased use among practitioners. As such, crowdsourcing presents great opportunities for information systems (IS) and business-related research. This paper presents preliminary findings from a foundational literature review of published crowdsourcing research from 2006 onward. We identify what crowdsourcing research is going on, where it is going on, and its foci. Our findings document increasing research interest in crowdsourcing and identify the primary publication outlets and home countries of authors involved in that research. Finally, we present a keyword analysis for identified articles, and relate those keywords to a preliminary framework describing crowdsourcing. These findings provide a good summary of current crowdsourcing research, and will help guide researchers interested in further crowdsourcing study.

[1]  Jeff Howe,et al.  Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business , 2008, Human Resource Management International Digest.

[2]  Volker Wulf,et al.  IT to support knowledge sharing in communities, towards a social capital analysis , 2006, J. Inf. Technol..

[3]  Christian Lüthje,et al.  Whom Should Firms Attract to Open Innovation Platforms? The Role of Knowledge Diversity and Motivation , 2011 .

[4]  Wei-Tsong Wang,et al.  Knowledge sharing in Wiki communities: An empirical study , 2011, Online Inf. Rev..

[5]  G. Marchionini,et al.  Awareness in collaborative information seeking , 2010, J. Assoc. Inf. Sci. Technol..

[6]  John Prpic,et al.  The Theory of Crowd Capital , 2012, 2013 46th Hawaii International Conference on System Sciences.

[7]  Jeffrey V. Nickerson,et al.  Generating Creative Ideas Through Crowds: An Experimental Study of Combination , 2011, ICIS.

[8]  G. Marchionini,et al.  Erratum: Awareness in collaborative information seeking (Journal of the American Society for Information Science and Technology (2010) 61:10 (1970-1986)) , 2010 .

[9]  Andrea Back,et al.  A Text Mining Approach to Evaluate Submissions to Crowdsourcing Contests , 2013, 2013 46th Hawaii International Conference on System Sciences.

[10]  Carliss Y. Baldwin,et al.  Modeling a Paradigm Shift: From Producer Innovation to User and Open Collaborative Innovation , 2010, Organ. Sci..

[11]  K. Stanoevska-Slabeva Enabled Innovation: Instruments and Methods of Internet-based Collaborative Innovation , 2011 .

[12]  Rowena Cullen,et al.  Who's Contributing: Do Personality Traits Influence the Level and Type of Participation in Online Communities , 2011, 2011 44th Hawaii International Conference on System Sciences.

[13]  Colin Jevons,et al.  Memo to Marketers: Quantitative Evidence for Change , 2012, Journal of Advertising Research.

[14]  Björn Hartmann,et al.  Collaboratively crowdsourcing workflows with turkomatic , 2012, CSCW.

[15]  Anindya Ghose,et al.  Learning-by-Doing and Project Choice: a Dynamic Structural Model of Crowdsourcing , 2010, ICIS.

[16]  Robert O. Briggs,et al.  Collaboration Engineering: Foundations and Opportunities: Editorial to the Special Issue on the Journal of the Association of Information Systems , 2009, J. Assoc. Inf. Syst..

[17]  Johan Bollen,et al.  Smartocracy: Social Networks for Collective Decision Making , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[18]  David Lazer,et al.  The Multiple Institutional Logics of Innovation , 2011 .

[19]  Kieran Conboy,et al.  A Framework for Investigating Open Innovation Processes in ISD , 2011, ICIS.

[20]  Nicola E. Stokburger-Sauer,et al.  Customer co-creation of travel services: the role of company support and customer satisfaction with the co-creation performance. , 2012 .

[21]  Patrick Finnegan,et al.  Leveraging 'The Crowd': An Exploration of how Solver Brokerages enhance Knowledge Mobility , 2010, ECIS.

[22]  Jan Marco Leimeister,et al.  Rating Scales for Collective Intelligence in Innovation Communities: Why Quick and Easy Decision Making Does Not Get it Right , 2010, ICIS.

[23]  J. Füller,et al.  Why Co‐Creation Experience Matters? Creative Experience and its Impact on the Quantity and Quality of Creative Contributions , 2011 .

[24]  Varun Grover,et al.  Leveraging Information Technology Infrastructure to Facilitate a Firm's Customer Agility and Competitive Activity: An Empirical Investigation , 2012, J. Manag. Inf. Syst..

[25]  David Alan Grier Not for All Markets , 2011, Computer.

[26]  Eoin Whelan,et al.  Exploring knowledge exchange in electronic networks of practice , 2007, J. Inf. Technol..

[27]  Daren C. Brabham Motivations for Participation in a Crowdsourcing Application to Improve Public Engagement in Transit Planning , 2012 .

[28]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[29]  Lara Khansa,et al.  Creating a Taxonomy for Mobile Commerce Innovations Using Social Network and Cluster Analyses , 2012, Int. J. Electron. Commer..

[30]  Joseph G. Davis,et al.  Web 3.0 and Crowdservicing , 2011, AMCIS.

[31]  Jeffrey V. Nickerson,et al.  Evaluating Design Solutions Using Crowds , 2011, AMCIS.

[32]  Kalle Lyytinen,et al.  Research Commentary - Digital Infrastructures: The Missing IS Research Agenda , 2010, Inf. Syst. Res..

[33]  I. C. Meijer The wisdom of crowds , 2010 .

[34]  Claudio Bartolini,et al.  Towards a Research Agenda for Enterprise Crowdsourcing , 2010, ISoLA.

[35]  Radhika Jain Investigation of Governance Mechanisms for Crowdsourcing Initiatives , 2010, AMCIS.

[36]  Thomas P. Walter,et al.  Mediterranean Conference on Information Systems ( MCIS ) 2011 TOWARDS MEASURING CROWDSOURCING SUCCESS : AN EMPIRICAL STUDY ON EFFECTS OF EXTERNAL FACTORS IN ONLINE IDEA CONTEST , 2017 .

[37]  Albert E. Mannes Are We Wise About the Wisdom of Crowds? The Use of Group Judgments in Belief Revision , 2009, Manag. Sci..

[38]  Dawn G. Gregg Designing for collective intelligence , 2010, CACM.

[39]  Brian A Vander Schee Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business , 2009 .

[40]  Dahui Li,et al.  Task Design, Motivation, and Participation in Crowdsourcing Contests , 2011, Int. J. Electron. Commer..

[41]  Yi Xu,et al.  Innovation Contests, Open Innovation, and Multiagent Problem Solving , 2008, Manag. Sci..

[42]  Fernando González-Ladrón-de-Guevara,et al.  Towards an integrated crowdsourcing definition , 2012, J. Inf. Sci..

[43]  Lieven De Marez,et al.  Smart Ideas for Smart Cities: Investigating Crowdsourcing for Generating and Selecting Ideas for ICT Innovation in a City Context , 2012, J. Theor. Appl. Electron. Commer. Res..

[44]  Martin Schader,et al.  Crowdsourcing Information Systems - Definition, Typology, and Design , 2012, ICIS.

[45]  Licia Capra,et al.  Quality control for real-time ubiquitous crowdsourcing , 2011, UbiCrowd '11.

[46]  Lee Sproull,et al.  The Role of Feedback in Managing the Internet-Based Volunteer Work Force , 2008, Inf. Syst. Res..

[47]  Eric Horvitz,et al.  Crowdsourcing General Computation , 2011 .

[48]  Chin-Laung Lei,et al.  Quadrant of euphoria: a crowdsourcing platform for QoE assessment , 2010, IEEE Network.

[49]  A. Pentland,et al.  Collective intelligence , 2006, IEEE Comput. Intell. Mag..

[50]  Arun Sundararajan,et al.  Optimal Design of Crowdsourcing Contests , 2009, ICIS.

[51]  Michel Avital,et al.  Generative Collectives , 2010, ICIS.

[52]  E. Bonabeau Decisions 2.0: the power of collective intelligence , 2009 .

[53]  Keng Siau,et al.  Co-creation and Collaboration in a Virtual World: A 3D Visualization Design Project in Second Life , 2010, J. Database Manag..

[54]  Eileen M. Trauth,et al.  Hanging with the right crowd: Matching crowdsourcing need to crowd characteristics , 2012, AMCIS.

[55]  Shu Han,et al.  Changing the Competitive Landscape: Continuous Innovation Through IT-Enabled Knowledge Capabilities , 2010, Inf. Syst. Res..

[56]  Robert A. Greve,et al.  The Use of Mobile Devices as Group Wisdom Support Systems to Support Dynamic Crowdsourcing Efforts , 2012, AMCIS.

[57]  Liang Chen,et al.  Comparing Strategies for Winning Expert-rated and Crowd-rated Crowdsourcing Contests: First Findings , 2012, AMCIS.

[58]  Christian Wagner,et al.  Innovating Collaborative Content Creation: The Role of Altruism and Wiki Technology , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[59]  Marcos R. S. Borges,et al.  Taking advantage of collective knowledge in emergency response systems , 2012, J. Netw. Comput. Appl..

[60]  A. Bullinger,et al.  Innovation Contests: A Review, Classification and Outlook , 2012 .

[61]  Joseph Lampel,et al.  Test-Driving the Future: How Design Competitions Are Changing Innovation , 2012 .

[62]  Gert-Jan de Vreede,et al.  Conceptual Foundations of Crowdsourcing: A Review of IS Research , 2013, 2013 46th Hawaii International Conference on System Sciences.

[63]  Qinghua Zhu,et al.  Exploring the Motivation of Participants in Crowdsourcing Contest , 2012, ICIS.

[64]  Martin Schader,et al.  Workers' Task Choice in Crowdsourcing and Human Computation Markets , 2012, ICIS.

[65]  Martin Schader,et al.  Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes , 2011, AMCIS.

[66]  Jeffrey V. Nickerson,et al.  Matching Mechanisms to Situations Through the Wisdom of the Crowd , 2009, ICIS.

[67]  Krittinee Nuttavuthisit If you can’t beat them, let them join: The development of strategies to foster consumers’ co-creative practices , 2010 .

[68]  Dawn G. Gregg Developing a collective intelligence application for special education , 2009, Decis. Support Syst..

[69]  Karl Reiner Lang,et al.  Digital Consumer Networks and Producer-Consumer Collaboration: Innovation and Product Development in the Video Game Industry , 2007, J. Manag. Inf. Syst..

[70]  Wael Soliman,et al.  Crowdsourcing as a Mobile Service - Case Study: Publishing Photography , 2012, ECIS.

[71]  David H. Wolpert,et al.  Collective Intelligence , 1999 .

[72]  Beibei Li,et al.  Designing Ranking Systems for Hotels on Travel Search Engines by Mining User-Generated and Crowd-Sourced Content , 2011, Mark. Sci..

[73]  Saggi Nevo,et al.  New Capabilities: Can IT Service Providers Leverage Crowdsourcing? , 2012, ICIS.

[74]  Andrew Rice,et al.  Open source software and crowdsourcing for energy analysis , 2012 .

[75]  Katja Thoring,et al.  Crowdsourcing with Semantic Differentials: A Game to Investigate the Meaning of Form , 2010, AMCIS.

[76]  Jan Marco Leimeister,et al.  Leveraging crowdsourcing – activationsupporting components for IT-based idea competitions , 2009 .

[77]  Michel Avital,et al.  Crowdpushing: the flip Side of Crowdsourcing , 2012, ECIS.

[78]  Yu-Hao Chuang,et al.  Why people share knowledge in virtual communities , 2011 .

[79]  Bart van den Hooff,et al.  The value of online networks of practice: the role of embeddedness and media use , 2010, J. Inf. Technol..

[80]  Henrik Ickler,et al.  Design of New Business Models for Service Integrators by Creating Information-Driven Value Webs Based on Customers' Collective Intelligence , 2009, 2009 42nd Hawaii International Conference on System Sciences.

[81]  Kwong-Sak Leung,et al.  A Survey of Crowdsourcing Systems , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[82]  Patrick Finnegan,et al.  'Orchestrating' sustainable crowdsourcing: A characterisation of solver brokerages , 2012, J. Strateg. Inf. Syst..

[83]  Yoris A. Au,et al.  Assessing Researcher Publication Productivity in the Leading Information Systems Journals: A 2005-2009 Update , 2011, Commun. Assoc. Inf. Syst..

[84]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[85]  Monika Skaržauskaitė The Application of Crowd Sourcing in Educational Activities , 2012 .

[86]  D. Lei,et al.  Collaborative Innovation with Customers: A Review of the Literature and Suggestions for Future Research , 2012 .

[87]  Karim R. Lakhani,et al.  Marginality and Problem-Solving Effectiveness in Broadcast Search , 2010, Organ. Sci..

[88]  Joel Podolny,et al.  Organizing Contests for Status: The Matthew Effect vs. the Mark Effect , 2011, Manag. Sci..

[89]  Yili Hong,et al.  An Empirical Investigation on Provider Pricing in Online Crowdsourcing Markets for IT Services , 2012, ICIS.

[90]  Kevin Crowston,et al.  Goals and Tasks: Two Typologies of Citizen Science Projects , 2012, 2012 45th Hawaii International Conference on System Sciences.

[91]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[92]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[93]  Brian S. Butler,et al.  Research Note - The Impact of Community Commitment on Participation in Online Communities , 2011, Inf. Syst. Res..

[94]  Sanjiv Erat,et al.  Managing Delegated Search Over Design Spaces , 2012, Manag. Sci..

[95]  Li Chen,et al.  Theory and Analysis of Company-Sponsored Value Co-Creation , 2012, J. Manag. Inf. Syst..

[96]  Li Liu,et al.  Factors affecting participation of solvers in crowdsourcing: an empirical study from China , 2012, Electron. Mark..

[97]  Param Vir Singh,et al.  Crowdsourcing Contests: A Dynamic Structural Model of the Impact of Incentive Structure on Solution Quality , 2012, ICIS.

[98]  Daren C. Brabham Crowdsourcing the Public Participation Process for Planning Projects , 2009 .

[99]  Holger Hoffmann,et al.  Towards the Automated Evaluation of Crowd Work: Machine-Learning Based Classification of Complex Texts Simplified by Laymen , 2013, 2013 46th Hawaii International Conference on System Sciences.

[100]  Sharon Swee-Lin Tan,et al.  Contribution Behaviors in Distributed Environments , 2008, MIS Q..

[101]  Young-Gul Kim,et al.  Why would online gamers share their innovation-conducive knowledge in the online game user community? Integrating individual motivations and social capital perspectives , 2011, Comput. Hum. Behav..

[102]  Jie Ren,et al.  Who's more creative, experts or the crowd? , 2011, AMCIS.

[103]  James Surowiecki The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[104]  Karim R. Lakhani,et al.  Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis , 2011, Manag. Sci..

[105]  Jeffrey Heer,et al.  Strategies for crowdsourcing social data analysis , 2012, CHI.

[106]  Henrik Ickler,et al.  Design of new business models for service integrators by creating information-driven value webs based on customers collective intelligence , 2009 .

[107]  Martin Schader,et al.  Exploring task properties in crowdsourcing - an empirical study on mechanical turk , 2011, ECIS.

[108]  Michael Rosemann,et al.  Crowdsourcing Information Systems - A Systems Theory Perspective , 2011 .

[109]  Jie Ren Exploring the Process of Web-based Crowdsourcing Innovation , 2011, AMCIS.

[110]  Neil Savage,et al.  Gaining wisdom from crowds , 2012, Commun. ACM.

[111]  Eileen M. Trauth,et al.  - in-Progress , 2012 .

[112]  Kalina Bontcheva,et al.  Crowdsourcing research opportunities: lessons from natural language processing , 2012, i-KNOW '12.

[113]  Daren C. Brabham MOVING THE CROWD AT THREADLESS , 2010 .

[114]  Jan Marco Leimeister,et al.  Collective Intelligence , 2010, Bus. Inf. Syst. Eng..

[115]  Florian Michahelles,et al.  AIS Electronic Library (AISeL) Crowdsourcing for “Kiosk of the Future” – A Retail Store Case Study , 2017 .

[116]  Jeffrey V. Nickerson,et al.  Structures for Creativity: The crowdsourcing of design , 2011 .

[117]  Johann Füller,et al.  Consumer Empowerment Through Internet-Based Co-creation , 2009, J. Manag. Inf. Syst..

[118]  Vladimir Zwass,et al.  Co-Creation: Toward a Taxonomy and an Integrated Research Perspective , 2010, Int. J. Electron. Commer..

[119]  Stephen J. Andriole,et al.  Seven Indisputable Technology Trends That Will Define 2015 , 2012, Commun. Assoc. Inf. Syst..

[120]  Alon Y. Halevy,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[121]  K. Lyytinen,et al.  Digital Infrastructures : The Missing IS Research Agenda , 2010 .

[122]  Taewoo Nam,et al.  The Wisdom of Crowds in Government 2.0: Information Paradigm Evolution toward Wiki-Government , 2010, AMCIS.

[123]  Rudy Hirschheim,et al.  Online Sourcing: Investigations from Service Clients' Perspective , 2011, AMCIS.

[124]  Kalle Lyytinen,et al.  Organizing for Innovation in the Digitized World , 2012, Organ. Sci..

[125]  Fei-Yue Wang,et al.  Understanding Crowd-Powered Search Groups: A Social Network Perspective , 2012, PloS one.

[126]  Henri Simula,et al.  The Rise and Fall of Crowdsourcing? , 2013, 2013 46th Hawaii International Conference on System Sciences.

[127]  David Hachen,et al.  Citizen Engineering: Methods for "Crowdsourcing" Highly Trustworthy Results , 2012, 2012 45th Hawaii International Conference on System Sciences.

[128]  Eldon Y. Li,et al.  Understanding Web 2.0 service models: A knowledge-creating perspective , 2011, Inf. Manag..

[129]  Matthew Lease,et al.  On Quality Control and Machine Learning in Crowdsourcing , 2011, Human Computation.

[130]  Wendy A. Warr,et al.  Social software: fun and games, or business tools? , 2008, J. Inf. Sci..

[131]  Barry L. Bayus,et al.  Crowdsourcing New Product Ideas over Time: An Analysis of the Dell IdeaStorm Community , 2013, Manag. Sci..

[132]  Jan Marco Leimeister,et al.  Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters? , 2011, ICIS.

[133]  Frank DiGiammarino,et al.  Virtual Networks: An Opportunity for Government: The Increasing Power of Computing Is Enabling a New Generation of Web-Based Applications-Web 2.0-To Harness Collective Intelligence in the Public Sector , 2008 .

[134]  Peter A. Gloor,et al.  The New Principles of a Swarm Business , 2007 .

[135]  Yongqiang Sun,et al.  Investigating the Non-Linear Relationships in the Expectancy Theory: The Case of Crowdsourcing Marketplace , 2012, AMCIS.

[136]  Aniket Kittur,et al.  CrowdForge: crowdsourcing complex work , 2011, UIST.

[137]  Monika C. Schuhmacher,et al.  Identification of Lead User Characteristics Driving the Quality of Service Innovation Ideas , 2012 .

[138]  Rolf Pfeifer,et al.  Crowdsourcing, Open Innovation and Collective Intelligence in the Scientific Method - A Research Agenda and Operational Framework , 2010, ALIFE.

[139]  Caroline Haythornthwaite,et al.  Crowds and communities: Light and heavyweight models of peer production , 2009 .

[140]  Mark Klein,et al.  Can We Exploit Collective Intelligence for Collaborative Deliberation? The Case of the Climate Change Collaboratorium , 2007 .

[141]  Mohammad Jafar Tarokh,et al.  Efficient Business Process Reengineering with , 2012 .