Mediterranean Conference on Information Systems ( MCIS ) 2011 TOWARDS MEASURING CROWDSOURCING SUCCESS : AN EMPIRICAL STUDY ON EFFECTS OF EXTERNAL FACTORS IN ONLINE IDEA CONTEST

Crowdsourcing campaigns in general and online idea contests in particular have reached popularity in practice and research. Based on the open innovation paradigm, idea marketplace providers like InnoCentive broker ideas between a potentially large-scaled crowd of solvers and seeking firms by hosting idea contests in the web. So far, research has mainly focused on the setup of idea marketplaces, design patterns to define idea contests or motivation factors to spur solvers. Mostly, extrinsic incentives, especially rewards for best ideas, are used to guarantee high amounts and good quality of ideas. In contrast to the predominant qualitative and conceptional research on online idea contests, we attempt an empirical study. We derive external factors, like rewards, contest-duration or market-maturity, to construct a model depicting effects of these factors on the outcome of online idea contests. We use a dataset from Atizo, containing over 25’000 submitted ideas and over 83’000 ratings to evaluate validity of our model. Our findings show that current design elements of online contest may not contribute to the commonly expected effects. One interesting aspect is that using rewards to maximize quality and quantity of ideas creates a conflict of goals as rewards can have perverse effects on the outcome.

[1]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[2]  Edward L. Deci,et al.  The hidden costs of rewards , 1976 .

[3]  Gerardine DeSanctis,et al.  A foundation for the study of group decision support systems , 1987 .

[4]  Teresa M. Amabile,et al.  A Model of Creativity and Innovation in Organizations , 1988 .

[5]  Economic Contests with Incomplete Information and Optimal Contest Design , 1988 .

[6]  J. Valacich,et al.  Effects of anonymity and evaluative tone on idea generation in computer-mediated groups , 1990 .

[7]  Jay F. Nunamaker,et al.  Electronic meeting systems , 1991, CACM.

[8]  Douglas R. Vogel,et al.  Electronic meeting systems to support information systems analysis and design , 1992 .

[9]  A. Kohn.  Why incentive plans cannot work , 1993 .

[10]  J. Sinacore.  Multiple regression: Testing and interpreting interactions , 1993 .

[11]  William M. Riggs,et al.  Incentives to innovate and the sources of innovation: the case of scientific instruments☆ , 1994 .

[12]  Karl T. Ulrich,et al.  Product Design and Development , 1995 .

[13]  B. Frey,et al.  The Cost of Price Incentives: An Empirical Analysis of Motivation Crowding-Out , 1997 .

[14]  Pierre Lévy,et al.  Collective Intelligence: Mankind's Emerging World in Cyberspace , 1997 .

[15]  Teresa M. Amabile,et al.  Motivating Creativity in Organizations: On Doing What You Love and Loving What You Do , 1997 .

[16]  Thomas A. Rietz,et al.  What Makes Markets Predict Well? Evidence from the Iowa Electronic Markets , 1997 .

[17]  B. Moldovanu,et al.  The Optimal Allocation of Prizes in Contests , 2001 .

[18]  E. Deci,et al.  A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. , 1999, Psychological bulletin.

[19]  Mark J. Safferstone.  Information Rules: A Strategic Guide to the Network Economy , 1999 .

[20]  A. Rustichini,et al.  Pay Enough or Don't Pay at All , 2000 .

[21]  Josh Lerner,et al.  The Simple Economics of Open Source , 2000 .

[22]  Ian L. Gale,et al.  Optimal Design of Research Contests , 2003 .

[23]  Nikolaus Franke,et al.  How Communities Support Innovative Activities : An Exploration of Assistance and Sharing Among Innovative Users of Sporting Equipment , 2001 .

[24]  H. Varian,et al.  The Economics Of Information Technology , 2004 .

[25]  E Bonabeau,et al.  Swarm Intelligence: A Whole New Way to Think about Business , 2001 .

[26]  Gary L. Lilien,et al.  Performance Assessment of the Lead User Idea-Generation Process for New Product Development , 2002, Manag. Sci..

[27]  Dick Stenmark,et al.  Group cohesiveness and extrinsic motivation in virtual groups: lessons from an action case study of electronic brainstorming , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[28]  E. Hippel,et al.  Customers As Innovators: A New Way to Create Value , 2002 .

[29]  Sonali K. Shah,et al.  How Communities Support Innovative Activities: An Exploration of Assistance and Sharing Among End-Users , 2003 .

[30]  Kouichi Kishida,et al.  Toward an understanding of the motivation of open source software developers , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[31]  Georg von Krogh,et al.  Open Source Software and the "Private-Collective" Innovation Model: Issues for Organization Science , 2003, Organ. Sci..

[32]  Oliver Gassmann,et al.  Towards a Theory of Open Innovation: Three Core Process Archetypes , 2004 .

[33]  J. Callahan,et al.  The Importance of Customer Input in the Development of Very New Products , 2004 .

[34]  James Surowiecki.  The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[35]  J. Marshall.  Open Innovation: The New Imperative for Creating and Profiting from Technology , 2004 .

[36]  Nicolas Michinov,et al.  Improving productivity and creativity in online groups through social comparison process: New evidence for asynchronous electronic brainstorming , 2005, Comput. Hum. Behav..

[37]  John Cullen,et al.  Democratizing Innovation , 2020, Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship.

[38]  Dominik Walcher,et al.  Toolkits for Idea Competitions: A Novel Method to Integrate Users in New Product Development , 2006 .

[39]  J. Tirole,et al.  Incentives and Prosocial Behavior , 2004 .

[40]  Sonali K. Shah.  Motivation, Governance, and the Viability of Hybrid Forms in Open Source Software Development , 2006, Manag. Sci..

[41]  A. Salter,et al.  Open for innovation: the role of openness in explaining innovation performance among U.K. manufacturing firms , 2006 .

[42]  H. Chesbrough.  Open Business Models: How to Thrive in the New Innovation Landscape , 2006 .

[43]  H. Chesbrough,et al.  Open Innovation and Strategy , 2007 .

[44]  Brian Fitzgerald,et al.  Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects , 2007 .

[45]  D. Ariely,et al.  Doing Good or Doing Well? Image Motivation and Monetary Incentives in Behaving Prosocially , 2007, SSRN Electronic Journal.

[46]  L. Jeppesen,et al.  The Value of Openness in Scientific Problem Solving , 2007 .

[47]  Jeff Howe,et al.  Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business , 2008, Human Resource Management International Digest.

[48]  J. West,et al.  Open innovation : researching a new paradigm , 2008 .

[49]  Andrea Back,et al.  Group wisdom support systems: Aggregating the insights of many through information technology , 2008 .

[50]  Daren C. Brabham.  Crowdsourcing as a Model for Problem Solving , 2008 .

[51]  Jan Marco Leimeister,et al.  Leveraging the Wisdom of Crowds: Designing an IT-Supported Ideas Competition for an ERP Software Company , 2008, Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008).

[52]  Yi Xu,et al.  Innovation Contests, Open Innovation, and Multiagent Problem Solving , 2008, Manag. Sci..

[53]  Arun Sundararajan,et al.  Optimal Design of Crowdsourcing Contests , 2009, ICIS.

[54]  Oded Nov,et al.  Motivational, Structural and Tenure Factors that Impact Online Community Photo Sharing , 2009, ICWSM.

[55]  Jan Marco Leimeister,et al.  Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition , 2009, J. Manag. Inf. Syst..

[56]  Nina Mazar,et al.  Large stakes and big mistakes , 2009 .

[57]  Karl T. Ulrich,et al.  Innovation Tournaments: Creating and Selecting Exceptional Opportunities , 2009 .

[58]  Jeffrey V. Nickerson,et al.  Matching Mechanisms to Situations Through the Wisdom of the Crowd , 2009, ICIS.

[59]  R. Briggs,et al.  Association for Information Systems , 2007 .

[60]  Paul A. Pavlou,et al.  International Conference on Information Systems ( ICIS ) 1-1-2009 Open Innovation : An Empirical Study of Online Contests , 2013 .

[61]  E. Bonabeau.  Decisions 2.0: the power of collective intelligence , 2009 .

[62]  Rosann Webb Collins,et al.  Information Markets: A Research Landscape , 2009, Commun. Assoc. Inf. Syst..

[63]  Jan Marco Leimeister,et al.  Leveraging crowdsourcing – activationsupporting components for IT-based idea competitions , 2009 .

[64]  Paul Michael Di Gangi,et al.  Steal my idea! Organizational adoption of user innovations from a user innovation community: A case study of Dell IdeaStorm , 2009, Decis. Support Syst..

[65]  Fang Wu,et al.  Crowdsourcing, attention and productivity , 2008, J. Inf. Sci..

[66]  Richard D. Wang,et al.  Tournaments for Ideas , 2010 .

[67]  Andrea Back,et al.  Crowdsourcing as a Business Model: An Exploration of Emergent Textbooks Harnessing the Wisdom of Crowds , 2010, Bled eConference.

[68]  Karl T. Ulrich,et al.  Idea Generation and the Quality of the Best Idea , 2009, Manag. Sci..

[69]  Heli Väätäjä,et al.  Rewarding in open innovation communities – how to motivate members , 2010 .