Can the Crowd Substitute Experts in Evaluating Creative jobs? The Case of Business Models

Crowdvoting offers companies the opportunity to get business innovation ideas evaluated by a large number of contributors (the crowd), as an alternative to the currently prevalent practice of involving experts. In this paper we investigate whether the evaluation of a large number of complex ideas by the crowd can be conformant with that of experts. The ideas used to compare these evaluations were generated in an experiment involving students, with eighty different business models featuring as complex ideas. The results of the evaluation by experts were compared with that of an online crowd involved via a crowdvoting-platform. Our results show that an anonymous online-crowd is not as adept as experts when it comes to evaluating complex ideas such as business models. In contrast with previous studies of crowd evaluations for simple aesthetic tasks, our study provides first evidence of the limitations of crowd evaluations, and warns against the substitutions of experts for the evaluation of more complex ideas.

[1]  Tobias Bachmeier,et al.  Business Model Generation A Handbook For Visionaries Game Changers And Challengers , 2016 .

[2]  Karl T. Ulrich,et al.  Idea Generation and the Quality of the Best Idea , 2009, Manag. Sci..

[3]  A. Johne,et al.  New service development: a review of the literature and annotated bibliography , 1998 .

[4]  Param Vir Singh,et al.  Crowdsourcing New Product Ideas Under Consumer Learning , 2014, Manag. Sci..

[5]  Thomas J. Howard,et al.  Describing the creative design process by the integration of engineering design and cognitive psychology literature , 2008 .

[6]  D. Teece.  Business Models, Business Strategy and Innovation , 2010 .

[7]  Dirk Schneckenberg,et al.  Open Evaluation of New Product Concepts at the Front End of Innovation: Objectives and Contingency Factors , 2017 .

[8]  Laura J. Kornish,et al.  The Importance of the Raw Idea in Innovation: Testing the Sow's Ear Hypothesis , 2012 .

[9]  R. Cooper.  Predevelopment activities determine new product success , 1988 .

[10]  Min Basadur,et al.  Collaborative Problem Solving Through Creativity in Problem Definition: Expanding the Pie , 2000 .

[11]  R. Amit,et al.  The Business Model: Recent Developments and Future Research , 2011 .

[12]  J. Hauser,et al.  The virtual customer , 2002 .

[13]  W. Lyddon.  Creativity in Context: Update to "The Social Psychology of Creativity" , 1998 .

[14]  Martin Schreier,et al.  The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas? , 2009 .

[15]  Abhijeet K. Vadera,et al.  Win or Lose the Battle for Creativity: The Power and Perils of Intergroup Competition , 2010 .

[16]  Clayton M. Christensen,et al.  Reinventing Your Business Model , 2008 .

[17]  Francesco Galati,et al.  Complexity of Judgment: What Makes Possible the Convergence of Expert and Nonexpert Ratings in Assessing Creativity , 2015 .

[18]  Xavier Lecocq,et al.  Business Model Evolution: In Search of Dynamic Consistency , 2010 .

[19]  Efraim Turban,et al.  What can crowdsourcing do for decision support? , 2014, Decis. Support Syst..

[20]  Erik Wästlund,et al.  Exploring Users' Appropriateness as a Proxy for Experts When Screening New Product/Service Ideas† , 2016 .

[21]  Bo T. Christensen,et al.  Distributed idea screening in stage–gate development processes , 2012 .

[22]  G. Pisano.  Which kind of collaboration is right for you , 2009 .

[23]  Carol Bruckner Coles,et al.  Establishing a continuing business model innovation process , 2004 .

[24]  Dominik Walcher,et al.  Toolkits for Idea Competitions: A Novel Method to Integrate Users in New Product Development , 2006 .

[25]  Wafa Hammedi,et al.  Antecedents and consequences of reflexivity in new product idea screening , 2011 .

[26]  Barry L. Bayus,et al.  Crowdsourcing New Product Ideas over Time: An Analysis of the Dell IdeaStorm Community , 2013, Manag. Sci..

[27]  Jan Marco Leimeister,et al.  Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters? , 2011, ICIS.

[28]  Ethan Mollick,et al.  Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts , 2015, Manag. Sci..

[29]  David Mazursky,et al.  How Do Financial Constraints Affect Creativity , 2014 .

[30]  R. Duane Ireland,et al.  Integrating entrepreneurship and strategic management actions to create firm wealth , 2001 .

[31]  Paul Michael Di Gangi,et al.  Getting Customers' Ideas to Work for You: Learning from Dell how to Succeed with Online User Innovation Communities , 2010, MIS Q. Executive.

[32]  Carol Bruckner Coles,et al.  The ultimate competitive advantage of continuing business model innovation , 2003 .

[33]  Olivier Toubia,et al.  Adaptive Idea Screening Using Consumers , 2007 .