The Bag of Stars: High-Speed Idea Filtering for Open Innovation

Open innovation platforms (web sites where crowds post ideas in a shared space) enable us to elicit huge volumes of potentially valuable solutions for problems we care about, but identifying the best ideas in these collections can be prohibitively expensive and time-consuming. This paper presents an approach, called the "bag of stars," which enables crowd to filter ideas with accuracy comparable to conventional (Likert scale) rating approaches, but in only a fraction of the time.

[1]  Matthew J. Salganik,et al.  Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market , 2006, Science.

[2]  Wolfgang Jank,et al.  Second-Generation Prediction Markets for Information Aggregation: A Comparison of Payoff Mechanisms , 2009 .

[3]  G. Jouret.  Inside Cisco's Search for the Next Big Idea , 2009 .

[4]  Thomas A. Rietz,et al.  Results from a Dozen Years of Election Futures Markets Research , 2008 .

[5]  L. Jeppesen,et al.  Getting Unusual Suspects to Solve R&D Puzzles , 2007 .

[6]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[7]  J. West,et al.  Open innovation : researching a new paradigm , 2008 .

[8]  Lukas Biewald,et al.  Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing , 2011, Human Computation.

[9]  David M. Pennock,et al.  Yoopick: A Combinatorial Sports Prediction Market , 2008, AAAI.

[10]  Eric Horvitz,et al.  What's your idea?: a case study of a grassroots innovation pipeline within a large software company , 2010, CHI.

[11]  K. Arrow.  Social Choice and Individual Values , 1951 .

[12]  Thomas A. Rietz,et al.  Information Systems Frontiers 5:1, 79–93, 2003 C ○ 2003 Kluwer Academic Publishers. Manufactured in The Netherlands. Prediction Markets as Decision Support Systems , 2022 .

[13]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[14]  Jock Given,et al.  The wealth of networks: How social production transforms markets and freedom , 2007, Inf. Econ. Policy.

[15]  Christoph Hienerth,et al.  The Wisdom of the Crowd vs . Expert Evaluation : A Conceptualization of Evaluation Validity , 2013 .

[16]  Christian Bason,et al.  Leading Public Sector Innovation: Co-creating for a Better Society , 2010 .

[17]  A. Tversky,et al.  Choices, Values, and Frames , 2000 .

[18]  Tom Fawcett,et al.  ROC Graphs: Notes and Practical Considerations for Researchers , 2007 .

[19]  Mark Klein,et al.  Enabling Large-Scale Deliberation Using Attention-Mediation Metrics , 2011, Computer Supported Cooperative Work (CSCW).

[20]  Alison Lee,et al.  Crowd Sourcing and Prediction Markets , 2011 .

[21]  Maozhen Li,et al.  A survey of emerging approaches to spam filtering , 2012, CSUR.

[22]  熊谷 ユリヤ,et al.  James Surowiecki, 『The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations』, Random House, 5,2004, $24.95 , 2005 .

[23]  Pernille Hemmer,et al.  The Wisdom of Crowds in Rank Ordering Problems , 2009 .

[24]  Jan Marco Leimeister,et al.  The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities , 2012, Int. J. Electron. Commer..

[25]  Michael S. Bernstein,et al.  The future of crowd work , 2013, CSCW.

[26]  Karim R. Lakhani,et al.  Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis , 2011, Manag. Sci..

[27]  Edmund A. Mennis.  The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations , 2006 .

[28]  Charles A. Holt,et al.  Risk Aversion and Incentive Effects: New Data without Order Effects , 2005 .

[29]  David C. Webb,et al.  Incentive design under loss aversion , 2007 .

[30]  Vassilis Kostakos,et al.  Is the Crowd's Wisdom Biased? A Quantitative Analysis of Three Online Communities , 2009, 2009 International Conference on Computational Science and Engineering.

[31]  George Wright,et al.  Expert Opinions in Forecasting: The Role of the Delphi Technique , 2001 .

[32]  Justin Wolfers,et al.  Using Prediction Markets to Track Information Flows: Evidence from Google , 2009, AMMA.

[33]  Rachel Greenstadt,et al.  Learning to Extract Quality Discourse in Online Communities , 2010, Collaboratively-Built Knowledge Sources and AI.

[34]  Andrea Back,et al.  A Text Mining Approach to Evaluate Submissions to Crowdsourcing Contests , 2013, 2013 46th Hawaii International Conference on System Sciences.

[35]  KleinMark.  Enabling Large-Scale Deliberation Using Attention-Mediation Metrics , 2012 .

[36]  L. V. Williams,et al.  Prediction Markets , 2003 .

[37]  Steven Gjerstad.  Risk Aversion, Beliefs, and Prediction Market Equilibrium , 2004 .

[38]  O. Bjelland,et al.  An Inside View of IBM's 'Innovation Jam' , 2008 .

[39]  Paul Michael Di Gangi,et al.  Steal my idea! Organizational adoption of user innovations from a user innovation community: A case study of Dell IdeaStorm , 2009, Decis. Support Syst..

[40]  Jan Marco Leimeister,et al.  Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters? , 2011, ICIS.

[41]  John Cullen,et al.  Democratizing Innovation , 2020, Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship.

[42]  Yi Xu,et al.  Innovation Contests, Open Innovation, and Multiagent Problem Solving , 2008, Manag. Sci..

[43]  Richard D. Wang,et al.  Tournaments for Ideas , 2010 .