Comparing Different Sensemaking Approaches for Large-Scale Ideation

Large-scale idea generation platforms often expose ideators to previous ideas. However, research suggests people generate better ideas if they see abstracted solution paths (e.g., descriptions of solution approaches generated through human sensemaking) rather than being inundated with all prior ideas. Automated and semi-automated methods can also offer interpretations of earlier ideas. To benefit from sensemaking in practice with limited resources, ideation platform developers need to weigh the cost-quality tradeoffs of different methods for surfacing solution paths. To explore this, we conducted an online study where 245 participants generated ideas for two problems in one of five conditions: 1) no stimuli, 2) exposure to all prior ideas, or solution paths extracted from prior ideas using 3) a fully automated workflow, 4) a hybrid human-machine approach, and 5) a fully manual approach. Contrary to expectations, human-generated paths did not improve ideation (as meas-ured by fluency and breadth of ideation) over simply showing all ideas. Machine-generated paths sometimes significantly improved fluency and breadth of ideation over no ideas (although at some cost to idea quality). These findings suggest that automated sensemaking can improve idea generation, but we need more research to understand the value of human sensemaking for crowd ideation.

[1]  Adriana Kovashka,et al.  Discovering Attribute Shades of Meaning with the Crowd , 2014, International Journal of Computer Vision.

[2]  Herbert A. Simon,et al.  THE MIND'S EYE IN CHESS , 1988 .

[3]  Yi Xu,et al.  Innovation Contests, Open Innovation, and Multiagent Problem Solving , 2008, Manag. Sci..

[4]  K. Boudreau,et al.  'Open' Disclosure of Innovations, Incentives and Follow-on Reuse: Theory on Processes of Cumulative Innovation and a Field Experiment in Computational Biology , 2015 .

[5]  T. M. Amabile.  The social psychology of creativity: A componential conceptualization. , 1983 .

[6]  Joost Duflou,et al.  Systematic innovation through patent based product aspect analysis , 2011 .

[7]  David M. Blei,et al.  Probabilistic topic models , 2012, Commun. ACM.

[8]  Steven M. Smith,et al.  Constraining effects of examples in a creative generation task , 1993, Memory & cognition.

[9]  Eric Horvitz,et al.  What's your idea?: a case study of a grassroots innovation pipeline within a large software company , 2010, CHI.

[10]  D. Gentner,et al.  On Mental Leaps: Analogy in Creative Thought (Keith J. Holyoak and Paul Thagard) , 1996 .

[11]  Krzysztof Z. Gajos,et al.  Toward Collaborative Ideation at Scale: Leveraging Ideas from Others to Generate More Creative and Diverse Ideas , 2015, CSCW.

[12]  Christian D. Schunn,et al.  The Impact of Analogies on Creative Concept Generation: Lessons From an In Vivo Study in Engineering Design , 2015, Cogn. Sci..

[13]  Krzysztof Z. Gajos,et al.  Providing Timely Examples Improves the Quantity and Quality of Generated Ideas , 2015, Creativity & Cognition.

[14]  Jinfeng Yi,et al.  Semi-Crowdsourced Clustering: Generalizing Crowd Labeling by Robust Distance Metric Learning , 2012, NIPS.

[15]  Jonathan Cagan,et al.  A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty , 2010 .

[16]  Steven Dow,et al.  Improving Crowd Innovation with Expert Facilitation , 2016, CSCW.

[17]  K. Holyoak,et al.  Schema induction and analogical transfer , 1983, Cognitive Psychology.

[18]  Wai-Tat Fu,et al.  Idea Visibility, Information Diversity, and Idea Integration in Electronic Brainstorming , 2011, HCI.

[19]  Scott R. Klemmer,et al.  Early and Repeated Exposure to Examples Improves Creative Work , 2012, CogSci.

[20]  Lydia B. Chilton,et al.  Cascade: crowdsourcing taxonomy creation , 2013, CHI.

[21]  S. Atkinson.  Explaining Creativity: The Science of Human Innovation , 2007 .

[22]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[23]  Jon Kolko.  A Theory of Synthesis , 2010 .

[24]  Donald A. Hantula,et al.  The medium matters: Mining the long-promised merit of group interaction in creative idea generation tasks in a meta-analysis of the electronic group brainstorming literature , 2007, Comput. Hum. Behav..

[25]  Jonathan Cagan,et al.  Fixation or inspiration? A meta-analytic review of the role of examples on design processes , 2015 .

[26]  K. Holyoak,et al.  Mental Leaps: Analogy in Creative Thought , 1994 .

[27]  D. Dahl,et al.  The Influence and Value of Analogical Thinking during New Product Ideation , 2002 .

[28]  Claudia Eckert,et al.  Fortune Favours Only the Prepared Mind: Why Sources of Inspiration are Essential for Continuing Creativity , 1998 .

[29]  J. Isaiah Harbison,et al.  Automated scoring of originality using semantic representations , 2014, CogSci.

[30]  Brian P. Bailey,et al.  Getting inspired!: understanding how and why examples are used in creative design practice , 2009, CHI.

[31]  Kevin Dunbar,et al.  Creativity Evaluation through Latent Semantic Analysis , 2009 .

[32]  D L Medin,et al.  Concepts and conceptual structure. , 1989, The American psychologist.

[33]  Adam Tauman Kalai,et al.  Adaptively Learning the Crowd Kernel , 2011, ICML.

[34]  J. Guilford.  The structure of intellect. , 1956, Psychological bulletin.

[35]  Olivier Toubia,et al.  Improving Online Idea Generation Platforms and Customizing the Task Structure on the Basis of Consumers' Domain-Specific Knowledge , 2015 .

[36]  K. Grauman,et al.  Discovering Shades of Attribute Meaning with the Crowd , 2014 .

[37]  Yisong Yue,et al.  Personalized collaborative clustering , 2014, WWW.

[38]  Aniket Kittur,et al.  Distributed analogical idea generation: inventing with crowds , 2014, CHI.

[39]  Karen Holtzblatt,et al.  Contextual design: using customer work models to drive systems design , 1996, CHI Extended Abstracts.

[40]  Filip Krynicki,et al.  Methods and models for the quantitative analysis of crowd brainstorming , 2014 .

[41]  J. Isaiah Harbinson,et al.  Automated scoring of originality using semantic representations , 2014 .

[42]  Jon Kolko,et al.  Exposing the Magic of Design: A Practitioner's Guide to the Methods and Theory of Synthesis , 2011 .

[43]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[44]  Karim R Lakhani,et al.  Using the crowd as an innovation partner. , 2013, Harvard business review.

[45]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[46]  Pedro Antunes,et al.  Attention-Based Management of Information Flows in Synchronous Electronic Brainstorming , 2009, CRIWG.

[47]  Lydia B. Chilton,et al.  Community Clustering: Leveraging an Academic Crowd to Form Coherent Conference Sessions , 2013, HCOMP.

[48]  Judith Gebauer,et al.  The Impact of User Interface Design on Idea Integration in Electronic Brainstorming: An Attention-Based View , 2013, J. Assoc. Inf. Syst..

[49]  Chong Wang,et al.  Reading Tea Leaves: How Humans Interpret Topic Models , 2009, NIPS.

[50]  Aniket Kittur,et al.  Apolo: making sense of large network data by combining rich user interaction and machine learning , 2011, CHI.

[51]  Ronen I. Brafman,et al.  Designing with interactive example galleries , 2010, CHI.

[52]  Christoph Meinel,et al.  Tagging User Research Data: How to Support the Synthesis of Information in Design Teams , 2015 .

[53]  M. Klein,et al.  A Roadmap for Open Innovation Systems , 2015 .

[54]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[55]  Jonathan Cagan,et al.  The Meaning of “Near” and “Far”: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output , 2012 .

[56]  Steven M. Smith,et al.  Metrics for measuring ideation effectiveness , 2003 .

[57]  Robert L. Goldstone,et al.  Innovation, imitation, and problem-solving in a networked group. , 2011, Nonlinear dynamics, psychology, and life sciences.

[58]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[59]  Adam Tauman Kalai,et al.  Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons , 2015, HCOMP.

[60]  Jay F. Nunamaker,et al.  On the Measurement of Ideation Quality , 2007, J. Manag. Inf. Syst..

[61]  John T. Stasko,et al.  Jigsaw: Supporting Investigative Analysis through Interactive Visualization , 2007, 2007 IEEE Symposium on Visual Analytics Science and Technology.

[62]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[63]  Cynthia M. Sifonis,et al.  The Role of Specificity and Abstraction in Creative Idea Generation , 2004 .

[64]  Aniket Kittur,et al.  Crowd synthesis: extracting categories and clusters from complex data , 2014, CSCW.

[65]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .