Testing tournament selection in creative problem solving using crowds

We tested a technique for creative problem solving, which used crowd-based genetic algorithms; one crowd generated initial ideas, another crowd evaluated the quality of these ideas, and yet another crowd combined pairs of ideas selected by the computer. The pairs were selected through a tournament method, in which two ideas chosen were biased toward higher quality. To test the technique, we asked a crowd to evaluate a subset of 468 solutions for the 2010 oil spill in the Gulf of Mexico produced by another crowd of 1853 individuals, and 311 solutions by 311 experts. The crowd evaluated the most creative crowd solutions as creative as the most creative expert solutions. Moreover, tournament selection led to greater improvement in the creativity of combined solutions than random selection, in which two solutions were chosen randomly. Creative problem solving using crowd-based genetic algorithms can work with good design.

[1]  S. Asch.  Effects of Group Pressure Upon the Modification and Distortion of Judgments , 1951 .

[2]  J. Guilford,et al.  The nature of human intelligence. , 1968 .

[3]  J. Davitz,et al.  A survey of studies contrasting the quality of group performance and individual performance, 1920-1957. , 1958, Psychological bulletin.

[4]  Alex Kosorukoff,et al.  Human based genetic algorithm , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[5]  Eytan Adar.  Why I Hate Mechanical Turk Research (and Workshops) , 2011 .

[6]  T. B. Ward.  Cognition, Creativity, and Entrepreneurship , 2004 .

[7]  B. Nijstad,et al.  Relative accessibility of domain knowledge and creativity: The effects of knowledge activation on the quantity and originality of generated ideas , 2007 .

[8]  Roni Reiter-Palmon,et al.  Encyclopedia of Creativity , 2011 .

[9]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[10]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[11]  Manuel Blum,et al.  Verbosity: a game for collecting common-sense facts , 2006, CHI.

[12]  G. Clore,et al.  Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. , 1983 .

[13]  Jeffrey V. Nickerson,et al.  Crowdsourcing Creativity: Combining Ideas in Networks , 2010 .

[14]  Jay F. Nunamaker,et al.  ELECTRONIC BRAINSTORMING AND GROUP SIZE , 1992 .

[15]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[16]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[17]  Michael D. Mumford,et al.  Process-based measures of creative problem-solving skills: IV. Category combination. Creativity Research Rips, LJ (1995). The current status of research on concept combination , 1997 .

[18]  Dale T. Miller,et al.  Combining Social Concepts: The Role of Causal Reasoning , 1990, Cogn. Sci..

[19]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[20]  Paul Thagard,et al.  Conceptual Combination and Scientific Discovery , 1984, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[21]  Jeffrey V. Nickerson,et al.  Structures for Creativity: The crowdsourcing of design , 2011 .

[22]  Ruey-Lin Hsiao.  Knowledge Sharing in a Global Professional Service Firm , 2008, MIS Q. Executive.

[23]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[24]  Alexandra Durcikova,et al.  How Knowledge Validation Processes Affect Knowledge Contribution , 2009, J. Manag. Inf. Syst..

[25]  Yochai Benkler,et al.  Coase's Penguin, or Linux and the Nature of the Firm , 2001, ArXiv.

[26]  Steven M. Smith,et al.  Creative Cognition: Theory, Research, and Applications , 1996 .

[27]  Ines Solomon.  Analogical Transfer and “Functional Fixedness” in the Science Classroom , 1994 .

[28]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[29]  M. Mumford.  Where Have We Been, Where Are We Going? Taking Stock in Creativity Research , 2003 .

[30]  James Surowiecki.  The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[31]  Yochai Benkler,et al.  The wealth of networks: how social production transforms markets and freedom , 2006 .

[32]  E. Salas,et al.  Productivity loss in brainstorming groups: A meta-analytic integration. , 1991 .

[33]  Robert Heckman,et al.  A Content Analytic Comparison of Learning Processes in Online and Face-to-Face Case Study Discussions , 2006, J. Comput. Mediat. Commun..

[34]  Jeffrey V. Nickerson,et al.  Evaluating Design Solutions Using Crowds , 2011, AMCIS.

[35]  Jeffrey V. Nickerson,et al.  Cooks or cobblers?: crowd creativity through combination , 2011, CHI.

[36]  G. Clore,et al.  Feelings and phenomenal experiences , 1996 .

[37]  Klarissa Ting-Ting Chang,et al.  Psychological Contracts and Knowledge Exchange in Virtual Teams , 2008, ICIS.

[38]  Lydia B. Chilton,et al.  Exploring iterative and parallel human computation processes , 2010, HCOMP '10.

[39]  Paul Thagard,et al.  The AHA! Experience: Creativity Through Emergent Binding in Neural Networks , 2011, Cogn. Sci..

[40]  K. Duncker,et al.  On problem-solving , 1945 .

[41]  Yasuaki Sakamoto,et al.  Feature Selection in Crowd Creativity , 2011, HCI.

[42]  Gerald C. Kane,et al.  Information Technology and Organizational Learning: An Investigation of Exploration and Exploitation Processes , 2007, Organ. Sci..

[43]  Yasuaki Sakamoto,et al.  Conceptual Combination versus Critical Combination: Devising Creative Solutions using the Sequential Application of Crowds , 2011, CogSci.

[44]  Roger B. Dannenberg,et al.  TagATune: A Game for Music and Sound Annotation , 2007, ISMIR.

[45]  M. J. Wilkenfeld,et al.  Similarity and emergence in conceptual combination , 2001 .

[46]  T. B. Ward,et al.  The Emergence of Novel Attributes in Concept Modification , 2002 .

[47]  Suzanne D. Pawlowski,et al.  The Delphi method as a research tool: an example, design considerations and applications , 2004, Inf. Manag..

[48]  Benjamin B. Bederson,et al.  Human computation: a survey and taxonomy of a growing field , 2011, CHI.

[49]  Jyotsna Vaid,et al.  Creative Thought: An Investigation of Conceptual Structures and Processes , 2001 .

[50]  Andrea Back,et al.  Group wisdom support systems: Aggregating the insights of many through information technology , 2008 .

[51]  K. Dugosh,et al.  Cognitive stimulation in brainstorming. , 2000, Journal of personality and social psychology.

[52]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[53]  A. Osborn.  Applied imagination : principles and procedures of creative problem-solving , 1957 .

[54]  Jeffrey V. Nickerson,et al.  Generating Creative Ideas Through Crowds: An Experimental Study of Combination , 2011, ICIS.

[55]  David A. Forsyth,et al.  Utility data annotation with Amazon Mechanical Turk , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[56]  Wai Fong Boh,et al.  Expertise and Collaboration in the Geographically Dispersed Organization , 2007, Organ. Sci..

[57]  Manuel Blum,et al.  Peekaboom: a game for locating objects in images , 2006, CHI.