Fluid Transformers and Creative Analogies: Exploring Large Language Models' Capacity for Augmenting Cross-Domain Analogical Creativity

Cross-domain analogical reasoning is a core creative ability that can be challenging for humans. Recent work has shown some proofs-of concept of Large language Models' (LLMs) ability to generate cross-domain analogies. However, the reliability and potential usefulness of this capacity for augmenting human creative work has received little systematic exploration. In this paper, we systematically explore LLMs capacity to augment cross-domain analogical reasoning. Across three studies, we found: 1) LLM-generated cross-domain analogies were frequently judged as helpful in the context of a problem reformulation task (median 4 out of 5 helpfulness rating), and frequently (~80% of cases) led to observable changes in problem formulations, and 2) there was an upper bound of 25% of outputs bring rated as potentially harmful, with a majority due to potentially upsetting content, rather than biased or toxic content. These results demonstrate the potential utility -- and risks -- of LLMs for augmenting cross-domain analogical creativity.

[1]  Anna A. Ivanova,et al.  Dissociating language and thought in large language models: a cognitive perspective , 2023, ArXiv.

[2]  Qihao Zhu,et al.  Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers , 2022, Journal of Mechanical Design.

[3]  Hiroaki Hayashi,et al.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing , 2021, ACM Comput. Surv..

[4]  Taylor W. Webb,et al.  Emergent Analogical Reasoning in Large Language Models , 2022, ArXiv.

[5]  Michael S. Bernstein,et al.  Evaluating Human-Language Model Interaction , 2022, ArXiv.

[6]  Jinjun Xiong,et al.  Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT , 2022, ArXiv.

[7]  S. Magleby,et al.  A Preliminary Approach to Select an Origami Source Pattern for Deployable Space Arrays , 2022, Volume 7: 46th Mechanisms and Robotics Conference (MR).

[8]  A. Kittur,et al.  Augmenting Scientific Creativity with an Analogical Search Engine , 2022, ACM Trans. Comput. Hum. Interact..

[9]  Ann Yuan,et al.  Wordcraft: Story Writing With Large Language Models , 2022, IUI.

[10]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[11]  Elena L. Glassman,et al.  A Selective Summary of Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence , 2022, IN2WRITING.

[12]  Jianxi Luo,et al.  Generative Pre-Trained Transformer for Design Concept Generation: An Exploration , 2021, Proceedings of the Design Society.

[13]  Lydia B. Chilton,et al.  Sparks: Inspiration for Science Writing using Language Models , 2021, IN2WRITING.

[14]  Tal August,et al.  All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text , 2021, ACL.

[15]  Thomas j Parashos,et al.  Framing Creative Work: Helping Novices Frame Better Problems through Interactive Scaffolding , 2021, Creativity & Cognition.

[16]  Yoichi Ochiai,et al.  BunCho: AI Supported Story Co-Creation via Unsupervised Multitask Learning to Increase Writers’ Creativity in Japanese , 2021, CHI Extended Abstracts.

[17]  Steven P. Dow,et al.  Finding Place in a Design Space , 2021, Proc. ACM Hum. Comput. Interact..

[18]  S. Muresan,et al.  MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding , 2021, NAACL.

[19]  Emily M. Bender,et al.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 , 2021, FAccT.

[20]  Laria Reynolds,et al.  Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm , 2021, CHI Extended Abstracts.

[21]  Wolfgang Effelsberg,et al.  Procedural Generation of Interactive Stories using Language Models , 2020, FDG.

[22]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[23]  Emily Denton,et al.  Social Biases in NLP Models as Barriers for Persons with Disabilities , 2020, ACL.

[24]  Lydia B. Chilton,et al.  How Novelists Use Generative Language Models: An Exploratory User Study , 2020, HAI-GEN+user2agent@IUI.

[25]  Micah B. Goldwater,et al.  Can a relational mindset boost analogical retrieval? , 2019, Cognitive research: principles and implications.

[26]  Lydia B. Chilton,et al.  Metaphoria: An Algorithmic Companion for Metaphor Creation , 2019, CHI.

[27]  Kazjon Grace,et al.  Designing with and for the Crowd: A Cognitive Study of Design Processes in NatureNet , 2018, Design Computing and Cognition '18.

[28]  T. Ormerod,et al.  An Exploration of Real-World Analogical Problem Solving in Novices , 2019, Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society.

[29]  Dafna Shahaf,et al.  31 SOLVENT : A Mixed Initiative System for Finding Analogies between Research Papers , 2018 .

[30]  Noah A. Smith,et al.  Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories , 2018, IUI.

[31]  Krzysztof Z. Gajos,et al.  Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation , 2017, Creativity & Cognition.

[32]  Dafna Shahaf,et al.  Accelerating Innovation Through Analogy Mining , 2017, KDD.

[33]  Thomas L. Griffiths,et al.  Evaluating vector-space models of analogy , 2017, CogSci.

[34]  Steven Dow,et al.  Improving Crowd Innovation with Expert Facilitation , 2016, CSCW.

[35]  Melissa Roemmele,et al.  Writing Stories with Help from Recurrent Neural Networks , 2016, AAAI.

[36]  Krzysztof Z. Gajos,et al.  Providing Timely Examples Improves the Quantity and Quality of Generated Ideas , 2015, Creativity & Cognition.

[37]  John S. Gero,et al.  Interpretation-driven mapping: A framework for conducting search and rerepresentation in parallel for computational analogy in design , 2015, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[38]  C. H. Dorst,et al.  Frame Innovation: Create New Thinking by Design , 2015 .

[39]  Christian D. Schunn,et al.  Do the best design ideas (really) come from conceptually distant sources of inspiration , 2015 .

[40]  Aaron,et al.  Far-Out Thinking: Generating Solutions to Distant Analogies Promotes Relational Thinking , 2014 .

[41]  D. Gentner,et al.  Comparison promotes learning and transfer of relational categories. , 2013, Journal of experimental psychology. Learning, memory, and cognition.

[42]  Jonathan Cagan,et al.  The Meaning of “Near” and “Far”: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output , 2012 .

[43]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[44]  U. Alon,et al.  A quantitative study of creative leaps , 2012, ICCC.

[45]  Jonathan Cagan,et al.  On the benefits and pitfalls of analogies for innovative design : Ideation performance based on analogical distance, commonness, and modality of examples , 2011 .

[46]  Ashok K. Goel,et al.  DANE: Fostering Creativity in and through Biologically Inspired Design , 2011 .

[47]  Daniel L. Schwartz,et al.  Parallel prototyping leads to better design results, more divergence, and increased self-efficacy , 2010, TCHI.

[48]  Jeffrey Loewenstein,et al.  How One's Hook Is Baited Matters for Catching an Analogy , 2010 .

[49]  Kazuhiro Ueda,et al.  Analogical modification in the creation of contemporary art , 2007, Cognitive Systems Research.

[50]  Arthur B. Markman,et al.  Wordtrees: A Method For Design By Analogy , 2008 .

[51]  Arthur B. Markman,et al.  Modality and representation in analogy , 2008, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[52]  Ashok K. Goel,et al.  Compound Analogical Design, Or How to Make a Surfboard Disappear , 2008 .

[53]  Bo T. Christensen,et al.  The relationship of analogical distance to analogical function and preinventive structure: the case of engineering design , 2007, Memory & cognition.

[54]  Arthur B. Markman,et al.  Representing Analogies: Increasing the Probability of Innovation , 2006 .

[55]  D. Dahl,et al.  The Influence and Value of Analogical Thinking during New Product Ideation , 2002 .

[56]  Dedre Gentner,et al.  Analogy in Scientific Discovery: The Case of Johannes Kepler , 2002 .

[57]  Nigel Cross,et al.  Creativity in the design process: co-evolution of problem–solution , 2001 .

[58]  Semyon Savransky,et al.  Engineering of Creativity: Introduction to TRIZ Methodology of Inventive Problem Solving , 2000 .

[59]  Andrew B. Hargadon,et al.  Technology brokering and innovation in a product development firm. , 1997 .

[60]  K. Dunbar How scientists think: On-line creativity and conceptual change in science. , 1997 .

[61]  Charles Cole,et al.  Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought , 1996 .

[62]  K. Holyoak,et al.  Mental Leaps: Analogy in Creative Thought , 1994 .

[63]  Ashok K. Goel,et al.  Innovation in Analogical Design: A Model-Based Approach , 1994 .

[64]  Kenneth D. Forbus,et al.  The Roles of Similarity in Transfer: Separating Retrievability From Inferential Soundness , 1993, Cognitive Psychology.

[65]  Dedre Gentner,et al.  ANALOGICAL REMINDING: A GOOD MATCH IS HARD TO FIND. , 1985 .

[66]  K. Holyoak,et al.  Schema induction and analogical transfer , 1983, Cognitive Psychology.

[67]  Dedre Gentner,et al.  Structure-Mapping: A Theoretical Framework for Analogy , 1983, Cogn. Sci..

[68]  K. Holyoak,et al.  Analogical problem solving , 1980, Cognitive Psychology.

[69]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[70]  H. Rittel,et al.  Dilemmas in a general theory of planning , 1973 .

[71]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[72]  M. Scheerer,et al.  Problem Solving , 1967, Nature.

[73]  W. Gordon Synectics: The Development of Creative Capacity , 1961 .