Fluid Transformers and Creative Analogies: Exploring Large Language Models’ Capacity for Augmenting Cross-Domain Analogical Creativity

Cross-domain analogical reasoning is a core creative ability that can be challenging for humans. Recent work has shown some proofs-of-concept of Large language Models’ (LLMs) ability to generate cross-domain analogies. However, the reliability and potential usefulness of this capacity for augmenting human creative work has received little systematic exploration. In this paper, we systematically explore LLMs capacity to augment cross-domain analogical reasoning. Across three studies, we found: 1) LLM-generated cross-domain analogies were frequently judged as helpful in the context of a problem reformulation task (median 4 out of 5 helpfulness rating), and frequently (∼ 80% of cases) led to observable changes in problem formulations, and 2) there was an upper bound of ∼ 25% of outputs being rated as potentially harmful, with a majority due to potentially upsetting content, rather than biased or toxic content. These results demonstrate the potential utility — and risks — of LLMs for augmenting cross-domain analogical creativity.

[1]  Chen Cao Scaffolding CS1 Courses with a Large Language Model-Powered Intelligent Tutoring System , 2023, IUI Companion.

[2]  Joel Chan,et al.  Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks , 2023, ArXiv.

[3]  Anna A. Ivanova,et al.  Dissociating language and thought in large language models: a cognitive perspective , 2023, ArXiv.

[4]  Qihao Zhu,et al.  Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers , 2022, Journal of Mechanical Design.

[5]  Taylor W. Webb,et al.  Emergent Analogical Reasoning in Large Language Models , 2022, ArXiv.

[6]  Michael S. Bernstein,et al.  Evaluating Human-Language Model Interaction , 2022, ArXiv.

[7]  Jinjun Xiong,et al.  Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT , 2022, ArXiv.

[8]  S. Magleby,et al.  A Preliminary Approach to Select an Origami Source Pattern for Deployable Space Arrays , 2022, Volume 7: 46th Mechanisms and Robotics Conference (MR).

[9]  A. Kittur,et al.  Augmenting Scientific Creativity with an Analogical Search Engine , 2022, ACM Trans. Comput. Hum. Interact..

[10]  Ann Yuan,et al.  Wordcraft: Story Writing With Large Language Models , 2022, IUI.

[11]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[12]  Elena L. Glassman,et al.  A Selective Summary of Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence , 2022, IN2WRITING.

[13]  Jianxi Luo,et al.  Generative Pre-Trained Transformer for Design Concept Generation: An Exploration , 2021, Proceedings of the Design Society.

[14]  Lydia B. Chilton,et al.  Sparks: Inspiration for Science Writing using Language Models , 2021, IN2WRITING.

[15]  Pascale Fung,et al.  Language Models are Few-shot Multilingual Learners , 2021, MRL.

[16]  Hiroaki Hayashi,et al.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing , 2021, ACM Comput. Surv..

[17]  Tal August,et al.  All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text , 2021, ACL.

[18]  Thomas j Parashos,et al.  Framing Creative Work: Helping Novices Frame Better Problems through Interactive Scaffolding , 2021, Creativity & Cognition.

[19]  Yoichi Ochiai,et al.  BunCho: AI Supported Story Co-Creation via Unsupervised Multitask Learning to Increase Writers’ Creativity in Japanese , 2021, CHI Extended Abstracts.

[20]  Steven Dow,et al.  CoNotate: Suggesting Queries Based on Notes Promotes Knowledge Discovery , 2021, CHI.

[21]  Steven P. Dow,et al.  Finding Place in a Design Space , 2021, Proc. ACM Hum. Comput. Interact..

[22]  Steven Dow,et al.  The "Active Search" Hypothesis: How Search Strategies Relate to Creative Learning , 2021, CHIIR.

[23]  S. Muresan,et al.  MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding , 2021, NAACL.

[24]  Emily M. Bender,et al.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 , 2021, FAccT.

[25]  Laria Reynolds,et al.  Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm , 2021, CHI Extended Abstracts.

[26]  Wolfgang Effelsberg,et al.  Procedural Generation of Interactive Stories using Language Models , 2020, FDG.

[27]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[28]  Solon Barocas,et al.  Language (Technology) is Power: A Critical Survey of “Bias” in NLP , 2020, ACL.

[29]  Emily Denton,et al.  Social Biases in NLP Models as Barriers for Persons with Disabilities , 2020, ACL.

[30]  Simo Hosio,et al.  Creativity on Paid Crowdsourcing Platforms , 2020, CHI.

[31]  Micah B. Goldwater,et al.  Can a relational mindset boost analogical retrieval? , 2019, Cognitive research: principles and implications.

[32]  Nanyun Peng,et al.  The Woman Worked as a Babysitter: On Biases in Language Generation , 2019, EMNLP.

[33]  Melissa A. Schilling,et al.  Exploring Uncharted Territory: Knowledge Search Processes in the Origination of Outlier Innovation , 2019, Organization Science.

[34]  Lydia B. Chilton,et al.  Metaphoria: An Algorithmic Companion for Metaphor Creation , 2019, CHI.

[35]  Dafna Shahaf,et al.  31 SOLVENT : A Mixed Initiative System for Finding Analogies between Research Papers , 2018 .

[36]  Kazjon Grace,et al.  Designing with and for the Crowd: A Cognitive Study of Design Processes in NatureNet , 2018, Design Computing and Cognition '18.

[37]  Noah A. Smith,et al.  Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories , 2018, IUI.

[38]  Krzysztof Z. Gajos,et al.  Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation , 2017, Creativity & Cognition.

[39]  Dafna Shahaf,et al.  Accelerating Innovation Through Analogy Mining , 2017, KDD.

[40]  Thomas L. Griffiths,et al.  Evaluating vector-space models of analogy , 2017, CogSci.

[41]  Steven Dow,et al.  Improving Crowd Innovation with Expert Facilitation , 2016, CSCW.

[42]  Melissa Roemmele,et al.  Writing Stories with Help from Recurrent Neural Networks , 2016, AAAI.

[43]  C. Schunn,et al.  The importance of iteration in creative conceptual combination , 2015, Cognition.

[44]  Krzysztof Z. Gajos,et al.  Providing Timely Examples Improves the Quantity and Quality of Generated Ideas , 2015, Creativity & Cognition.

[45]  John S. Gero,et al.  Interpretation-driven mapping: A framework for conducting search and rerepresentation in parallel for computational analogy in design , 2015, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[46]  C. H. Dorst,et al.  Frame Innovation: Create New Thinking by Design , 2015 .

[47]  Aniket Kittur,et al.  Distributed analogical idea generation: inventing with crowds , 2014, CHI.

[48]  D. Gentner,et al.  Comparison promotes learning and transfer of relational categories. , 2013, Journal of experimental psychology. Learning, memory, and cognition.

[49]  Jonathan Cagan,et al.  The Meaning of “Near” and “Far”: The Impact of Structuring Design Databases and the Effect of Distance of Analogy on Design Output , 2012 .

[50]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[51]  Jonathan Cagan,et al.  On the benefits and pitfalls of analogies for innovative design : Ideation performance based on analogical distance, commonness, and modality of examples , 2011 .

[52]  Daniel L. Schwartz,et al.  Parallel prototyping leads to better design results, more divergence, and increased self-efficacy , 2010, TCHI.

[53]  Arthur B. Markman,et al.  Wordtrees: A Method For Design By Analogy , 2008 .

[54]  Arthur B. Markman,et al.  Modality and representation in analogy , 2008, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[55]  Kazuhiro Ueda,et al.  Analogical modification in the creation of contemporary art , 2007, Cognitive Systems Research.

[56]  Corina Sas,et al.  Why bad ideas are a good idea , 2006 .

[57]  D. Dahl,et al.  The Influence and Value of Analogical Thinking during New Product Ideation , 2002 .

[58]  Nigel Cross,et al.  Creativity in the design process: co-evolution of problem–solution , 2001 .

[59]  Semyon Savransky,et al.  Engineering of Creativity: Introduction to TRIZ Methodology of Inventive Problem Solving , 2000 .

[60]  S. Ohlsson,et al.  Constraint relaxation and chunk decomposition in insight problem solving , 1999 .

[61]  Nigel Cross,et al.  Expertise in engineering design , 1998 .

[62]  Andrew B. Hargadon,et al.  Technology brokering and innovation in a product development firm. , 1997 .

[63]  Charles Cole,et al.  Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought , 1996 .

[64]  Kenneth D. Forbus,et al.  The Roles of Similarity in Transfer: Separating Retrievability From Inferential Soundness , 1993, Cognitive Psychology.

[65]  Vinod Goel,et al.  The Structure of Design Problem Spaces , 1992, Cogn. Sci..

[66]  Craig A. Kaplan,et al.  In search of insight , 1990, Cognitive Psychology.

[67]  Dedre Gentner,et al.  ANALOGICAL REMINDING: A GOOD MATCH IS HARD TO FIND. , 1985 .

[68]  R. J. Bogumil,et al.  The reflective practitioner: How professionals think in action , 1985, Proceedings of the IEEE.

[69]  K. Holyoak,et al.  Schema induction and analogical transfer , 1983, Cognitive Psychology.

[70]  K. Holyoak,et al.  Analogical problem solving , 1980, Cognitive Psychology.

[71]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[72]  Howard E. Gruber,et al.  Darwin on man. A psychological study of scientific creativity, together with Darwin's early and unpublished notebooks , 1975, Medical History.

[73]  H. Rittel,et al.  Dilemmas in a general theory of planning , 1973 .

[74]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[75]  Lydia B. Chilton,et al.  How Novelists Use Generative Language Models: An Exploratory User Study , 2020, HAI-GEN+user2agent@IUI.

[76]  T. Ormerod,et al.  An Exploration of Real-World Analogical Problem Solving in Novices , 2019, Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society.

[77]  Christian D. Schunn,et al.  Do the best design ideas (really) come from conceptually distant sources of inspiration , 2015 .

[78]  Aaron,et al.  Far-Out Thinking: Generating Solutions to Distant Analogies Promotes Relational Thinking , 2014 .

[79]  U. Alon,et al.  A quantitative study of creative leaps , 2012, ICCC.

[80]  Ashok K. Goel,et al.  DANE: Fostering Creativity in and through Biologically Inspired Design , 2011 .

[81]  Jeffrey Loewenstein,et al.  How One's Hook Is Baited Matters for Catching an Analogy , 2010 .

[82]  Ashok K. Goel,et al.  Compound Analogical Design, Or How to Make a Surfboard Disappear , 2008 .

[83]  Bo T. Christensen,et al.  The relationship of analogical distance to analogical function and preinventive structure: the case of engineering design , 2007, Memory & cognition.

[84]  Arthur B. Markman,et al.  Representing Analogies: Increasing the Probability of Innovation , 2006 .

[85]  Dedre Gentner,et al.  Analogy in Scientific Discovery: The Case of Johannes Kepler , 2002 .

[86]  K. Dunbar How scientists think: On-line creativity and conceptual change in science. , 1997 .

[87]  K. Holyoak,et al.  Mental Leaps: Analogy in Creative Thought , 1994 .

[88]  Ashok K. Goel,et al.  Innovation in Analogical Design: A Model-Based Approach , 1994 .

[89]  Louis L. Bucciarelli,et al.  Designing Engineers , 1994 .

[90]  Dedre Gentner,et al.  Structure-Mapping: A Theoretical Framework for Analogy , 1983, Cogn. Sci..

[91]  M. Scheerer,et al.  Problem Solving , 1967, Nature.

[92]  W. Gordon Synectics: The Development of Creative Capacity , 1961 .