Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts

Creative coding tasks are often exploratory in nature. When producing digital artwork, artists usually begin with a high-level semantic construct such as a"stained glass filter"and programmatically implement it by varying code parameters such as shape, color, lines, and opacity to produce visually appealing results. Based on interviews with artists, it can be effortful to translate semantic constructs to program syntax, and current programming tools don't lend well to rapid creative exploration. To address these challenges, we introduce Spellburst, a large language model (LLM) powered creative-coding environment. Spellburst provides (1) a node-based interface that allows artists to create generative art and explore variations through branching and merging operations, (2) expressive prompt-based interactions to engage in semantic programming, and (3) dynamic prompt-driven interfaces and direct code editing to seamlessly switch between semantic and syntactic exploration. Our evaluation with artists demonstrates Spellburst's potential to enhance creative coding practices and inform the design of computational creativity tools that bridge semantic and syntactic spaces.

[1]  Sarah E. Chasins,et al.  Understanding Version Control as Material Interaction with Quickpose , 2023, CHI.

[2]  Mark A. Lemley,et al.  Foundation Models and Fair Use , 2023, SSRN Electronic Journal.

[3]  Carrie J. Cai,et al.  The Prompt Artists , 2023, Creativity & Cognition.

[4]  Carrie J. Cai,et al.  A Word is Worth a Thousand Pictures: Prompts as AI Design Material , 2023, ArXiv.

[5]  Douglas C. Schmidt,et al.  A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT , 2023, ArXiv.

[6]  E. Paulos,et al.  Towards Creative Version Control , 2022, Proc. ACM Hum. Comput. Interact..

[7]  E. Horvitz,et al.  Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming , 2022, ArXiv.

[8]  Yuanzhen Li,et al.  DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Amit H. Bermano,et al.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion , 2022, ICLR.

[10]  N. Polikarpova,et al.  Grounded Copilot: How Programmers Interact with Code-Generating Models , 2022, Proc. ACM Program. Lang..

[11]  Alberto A. P. Cattaneo,et al.  Mixplorer: Scaffolding Design Space Exploration through Genetic Recombination of Multiple Peoples’ Designs to Support Novices’ Creativity , 2022, CHI.

[12]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Lydia B. Chilton,et al.  Design Guidelines for Prompt Engineering Text-to-Image Generative Models , 2021, CHI.

[14]  Robert DeLine,et al.  Fork It: Supporting Stateful Alternatives in Computational Notebooks , 2021, CHI.

[15]  Emily M. Bender,et al.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 , 2021, FAccT.

[16]  Jingyi Li,et al.  What We Can Learn From Visual Artists About Software Development , 2021, CHI.

[17]  B. Gaut,et al.  Creativity and Imagination , 2021 .

[18]  James McCann,et al.  Design Adjectives: A Framework for Interactive Model-Guided Exploration of Parameterized Design Spaces , 2020, UIST.

[19]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[20]  Daniel Cardoso Llach,et al.  Stamper: An Artboard-Oriented Creative Coding Environment , 2020, CHI Extended Abstracts.

[21]  Piyush Agrawal,et al.  Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network , 2019, ArXiv.

[22]  S. Shamay-Tsoory,et al.  The two-fold model of creativity: the neural underpinnings of the generation and evaluation of creative ideas , 2019, Current Opinion in Behavioral Sciences.

[23]  Peter Dalsgaard,et al.  Mapping the Landscape of Creativity Support Tools in HCI , 2019, CHI.

[24]  Kesler Tanner,et al.  Poirot: A Web Inspector for Designers , 2019, CHI.

[25]  Luis Guerrero,et al.  Adobe , 2018, The Encyclopedia of Archaeological Sciences.

[26]  Radomír Mech,et al.  Extending Manual Drawing Practices with Artist-Centric Programming Tools , 2018, CHI.

[27]  Stefan Palan,et al.  Prolific.ac—A subject pool for online experiments , 2017 .

[28]  Brad A. Myers,et al.  Exploring exploratory programming , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[29]  Michael Tempel.  Generative art for all , 2017 .

[30]  Brad A. Myers,et al.  Variolite: Supporting Exploratory Programming by Data Scientists , 2017, CHI.

[31]  Takeo Igarashi,et al.  Micro-Versioning Tool to Support Experimentation in Exploratory Programming , 2017, CHI.

[32]  Radomír Mech,et al.  Supporting Expressive Procedural Art Creation through Direct Manipulation , 2017, CHI.

[33]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Aaron D. Knochel,et al.  If Art Education Then Critical Digital Making: Computational Thinking and Creative Code , 2015 .

[35]  Naghmi Shireen,et al.  GEM-NI: A System for Creating and Managing Alternatives In Generative Design , 2015, CHI.

[36]  Dhruva R. Chakrabarti,et al.  Atlas , 2014, OOPSLA.

[37]  Daniel Shiffman,et al.  The Nature of Code , 2012, How to Engineer Software.

[38]  John Maloney,et al.  The Scratch Programming Language and Environment , 2010, TOCE.

[39]  John Zimmerman,et al.  How to support designers in getting hold of the immaterial material of software , 2010, CHI.

[40]  Scott R. Klemmer,et al.  Design as exploration: creating interface alternatives through parallel authoring and runtime tuning , 2008, UIST '08.

[41]  R. Gibbs.  The Cambridge Handbook of Metaphor and Thought , 2008 .

[42]  C. Forceville.  Metaphor in pictures and multimodal representations , 2008 .

[43]  Ben Shneiderman,et al.  Creativity support tools: accelerating discovery and innovation , 2007, CACM.

[44]  Mark Turner,et al.  The Artful Mind: Cognitive Science and the Riddle of Human Creativity , 2006 .

[45]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[46]  B. Nijstad,et al.  Cognitive stimulation and interference in groups: Exposure effects in an idea generation task , 2002 .

[47]  B. Shneiderman.  Creativity support tools , 2002, CACM.

[48]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[49]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[50]  A. Strauss.  Basics Of Qualitative Research , 1992 .

[51]  Brad A. Myers,et al.  Taxonomies of visual programming and program visualization , 1990, J. Vis. Lang. Comput..

[52]  Items , 1891, Geological Society, London, Special Publications.

[53]  Brayan Stiven Torrres Ovalle.  GitHub Copilot , 2022, Encuentro Internacional de Educación en Ingeniería.

[54]  B. Gaut,et al.  Creativity and Art: Three Roads to Surprise , 2012 .

[55]  Casey Reas,et al.  Processing: a programming handbook for visual designers and artists , 2007 .

[56]  Yasmin B. Kafai,et al.  Creative Coding: Programming for Personal Expression , 2005 .

[57]  S. Halliwell.  The Aesthetics of Mimesis , 2002 .

[58]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .