Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools

AI-based design tools are proliferating in professional software to assist engineering and industrial designers in complex manufacturing and design tasks. These tools take on more agentic roles than traditional computer-aided design tools and are often portrayed as “co-creators.” Yet, working effectively with such systems requires different skills than working with complex CAD tools alone. To date, we know little about how engineering designers learn to work with AI-based design tools. In this study, we observed trained designers as they learned to work with two AI-based tools on a realistic design task. We find that designers face many challenges in learning to effectively co-create with current systems, including challenges in understanding and adjusting AI outputs and in communicating their design goals. Based on our findings, we highlight several design opportunities to better support designer-AI co-creation.

[1]  Luke M. Guerdan,et al.  Towards a Learner-Centered Explainable AI: Lessons from the learning sciences , 2022, ArXiv.

[2]  Philip J. Guo,et al.  "There’s no way to keep up!": Diverse Motivations and Challenges Faced by Informal Learners of ML , 2022, 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[3]  G. Fitzmaurice,et al.  Supercharging Trial-and-Error for Learning Complex Software Applications , 2022, CHI.

[4]  Justin D. Weisz,et al.  Investigating Explainability of Generative AI for Code through Scenario-based Design , 2022, IUI.

[5]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  David Sontag,et al.  Teaching Humans When To Defer to a Classifier via Examplars , 2021, AAAI.

[7]  Adam Perer,et al.  Characterizing Human Explanation Strategies to Inform the Design of Explainable AI for Building Damage Assessment , 2021, ArXiv.

[8]  Aldair E. Gongora,et al.  Designing Composites with Target Effective Young’s Modulus using Reinforcement Learning , 2021, SCF.

[9]  David A. Joyner,et al.  Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant , 2021, CHI.

[10]  Alec Radford,et al.  Zero-Shot Text-to-Image Generation , 2021, ICML.

[11]  Rui Zhang,et al.  "An Ideal Human" , 2021, Proc. ACM Hum. Comput. Interact..

[12]  Guy Hoffman,et al.  Design Intention Inference for Virtual Co-Design Agents , 2020, IVA.

[13]  Andrea Bunt,et al.  “I Would Just Ask Someone”: Learning Feature-Rich Design Software in the Modern Workplace , 2020, 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[14]  Humphrey Yang,et al.  SimuLearn: Fast and Accurate Simulator to Support Morphing Materials Design and Workflows , 2020, UIST.

[15]  Vincent Aleven,et al.  A Conceptual Framework for Human–AI Hybrid Adaptivity in Education , 2020, AIED.

[16]  Seiichi Uchida,et al.  Guided neural style transfer for shape stylization , 2020, PloS one.

[17]  Ben Shneiderman,et al.  From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People , 2020, CHI Extended Abstracts.

[18]  Andrea Bunt,et al.  Learning Through Exploration: How Children, Adults, and Older Adults Interact with a New Feature-Rich Application , 2020, CHI.

[19]  Nur Yildirim,et al.  Digital Fabrication Tools at Work: Probing Professionals' Current Needs and Desired Futures , 2020, CHI.

[20]  Cheng-Zhi Anna Huang,et al.  Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models , 2020, CHI.

[21]  Greg Mori,et al.  House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generation , 2020, ECCV.

[22]  Lauren Wilcox,et al.  "Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making , 2019, Proc. ACM Hum. Comput. Interact..

[23]  Eric Horvitz,et al.  Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance , 2019, HCOMP.

[24]  Eric Horvitz,et al.  Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff , 2019, AAAI.

[25]  Joseph A. Allen,et al.  Understanding Team Learning Dynamics Over Time , 2019, Front. Psychol..

[26]  Virginia Braun,et al.  Reflecting on reflexive thematic analysis , 2019, Qualitative Research in Sport, Exercise and Health.

[27]  Catherine C. Chase,et al.  How teacher talk guidance during Invention activities shapes students’ cognitive engagement and transfer , 2019, International Journal of STEM Education.

[28]  Paul N. Bennett,et al.  Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems , 2019, CHI.

[29]  Vicente Ordonez,et al.  Text2Scene: Generating Compositional Scenes From Textual Descriptions , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Hyunseung Bang,et al.  Side-by-Side Human–Computer Design Using a Tangible User Interface , 2018, Design Computing and Cognition '18.

[31]  Xiang 'Anthony' Chen,et al.  Forte: User-Driven Generative Design , 2018, CHI.

[32]  Tovi Grossman,et al.  Dream Lens: Exploration and Visualization of Large-Scale Generative Design Datasets , 2018, CHI.

[33]  Rubaiat Habib Kazi,et al.  DreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design , 2017, UIST.

[34]  Ahmed K. Noor,et al.  AI and the Future of the Machine Design , 2017 .

[35]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[36]  Matthias Scheutz,et al.  A Framework for Developing and Using Shared Mental Models in Human-Agent Teams , 2017 .

[37]  Travis J. Wiltshire,et al.  Technology as Teammate: Examining the Role of External Cognition in Support of Team Cognitive Processes , 2016, Front. Psychol..

[38]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Alex Endert,et al.  InterAxis: Steering Scatterplot Axes via Observation-Level Interaction , 2016, IEEE Transactions on Visualization and Computer Graphics.

[40]  Levent Burak Kara,et al.  DMS2015-33: Generative interface structure design for supporting existing objects , 2015, J. Vis. Lang. Comput..

[41]  Adam Fourney,et al.  These Aren't the Commands You're Looking For: Addressing False Feedforward in Feature-Rich Software , 2015, UIST.

[42]  Brian Magerko,et al.  Drawing Apprentice: An Enactive Co-Creative Agent for Artistic Collaboration , 2015, Creativity & Cognition.

[43]  Naghmi Shireen,et al.  GEM-NI: A System for Creating and Managing Alternatives In Generative Design , 2015, CHI.

[44]  C. Chabris,et al.  Reading the Mind in the Eyes or Reading between the Lines? Theory of Mind Predicts Collective Intelligence Equally Well Online and Face-To-Face , 2014, PloS one.

[45]  Albert T. Corbett,et al.  The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning , 2012, Cogn. Sci..

[46]  Takeo Igarashi,et al.  Guided exploration of physically valid shapes for furniture design , 2012, ACM Trans. Graph..

[47]  Tovi Grossman,et al.  Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning , 2011, UIST.

[48]  Bill Tomlinson,et al.  Comparing activity theory with distributed cognition for video analysis: beyond "kicking the tires" , 2011, CHI.

[49]  Jessica Mesmer-Magnus,et al.  Measuring Shared Team Mental Models: A Meta-Analysis , 2010 .

[50]  Françoise Détienne,et al.  An approach to assess the quality of collaboration in technology-mediated design situations , 2009, ECCE.

[51]  Diane J. Litman,et al.  Adapting to Student Uncertainty Improves Tutoring Dialogues , 2009, AIED.

[52]  Kirsty Young,et al.  Direct from the source: the value of 'think-aloud' data in understanding learning , 2009 .

[53]  Michelene T. H. Chi,et al.  Knowledge convergence and collaborative learning , 2007 .

[54]  Nikol Rummel,et al.  A rating scheme for assessing the quality of computer-supported collaboration processes , 2007, Int. J. Comput. Support. Collab. Learn..

[55]  Michelene T. H. Chi,et al.  Laboratory Methods for Assessing Experts' and Novices' Knowledge , 2006 .

[56]  Paul J. Feltovich,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[57]  Brady Clark,et al.  Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems , 2006, Int. J. Artif. Intell. Educ..

[58]  Nigel Cross,et al.  Creativity in the design process: co-evolution of problem–solution , 2001 .

[59]  S. Mohammed,et al.  Cognitive Diversity and Consensus in Group Decision Making: The Role of Inputs, Processes, and Outcomes. , 2001, Organizational behavior and human decision processes.

[60]  S. Mohammed,et al.  Team mental models in a team knowledge framework: expanding theory and measurement across disciplinary boundaries , 2001 .

[61]  Bryan Lawson,et al.  Computers, words and pictures , 1997 .

[62]  Maarten van Someren,et al.  The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes , 1994 .

[63]  John S. Gero,et al.  An evolutionary process model of design , 1992 .

[64]  R. J. Bogumil,et al.  The reflective practitioner: How professionals think in action , 1985, Proceedings of the IEEE.

[65]  Christopher McComb,et al.  A cautionary tale about the impact of AI on human design teams , 2021 .

[66]  Parmit K. Chilana,et al.  Challenges in Getting Started in Motion Graphic Design: Perspectives from Casual and Professional Motion Designers , 2021 .

[67]  Harmanpreet Kaur,et al.  Building Shared Mental Models between Humans and AI for Effective Collaboration , 2019 .

[68]  Antti Oulasvirta,et al.  Group Cognition and Collaborative AI , 2018, Human and Machine Learning.

[69]  David Benjamin,et al.  Nature-Based Hybrid Computational Geometry System for Optimizing Component Structure , 2018 .

[70]  Brian Magerko,et al.  An Enactive Model of Creativity for Computational Collaboration and Co-creation , 2015, Creativity in the Digital Age.

[71]  Sivam Krish,et al.  A practical generative design method , 2011, Comput. Aided Des..

[72]  P. Kirschner,et al.  Team learning: building shared mental models , 2011 .

[73]  George Head,et al.  Effective collaboration: deep collaboration as an essential element of the learning process , 2003 .

[74]  Ozgur Eris,et al.  ASKING GENERATIVE DESIGN QUESTIONS: A FUNDAMENTAL COGNITIVE MECHANISM IN DESIGN THINKING , 2003 .

[75]  G. Allaire,et al.  A level-set method for shape optimization , 2002 .

[76]  Kurt VanLehn,et al.  A model of the self-explanation effect. , 1992 .

[77]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.