Sketching & Drawing as Future Inquiry in HCI

Creating visual imagery helps us to situate ourselves within unknown worlds, processes, make connections, and find solutions. By exploring drawn ideas for novel technologies, we can examine the implications of their place in the world. Drawing, or sketching, for future inquiry in Human Computer Interaction (HCI) can be a stand-alone investigative approach, part of a wider ‘world-building’ in design fiction, or simply ideation around a concept. By examining instances of existing practice in HCI, in this paper we establish recommendations and rationales for those wishing to utilise sketching and drawing within their research. We examine approaches ranging from ideation, diagramming, scenario building, comics creation and artistic representation to create a model for sketching and drawing as future inquiry for HCI. This work also reflects on the ways in which these arts can inform and elucidate research and practice in HCI, and makes recommendations for the field, within its teaching, processes and outcomes.

[1]  Miriam Sturdee,et al.  Using Design Fiction to Inform Shape-Changing Interface Design and Use , 2017 .

[2]  Mieke Haesen,et al.  Get Your Requirements Straight: Storyboarding Revisited , 2009, INTERACT.

[3]  Miriam Sturdee,et al.  SketchBlog #1 , 2018, Interactions.

[4]  Francis C. Moon,et al.  The Machines of Leonardo Da Vinci and Franz Reuleaux: Kinematics of Machines from the Renaissance to the 20th Century , 2007 .

[5]  Miriam Sturdee,et al.  Sketching in HCI: Hands-on Course of Sketching Techniques , 2019, CHI Extended Abstracts.

[6]  Nancy C. Andreasen,et al.  Creativity in art and science: are there two cultures? , 2012, Dialogues in clinical neuroscience.

[7]  Liqing Zhang,et al.  MindFinder: interactive sketch-based image search on millions of images , 2010, ACM Multimedia.

[8]  Norbert Seyff,et al.  Sketching and notation creation with FlexiSketch Team: Evaluating a new means for collaborative requirements elicitation , 2015, 2015 IEEE 23rd International Requirements Engineering Conference (RE).

[9]  Miriam Sturdee,et al.  SketchingDIS: Hand-drawn Sketching in HCI , 2017, Conference on Designing Interactive Systems.

[10]  Paul A. Cairns,et al.  Sketching sketching: outlines of a collaborative design method , 2009, BCS HCI.

[11]  Miriam Sturdee,et al.  Visual Methods for the Design of Shape-Changing Interfaces , 2019, INTERACT.

[12]  Hiroshi Ishii,et al.  Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices , 2012, UIST.

[13]  Joseph Lindley,et al.  More-Than Human Centred Design: Considering Other Things , 2019, The Design Journal.

[14]  Pieter Jan Stappers,et al.  Toward Electronic Napkins and Beermats: Computer Support for Visual Ideation Skills , 1999, Visual Representations and Interpretations.

[15]  Tomás Dorta.  Design Flow and Ideation , 2008, International Journal of Architectural Computing.

[16]  Edward De Bono,et al.  De Bono's Thinking Course , 1982 .

[17]  Eli Blevis,et al.  Sharing the hidden treasure in pictorials , 2015, Interactions.

[18]  Stephanie Foehrenbach.  Learn to Sketch (Even if You Can't Draw): Hands-on Sketching Course , 2015, CHI Extended Abstracts.

[19]  Miriam Sturdee,et al.  Feeling SketCHI? , 2018, Interactions.

[20]  Aaron Marcus,et al.  HCI and User-Experience Design , 2015, Human–Computer Interaction Series.

[21]  Rahinah Ibrahim,et al.  Comparison of CAD and manual sketching tools for teaching architectural design , 2010 .

[22]  John C. Grundy,et al.  An e-whiteboard application to support early design-stage sketching of UML diagrams , 2003, IEEE Symposium on Human Centric Computing Languages and Environments, 2003. Proceedings. 2003.

[23]  Frederic Fol Leymarie,et al.  Portrait drawing by Paul the robot , 2013, Comput. Graph..

[24]  Abigail Sellen,et al.  User sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback , 2006, NordiCHI '06.

[25]  Jason Hong,et al.  Sketch it, make it: sketching precise drawings for laser cutting , 2012, CHI EA '12.

[26]  Simon Grand,et al.  Design Fiction: A Method Toolbox for Design Research in a Complex World , 2010 .

[27]  Tracy Anne Hammond,et al.  Maestoso: An Intelligent Educational Sketching Tool for Learning Music Theory , 2015, AAAI.

[28]  Tracy Anne Hammond,et al.  Sketching Cognition and Creativity: Leveraging Sketch Interfaces for Enhancing Creativity and Cognition , 2019, Creativity & Cognition.

[29]  Miriam Sturdee,et al.  SketCHI: Hands-On Special Interest Group on Sketching in HCI , 2018, CHI Extended Abstracts.

[30]  Chris Speed,et al.  Thing Ethnography: Doing Design Research with Non-Humans , 2016, Conference on Designing Interactive Systems.

[31]  M. Sheelagh T. Carpendale,et al.  Follow that sketch: Lifecycles of diagrams and sketches in software development , 2011, 2011 6th International Workshop on Visualizing Software for Understanding and Analysis (VISSOFT).

[32]  Satoshi Matsuoka,et al.  Teddy: A Sketching Interface for 3D Freeform Design , 1999, SIGGRAPH Courses.

[33]  M. Sheelagh T. Carpendale,et al.  SketCHI 2.0: Hands-On Special Interest Group on Sketching in HCI , 2019, CHI Extended Abstracts.

[34]  Lizzie Coles-Kemp,et al.  A Tactile Visual Library To Support User Experience Storytelling , 2014 .

[35]  M. Sheelagh T. Carpendale,et al.  The Emerging Genre of Data Comics , 2017, IEEE Computer Graphics and Applications.

[36]  Miriam Sturdee,et al.  Sketching as a support mechanism for the design and development of shape-changing interfaces , 2018 .

[37]  Bongshin Lee,et al.  iSketchVis: Data Exploration through Sketch-based Charting , 2011 .

[38]  Takuya Nojima,et al.  Hairlytop interface: An interactive surface display comprised of hair-like soft actuators , 2013, 2013 World Haptics Conference (WHC).

[39]  Bill Buxton,et al.  Sketching User Experiences: Getting the Design Right and the Right Design , 2007 .

[40]  Jakob Grue Simonsen,et al.  Sketching Shape-changing Interfaces: Exploring Vocabulary, Metaphors Use, and Affordances , 2016, CHI.

[41]  Miriam Sturdee,et al.  Drawing Design Futures for Shape-Changing Interfaces , 2017, Conference on Designing Interactive Systems.

[42]  Miriam Sturdee,et al.  Design fiction as world building , 2017 .

[43]  Alma Leora Culén,et al.  Innovation in HCI: what can we learn from design thinking? , 2014, NordiCHI.

[44]  Mark Blythe,et al.  Research Fiction: Storytelling, Plot and Design , 2017, CHI.

[45]  Peter C. Wright,et al.  Anti-Solutionist Strategies: Seriously Silly Design Fiction , 2016, CHI.

[46]  James Pierce,et al.  Dark Clouds, Io&#!+, and [Crystal Ball Emoji]: Projecting Network Anxieties with Alternative Design Metaphors , 2017, Conference on Designing Interactive Systems.

[47]  Miriam Sturdee,et al.  Implications for Adoption , 2017, CHI.

[48]  Lin Liang,et al.  Scribble-a-Secret: Similarity-based password authentication using sketches , 2008, 2008 19th International Conference on Pattern Recognition.

[49]  Miriam Sturdee,et al.  Design Fiction: How to Build a Voight-Kampff Machine , 2016, CHI Extended Abstracts.

[50]  M. Sheelagh T. Carpendale,et al.  Sketch & The Lizard King: Supporting Image Inclusion in HCI Publishing , 2018, CHI Extended Abstracts.

[51]  Ben Jonson,et al.  Design ideation: the conceptual sketch in the digital age , 2005 .

[52]  M. Sheelagh T. Carpendale,et al.  Sketching User Experiences - The Workbook , 2011 .

[53]  Jason Hong,et al.  Computational Support for Sketching in Design: A Review , 2009, Found. Trends Hum. Comput. Interact..

[54]  Blake Williford,et al.  SketchTivity: Improving Creativity by Learning Sketching with an Intelligent Tutoring System , 2017, Creativity & Cognition.

[55]  Mark Blythe,et al.  RtD Comics: A Medium for Representing Research Through Design , 2016, Conference on Designing Interactive Systems.

[56]  Adrien Bousseau,et al.  How Novices Sketch and Prototype Hand-Fabricated Objects , 2016, CHI.

[57]  Alberto Alessi.  The Dream Factory: Alessi Since 1921 , 1998 .

[58]  Koji Yatani,et al.  Sketching and Drawing in the Design of Open Source Software , 2010, 2010 IEEE Symposium on Visual Languages and Human-Centric Computing.

[59]  Miriam Sturdee,et al.  A Public Ideation of Shape-Changing Applications , 2015, ITS.

[60]  James A. Landay,et al.  Sketching storyboards to illustrate interface behaviors , 1996, CHI Conference Companion.

[61]  Stuart Reeves,et al.  Alternate endings: using fiction to explore design futures , 2014, CHI Extended Abstracts.

[62]  Ron Wakkary,et al.  Morse Things: A Design Inquiry into the Gap Between Things and Us , 2017, Conference on Designing Interactive Systems.

[63]  P. Coulton,et al.  Turning Philosophy with a Speculative Lathe: object-oriented ontology, carpentry, and design fiction , 2018, DRS2018: Catalyst.

[64]  M. Sheelagh T. Carpendale,et al.  Sketching Sustainability in Computing , 2019, Creativity & Cognition.

[65]  Amy Bruckman,et al.  An HCI Approach to Computing in the Real World , 2008, JERC.

[66]  Benjamin Bach,et al.  Picturing Science: Design Patterns in Graphical Abstracts , 2018, Diagrams.

[67]  Steve Benford,et al.  Ambiguity as a resource for design , 2003, CHI '03.

[68]  Neil Cohn,et al.  Explaining ‘I Can’t Draw’: Parallels between the Structure and Development of Language and Drawing , 2012, Human Development.

[69]  Miriam Sturdee,et al.  Applied Sketching in HCI: Hands-on Course of Sketching Techniques , 2018, CHI Extended Abstracts.

[70]  Nasir D. Memon,et al.  New Me: Understanding Expert and Non-Expert Perceptions and Usage of the Tor Anonymity Network , 2017, SOUPS.

[71]  Gabriela Goldschmidt,et al.  Manual Sketching: Why Is It Still Relevant? , 2017 .