Interaction Illustration Taxonomy: Classification of Styles and Techniques for Visually Representing Interaction Scenarios

Static illustrations are ubiquitous means to represent interaction scenarios. Across papers and reports, these visuals demonstrate people’s use of devices, explain systems, or show design spaces. Creating such figures is challenging, and very little is known about the overarching strategies for visually representing interaction scenarios. To mitigate this task, we contribute a unified taxonomy of design elements that compose such figures. In particular, we provide a detailed classification of Structural and Interaction strategies, such as composition, visual techniques, dynamics, representation of users, and many others – all in context of the type of scenarios. This taxonomy can inform researchers’ choices when creating new figures, by providing a concise synthesis of visual strategies, and revealing approaches they were not aware of before. Furthermore, to support the community for creating further taxonomies, we also provide three open-source software facilitating the coding process and visual exploration of the coding scheme.

[1]  A. D. Manning,et al.  Understanding Comics: The Invisible Art , 1993 .

[2]  Bill Buxton,et al.  Sketching User Experiences: Getting the Design Right and the Right Design , 2007 .

[3]  Pedro Lopes,et al.  Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation , 2018, CHI.

[4]  Patrick Baudisch,et al.  VirtualSpace - Overloading Physical Space with Multiple Virtual Reality Users , 2018, CHI.

[5]  Adrian David Cheok,et al.  Electronic taste stimulation , 2011, UbiComp '11.

[6]  P. C. Malshe What is this interaction? , 1994, The Journal of the Association of Physicians of India.

[7]  Nikos Sidiropoulos,et al.  SinaPlot: an enhanced chart for simple and truthful representation of single observations over multiple classes , 2015, bioRxiv.

[8]  Daniel Vogel,et al.  Authoring Illustrations of Human Movements by Iterative Physical Demonstration , 2016, UIST.

[9]  Pat Hanrahan,et al.  Designing effective step-by-step assembly instructions , 2003, ACM Trans. Graph..

[10]  HeerJeffrey,et al.  D3 Data-Driven Documents , 2011 .

[11]  Rubaiat Habib Kazi,et al.  Your Paper is Dead!: Bringing Life to Research Articles with Animated Figures , 2015, CHI Extended Abstracts.

[12]  Jacques Bertin,et al.  Semiology of Graphics - Diagrams, Networks, Maps , 2010 .

[13]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[14]  Saul Greenberg,et al.  Cross-device interaction via micro-mobility and f-formations , 2012, UIST.

[15]  Niels Henze,et al.  The Effect of Offset Correction and Cursor on Mid-Air Pointing in Real and Virtual Environments , 2018, CHI.

[16]  Per Ola Kristensson,et al.  Change Blindness in Proximity-Aware Mobile Interfaces , 2018, CHI.

[17]  Ryohei Nakatsu,et al.  Taste/IP: the sensation of taste for digital communication , 2012, ICMI '12.

[18]  Steven Franconeri,et al.  The Connected Scatterplot for Presenting Paired Time Series , 2016, IEEE Transactions on Visualization and Computer Graphics.

[19]  J. Fleiss Measuring nominal scale agreement among many raters. , 1971 .

[20]  Rubaiat Habib Kazi,et al.  Draco: bringing life to illustrations with kinetic textures , 2014, CHI.

[21]  Andrea Bunt,et al.  Crowdsourcing vs Laboratory-Style Social Acceptability Studies?: Examining the Social Acceptability of Spatial User Interactions for Head-Worn Displays , 2018, CHI.

[22]  Malcolm L. Fleming,et al.  Classification and analysis of instructional illustrations , 1967 .

[23]  M. S. T. Carpendale,et al.  Considering Visual Variables as a Basis for Information Visualisation , 2003 .

[24]  Rubaiat Habib Kazi,et al.  Kitty: sketching dynamic and interactive illustrations , 2014, UIST.

[25]  ShareSpace , 2018, Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.

[26]  Eujin Pei,et al.  A Taxonomic Classification of Visual Design Representations Used by Industrial Designers and Engineering Designers , 2011 .

[27]  伊万·波派列夫,et al.  Radar-Based Gesture-Recognition through a Wearable Device , 2015 .

[28]  J. Cutting Representing Motion in a Static Image: Constraints and Parallels in Art, Science, and Popular Culture , 2002, Perception.

[29]  Benjamin Bach,et al.  Picturing Science: Design Patterns in Graphical Abstracts , 2018, Diagrams.

[30]  ElectricItch , 2018, Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.

[31]  J. C. Dunn,et al.  A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters , 1973 .

[32]  Michael Nebeling,et al.  User-Driven Design Principles for Gesture Representations , 2018, CHI.

[33]  Klaus Krippendorff,et al.  Estimating the Reliability, Systematic Error and Random Error of Interval Data , 1970 .

[34]  Edward Lank,et al.  Chameleon: Bringing Interactivity to Static Digital Documents , 2020, CHI.

[35]  M. Sheelagh T. Carpendale,et al.  Sketching User Experiences - The Workbook , 2011 .

[36]  Shumin Zhai,et al.  Understanding the Uncertainty in 1D Unidirectional Moving Target Selection , 2018, CHI.

[37]  Jos Prickaerts,et al.  Graphical Abstracts , 2014, Current neuropharmacology.

[38]  CueAuth , 2018, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.

[39]  Scenograph , 2018, Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.

[40]  Geehyuk Lee,et al.  Exploring Multimodal Watch-back Tactile Display using Wind and Vibration , 2018, CHI.

[41]  Robert Newman,et al.  Sketching, concept development and automotive design , 2003 .

[42]  Carl Gutwin,et al.  Improving Discoverability and Expert Performance in Force-Sensitive Text Selection for Touch Devices with Mode Gauges , 2018, CHI.

[43]  Andrea Forte,et al.  Reliability and Inter-rater Reliability in Qualitative Research , 2019, Proc. ACM Hum. Comput. Interact..

[44]  Pei-Yu Chi,et al.  Designing Video-Based Interactive Instructions , 2016 .

[45]  Velko Vechev,et al.  MagTics: Flexible and Thin Form Factor Magnetic Actuators for Dynamic and Wearable Haptic Feedback , 2017, UIST.

[46]  Nicolai Marquardt,et al.  Esquisse: Using 3D Models Staging to Facilitate the Creation of Vector-Based Trace Figures , 2019, INTERACT.

[47]  Clemens Nylandsted Klokmose,et al.  Situated Sketching and Enactment for Pervasive Displays , 2019, ISS.

[48]  Gilles Bailly,et al.  IconHK: Using Toolbar button Icons to Communicate Keyboard Shortcuts , 2017, CHI.