Digital fabrication approaches for the design and development of shape-changing displays

Interactive shape-changing displays enable dynamic representations of data and information through physically reconfigurable geometry. The actuated physical deformations of these displays can be utilised in a wide range of new application areas, such as dynamic landscape and topographical modelling, architectural design, physical telepresence and object manipulation. Traditionally, shape-changing displays have a high development cost in mechanical complexity, technical skills and time/finances required for fabrication. There is still a limited number of robust shape-changing displays that go beyond one-off prototypes. Specifically, there is limited focus on low-cost/accessible design and development approaches involving digital fabrication (e.g. 3D printing). To address this challenge, this thesis presents accessible digital fabrication approaches that support the development of shape-changing displays with a range of application examples – such as physical terrain modelling and interior design artefacts. Both laser cutting and 3D printing methods have been explored to ensure generalisability and accessibility for a range of potential users. The first design-led content generation explorations show that novice users, from the general public, can successfully design and present their own application ideas using the physical animation features of the display. By engaging with domain experts in designing shape-changing content to represent data specific to their work domains the thesis was able to demonstrate the utility of shape-changing displays beyond novel systems and describe practical use-case scenarios and applications through rapid prototyping methods. This thesis then demonstrates new ways of designing and building shape-changing displays that goes beyond current implementation examples available (e.g. pin arrays and continuous surface shape-changing displays). To achieve this, the thesis demonstrates how laser cutting and 3D printing can be utilised to rapidly fabricate deformable surfaces for shape-changing displays with embedded electronics. This thesis is concluded with a discussion of research implications and future direction for this work.

[1]  Faisal Taher,et al.  A Characterization of Actuation Techniques for Generating Movement in Shape-Changing Interfaces , 2017, Int. J. Hum. Comput. Interact..

[2]  Céline Coutrix,et al.  Morphees+: Studying Everyday Reconfigurable Objects for the Design and Taxonomy of Reconfigurable UIs , 2018, CHI.

[3]  Pierre Dragicevic,et al.  Supporting the design and fabrication of physical visualizations , 2014, CHI.

[4]  Miriam Sturdee,et al.  Using Design Fiction to Inform Shape-Changing Interface Design and Use , 2017 .

[5]  Hiroshi Ishii,et al.  Sensetable: a wireless object tracking platform for tangible user interfaces , 2001, CHI.

[6]  Tong Lu,et al.  iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing , 2015, CHI.

[7]  Miriam Sturdee,et al.  Drawing Design Futures for Shape-Changing Interfaces , 2017, Conference on Designing Interactive Systems.

[8]  Hiroshi Ishii,et al.  mediaBlocks: physical containers, transports, and controls for online media , 1998, SIGGRAPH.

[9]  Dhairya Dand,et al.  Obake: interactions on a 2.5D elastic display , 2013, UIST '13 Adjunct.

[10]  Faisal Taher,et al.  ShapeClip: Towards Rapid Prototyping with Shape-Changing Displays for Designers , 2015, CHI.

[11]  Roel Vertegaal,et al.  An inflatable hemispherical multi-touch display , 2010, TEI.

[12]  Hiroshi Ishii,et al.  TRANSFORM: Embodiment of "Radical Atoms" at Milano Design Week , 2015, CHI Extended Abstracts.

[13]  Tomas Diez Personal Fabrication: Fab Labs as Platforms for Citizen-Based Innovation, from Microcontrollers to Cities , 2012 .

[14]  L. Ou,et al.  A study of colour emotion and colour preference. Part I: Colour emotions for single colours , 2004 .

[15]  Wendy E. Mackay,et al.  Stretchis: Fabricating Highly Stretchable User Interfaces , 2016, UIST.

[16]  WhiteJoseph,et al.  A Novel Approach to 3D-Printed Fabrics and Garments , 2015 .

[17]  Pierre Dragicevic,et al.  Evaluating the efficiency of physical visualizations , 2013, CHI.

[18]  Miriam Sturdee,et al.  ActuEating: Designing, Studying and Exploring Actuating Decorative Artefacts , 2018, Conference on Designing Interactive Systems.

[19]  Pierre Dragicevic,et al.  An Interaction Model for Visualizations Beyond The Desktop , 2013, IEEE Transactions on Visualization and Computer Graphics.

[20]  Pedro Lopes,et al.  Metamaterial Mechanisms , 2016, UIST.

[21]  Roel Vertegaal,et al.  Organic experiences: (re)shaping interactions with deformable displays , 2013, CHI Extended Abstracts.

[22]  Allison Druin,et al.  Technology probes: inspiring design for and with families , 2003, CHI '03.

[23]  Hiroshi Ishii,et al.  aeroMorph - Heat-sealing Inflatable Shape-change Materials for Interaction Design , 2016, UIST.

[24]  Johan Kildal,et al.  Interacting with Deformable User Interfaces: Effect of Material Stiffness and Type of Deformation Gesture , 2012, HAID.

[25]  Martin Schmitz,et al.  Flexibles: Deformation-Aware 3D-Printed Tangibles for Capacitive Touchscreens , 2017, CHI.

[26]  Pierre Dragicevic,et al.  Zooids: Building Blocks for Swarm User Interfaces , 2016, UIST.

[27]  Youngwoo Park,et al.  The Trial of Bendi in a Coffeehouse: Use of a Shape-Changing Device for a Tactile-Visual Phone Conversation , 2015, CHI.

[28]  B. Szerszynski Drift as a Planetary Phenomenon , 2018, Performance Research.

[29]  Hiroshi Ishii,et al.  Illuminating clay: a 3-D tangible interface for landscape analysis , 2002, CHI.

[30]  Hiroshi Ishii,et al.  Tangible bits: beyond pixels , 2008, TEI.

[31]  Marcos Serrano,et al.  Visual Composition of Graphical Elements on Non-Rectangular Displays , 2017, CHI.

[32]  D. Leo Engineering Analysis of Smart Material Systems , 2007 .

[33]  John Hardy,et al.  Toolkit support for interactive projected displays , 2012, MUM.

[34]  Andreas Butz,et al.  Activity Sculptures: Exploring the Impact of Physical Visualizations on Running Activity , 2014, IEEE Trans. Vis. Comput. Graph..

[35]  Steve Marschner,et al.  Mechanical characterization of structured sheet materials , 2018, ACM Trans. Graph..

[36]  Faisal Taher,et al.  Exploring Interactions with Physically Dynamic Bar Charts , 2015, CHI.

[37]  Sungmin Cho,et al.  ElaScreen: exploring multi-dimensional data using elastic screen , 2013, CHI Extended Abstracts.

[38]  Naoki Kawakami,et al.  GelForce: a vision-based traction field computer interface , 2005, CHI Extended Abstracts.

[39]  Kaufui Wong,et al.  A Review of Additive Manufacturing , 2012 .

[40]  Hiroshi Ishii,et al.  Haptic Edge Display for Mobile Tactile Interaction , 2016, CHI.

[41]  Sean Follmer,et al.  Grand Challenges in Shape-Changing Interface Research , 2018, CHI.

[42]  Michael Eisenberg,et al.  Towards the crafting of personal health technologies , 2014, Conference on Designing Interactive Systems.

[43]  Yvonne Rogers,et al.  Introduction to the Special Issue of “The Turn to The Wild” , 2013, TCHI.

[44]  Michael R. Powers The Shapes of Things to Come , 2014 .

[45]  Andrea Ehrmann,et al.  Combining 3D printed forms with textile structures - mechanical and geometrical properties of multi-material systems , 2015 .

[46]  Jakob Grue Simonsen,et al.  Is once enough?: on the extent and content of replications in human-computer interaction , 2014, CHI.

[47]  Jon Froehlich,et al.  Designing 3D-Printed Deformation Behaviors Using Spring-Based Structures: An Initial Investigation , 2017, UIST.

[48]  Hiroshi Ishii,et al.  Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices , 2012, UIST.

[49]  Roel Vertegaal,et al.  An Evaluation of Shape Changes for Conveying Emotions , 2016, CHI.

[50]  Linda Worbin,et al.  Reach: dynamic textile patterns for communication and social expression , 2005, CHI EA '05.

[51]  Petr Felkel,et al.  Straight Skeleton Implementation , 1998 .

[52]  Jie Qi,et al.  Animating paper using shape memory alloys , 2012, CHI.

[53]  Stephen A. Brewster,et al.  Levitating Particle Displays with Interactive Voxels , 2018, PerDis.

[54]  Sriram Subramanian,et al.  Is my phone alive?: a large-scale study of shape change in handheld devices using videos , 2014, CHI.

[55]  Ivan Poupyrev,et al.  Actuation and tangible user interfaces: the Vaucanson duck, robots, and shape displays , 2007, TEI.

[56]  Gregory D. Abowd,et al.  User Interface Design in the 21st Century , 2016, Computer.

[57]  Takeshi Naemura,et al.  KineReels: extension actuators for dynamic 3D shape , 2011, SIGGRAPH '11.

[58]  Hiroshi Ishii,et al.  Relief: a scalable actuated shape display , 2010, TEI '10.

[59]  Alexander Calder Calder; an autobiography with pictures , 1966 .

[60]  Jamie Paik,et al.  Mori: A Modular Origami Robot , 2017, IEEE/ASME Transactions on Mechatronics.

[61]  Hiroshi Ishii,et al.  AnimaStage: Hands-on Animated Craft on Pin-based Shape Displays , 2017, Conference on Designing Interactive Systems.

[62]  Anne Roudaut,et al.  Frozen Suit: Designing a Changeable Stiffness Suit and its Application to Haptic Games , 2017, CHI.

[63]  Xu Jia,et al.  How users manipulate deformable displays as input devices , 2010, CHI.

[64]  Ivan Poupyrev,et al.  Gummi: a bendable computer , 2004, CHI '04.

[65]  Woohun Lee,et al.  Kinetic tiles: modular construction units for interactive kinetic surfaces , 2010, UIST '10.

[66]  Wendy E. Mackay,et al.  HCI, natural science and design: a framework for triangulation across disciplines , 1997, DIS '97.

[67]  Suranga Nanayakkara,et al.  AugmentedForearm: exploring the design space of a display-enhanced forearm , 2013, AH.

[68]  Marcos Serrano,et al.  Investigating Text Legibility on Non-Rectangular Displays , 2016, CHI.

[69]  Anthony Dunne,et al.  Speculative Everything: Design, Fiction, and Social Dreaming , 2013 .

[70]  Chris Harrison,et al.  Providing dynamically changeable physical buttons on a visual display , 2009, CHI.

[71]  Hiroshi Ishii,et al.  The tangible user interface and its evolution , 2008, CACM.

[72]  Jun Rekimoto,et al.  Lumen: interactive visual and shape display for calm computing , 2004, SIGGRAPH '04.

[73]  Miriam Sturdee,et al.  Analysis and Classification of Shape-Changing Interfaces for Design and Application-based Research , 2018, ACM Comput. Surv..

[74]  Michael S. Horn,et al.  Tangible programming and informal science learning: making TUIs work for museums , 2008, IDC.

[75]  Akiya Kamimura,et al.  MimicTile: a variable stiffness deformable user interface for mobile devices , 2012, CHI.

[76]  Tomás Dorta,et al.  The ideation gap:: hybrid tools, design flow and practice , 2008 .

[77]  Kasper Hornbæk,et al.  Exploring the Challenges of Making Data Physical , 2015, CHI Extended Abstracts.

[78]  Ivan Poupyrev,et al.  PAPILLON: designing curved display surfaces with printed optics , 2013, UIST.

[79]  Wendy E. Mackay,et al.  CHI '13 Extended Abstracts on Human Factors in Computing Systems , 2013, CHI 2013.

[80]  Cristina Sylla,et al.  TUIs vs. GUIs: comparing the learning potential with preschoolers , 2012, Personal and Ubiquitous Computing.

[81]  Mark Burry,et al.  Aegis Hyposurface©: The Bordering of University and Practice , 2001 .

[82]  James R. Miller,et al.  Conference Companion on Human Factors in Computing Systems , 1995, CHI 1995.

[83]  Kris Luyten,et al.  Silicone Devices: A Scalable DIY Approach for Fabricating Self-Contained Multi-Layered Soft Circuits using Microfluidics , 2018, CHI.

[84]  Joseph A. Paradiso,et al.  ChainFORM: A Linear Integrated Modular Hardware System for Shape Changing Interfaces , 2016, UIST.

[85]  Rainer Groh,et al.  HCI meets Material Science: A Literature Review of Morphing Materials for the Design of Shape-Changing Interfaces , 2018, CHI.

[86]  William W. Gaver,et al.  The history tablecloth: illuminating domestic activity , 2006, DIS '06.

[87]  Pierre Dragicevic,et al.  Opportunities and Challenges for Data Physicalization , 2015, CHI.

[88]  Stefanie Müller,et al.  Laser cutters , 2015, Interactions.

[89]  Miguel Bruns Alonso,et al.  How to design for transformation of behavior through interactive materiality , 2012, NordiCHI.

[90]  John Zimmerman,et al.  Research through design as a method for interaction design research in HCI , 2007, CHI.

[91]  Ben Shneiderman,et al.  Readings in information visualization - using vision to think , 1999 .

[92]  Pirjo Kääriäinen,et al.  Surface tailoring and design-driven prototyping of fabrics with 3D-printing: An all-cellulose approach , 2018 .

[93]  Philip Ross,et al.  Designing behavior in interaction : using aesthetic experience as a mechanism for design , 2010 .

[94]  Jürgen Steimle,et al.  HotFlex: Post-print Customization of 3D Prints Using Embedded State Change , 2016, CHI.

[95]  Franz Aurenhammer,et al.  Voronoi diagrams—a survey of a fundamental geometric data structure , 1991, CSUR.

[96]  Hiroshi Ishii,et al.  Tangible Query Interfaces: Physically Constrained Tokens for Manipulating Database Queries , 2003, INTERACT.

[97]  Majken Kirkegaard Rasmussen,et al.  Causing commotion with a shape-changing bench: experiencing shape-changing interfaces in use , 2014, CHI.

[98]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[99]  Donghee Son,et al.  Deformable devices with integrated functional nanomaterials for wearable electronics , 2016, Nano Convergence.

[100]  Martin Schmitz,et al.  Capricate: A Fabrication Pipeline to Design and 3D Print Capacitive Touch Sensors for Interactive Objects , 2015, UIST.

[101]  Steve Benford,et al.  A Framework for Tangible User Interfaces , 2003 .

[102]  Miriam Sturdee,et al.  A Public Ideation of Shape-Changing Applications , 2015, ITS.

[103]  Richard Banks,et al.  Slow technology: critical reflection and future directions , 2012, DIS '12.

[104]  Roel Vertegaal,et al.  DisplaySkin: Exploring Pose-Aware Displays on a Flexible Electrophoretic Wristband , 2015, Tangible and Embedded Interaction.

[105]  Roel Vertegaal,et al.  s: Towa ctive Se , 2022 .

[106]  Philip Tuddenham,et al.  Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks , 2010, CHI.

[107]  Hiroshi Ishii,et al.  Pinwheels: visualizing information flow in an architectural space , 2001, CHI Extended Abstracts.

[108]  Hiroshi Ishii,et al.  PneUI: pneumatically actuated soft composite materials for shape changing interfaces , 2013, UIST.

[109]  Oren Zuckerman,et al.  To TUI or not to TUI: Evaluating performance and preference in tangible vs. graphical user interfaces , 2013, Int. J. Hum. Comput. Stud..

[110]  Ivan Poupyrev,et al.  Printed optics: 3D printing of embedded optical elements for interactive devices , 2012, UIST.

[111]  Hutao Cui,et al.  Towards Peak Torque Minimization for Modular Self-Folding Robots , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[112]  Hiroshi Ishii,et al.  I/O brush: drawing with everyday objects as ink , 2004, CHI.

[113]  Yvonne Rogers,et al.  Physikit: Data Engagement Through Physical Ambient Visualizations in the Home , 2016, CHI.

[114]  Sriram Subramanian,et al.  Tilt displays: designing display surfaces with multi-axis tilting and actuation , 2012, Mobile HCI.

[115]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[116]  Patrick Olivier,et al.  Making design probes work , 2013, CHI.

[117]  Wojciech Matusik,et al.  Computational design of mechanical characters , 2013, ACM Trans. Graph..

[118]  Markus Löchtefeld,et al.  Morphees: toward high "shape resolution" in self-actuated flexible mobile devices , 2013, CHI.

[119]  Jun-ichiro Watanabe,et al.  Bookisheet: bendable device for browsing content using the metaphor of leafing through the pages , 2008, UbiComp.

[120]  Stephen Travis Pope,et al.  A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System , 1998 .

[121]  Hiroo Iwata,et al.  Project FEELEX: adding haptic surface to graphics , 2001, SIGGRAPH.

[122]  Skylar Tibbits,et al.  4D Printing: Multi‐Material Shape Change , 2014 .

[123]  L. See A Historical Introduction to the Philosophy of Science , 1993 .

[124]  Hiroshi Ishii,et al.  Direct and gestural interaction with relief: a 2.5D shape display , 2011, UIST '11.

[125]  Audrey Girouard,et al.  Fabricating Bendy: Design and Development of Deformable Prototypes , 2014, IEEE Pervasive Computing.

[126]  Yuki Funabora,et al.  Flexible Fabric Actuator Realizing 3D Movements Like Human Body Surface for Wearable Devices , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[127]  Sriram Subramanian,et al.  LeviPath: Modular Acoustic Levitation for 3D Path Visualisations , 2015, CHI.

[128]  Roel Vertegaal,et al.  Snaplet: using body shape to inform function in mobile flexible display devices , 2011, CHI Extended Abstracts.

[129]  V. Varadan,et al.  Smart Material Systems and MEMS: Design and Development Methodologies , 2006 .

[130]  Edward H. Adelson,et al.  deForm: an interactive malleable surface for capturing 2.5D arbitrary objects, tools and touch , 2011, UIST.

[131]  Susmita Kamila,et al.  Introduction, classification and applications of smart materials: An overview , 2013 .

[132]  Rainer Groh,et al.  FlexiWall: Exploring Layered Data with Elastic Displays , 2014, ITS '14.

[133]  Gerrit C. van der Veer,et al.  CHI '05 Extended Abstracts on Human Factors in Computing Systems , 2005, CHI 2005.

[134]  Hiroshi Ishii,et al.  Emerging frameworks for tangible user interfaces , 2000, IBM Syst. J..

[135]  S. Ganapathy,et al.  A new general triangulation method for planar contours , 1982, SIGGRAPH.

[136]  Tamás D. Gedeon,et al.  ForceForm: a dynamically deformable interactive surface , 2013, ITS.

[137]  Allan Rennie,et al.  Adaptive Materials:Utilising additive manufactured scaffolds to control self-organising material aggregation , 2015 .

[138]  Faisal Taher,et al.  Investigating the Use of a Dynamic Physical Bar Chart for Data Exploration and Presentation , 2017, IEEE Transactions on Visualization and Computer Graphics.

[139]  Charles Perin,et al.  Dynamic Composite Data Physicalization Using Wheeled Micro-Robots , 2019, IEEE Transactions on Visualization and Computer Graphics.

[140]  Jamie Paik,et al.  A Compact Modular Soft Surface With Reconfigurable Shape and Stiffness , 2019, IEEE/ASME Transactions on Mechatronics.

[141]  Roel Vertegaal,et al.  MorePhone: a study of actuated shape deformations for flexible thin-film smartphone notifications , 2013, CHI.

[142]  Ming-Chuan Leu,et al.  Progress in Additive Manufacturing and Rapid Prototyping , 1998 .

[143]  Jan O. Borchers,et al.  Sketch&Stitch: Interactive Embroidery for E-textiles , 2018, CHI.

[144]  William Gobson,et al.  Multimedia: From Wagner to Virtual Reality , 2001 .

[145]  Gary Priestnall,et al.  Projection Augmented Relief Models (PARM): tangible displays for geographic information , 2012, EVA.

[146]  Sriram Subramanian,et al.  TableHop: An Actuated Fabric Display Using Transparent Electrodes , 2016, CHI.

[147]  Masatoshi Ishikawa,et al.  The deformable workspace: A membrane between real and virtual space , 2008, 2008 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems.

[148]  Abigail Sellen,et al.  Designing web-connected physical artefacts for the 'aesthetic' of the home , 2013, CHI.

[149]  Jamie Zigelbaum,et al.  Shape-changing interfaces , 2011, Personal and Ubiquitous Computing.

[150]  Qi Ge,et al.  Active materials by four-dimension printing , 2013 .

[151]  Mathias Müller,et al.  New Impressions in Interaction Design: A Task Taxonomy for Elastic Displays , 2018, i-com.

[152]  HolmanDavid,et al.  Organic user interfaces , 2008 .

[153]  Allan Rennie,et al.  A Short Review on 4D Printing , 2018 .

[154]  Björn Hartmann,et al.  A series of tubes: adding interactivity to 3D prints using internal pipes , 2014, UIST.

[155]  Eujin Pei,et al.  Direct 3D printing of polymers onto textiles: experimental studies and applications , 2015 .

[156]  Stefanie Müller,et al.  LaserStacker: Fabricating 3D Objects by Laser Cutting and Welding , 2015, UIST.

[157]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[158]  Hiroshi Ishii,et al.  Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays , 2013, CHI.

[159]  Majken Kirkegaard Rasmussen,et al.  Shape-changing interfaces: a review of the design space and open research questions , 2012, CHI.

[160]  Sean Follmer,et al.  shapeShift: 2D Spatial Manipulation and Self-Actuation of Tabletop Shape Displays for Tangible and Haptic Interaction , 2018, CHI.

[161]  Hiroshi Ishii,et al.  Recompose: direct and gestural interaction with an actuated surface , 2011, CHI EA '11.

[162]  Hiroshi Ishii,et al.  The metaDESK: models and prototypes for tangible user interfaces , 1997, UIST '97.

[163]  Albrecht Schmidt,et al.  How to evaluate public displays , 2012, PerDis.

[164]  Hiroshi Ishii,et al.  inFORM: dynamic physical affordances and constraints through shape and object actuation , 2013, UIST.

[165]  Jim Ruppert,et al.  A Delaunay Refinement Algorithm for Quality 2-Dimensional Mesh Generation , 1995, J. Algorithms.

[166]  Aluna Everitt,et al.  PolySurface: A Design Approach for Rapid Prototyping of Shape-Changing Displays Using Semi-Solid Surfaces , 2017, Conference on Designing Interactive Systems.

[167]  Faisal Taher,et al.  ShapeCanvas: An Exploration of Shape-Changing Content Generation by Members of the Public , 2016, CHI.

[168]  Ivan Poupyrev,et al.  Proceedings of the 26th annual ACM symposium on User interface software and technology , 2013, UIST 2013.

[169]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[170]  D. Maynes-Aminzade,et al.  The actuated workbench: computer-controlled actuation in tabletop tangible interfaces , 2003, ACM Trans. Graph..

[171]  Eva Hornecker,et al.  Learning from interactive museum installations about interaction design for public settings , 2006, OZCHI.

[172]  Kasper Hornbæk,et al.  User-defined gestures for elastic, deformable displays , 2014, AVI.