Internet of Tangible Things (IoTT): Challenges and Opportunities for Tangible Interaction with IoT

In the Internet of Things era, an increasing number of everyday objects are able to offer innovative services to the user. However, most of these devices provide only smartphone or web user interfaces. As a result, the interaction is disconnected from the physical world, decreasing the user experience and increasing the risk of user alienation from the physical world. We argue that tangible interaction can counteract this trend and this article discusses the potential benefits and the still open challenges of tangible interaction applied to the Internet of Things. After an analysis of open challenges for Human-Computer Interaction in IoT, we summarize current trends in tangible interaction and extrapolate eight tangible interaction properties that could be exploited for designing novel interactions with IoT objects. Through a systematic review of tangible interaction applied to IoT, we show what has been already explored in the systems that pioneered the field and the future explorations that still have to be conducted. In order to guide future work in this field, we propose a design card set for supporting the generation of tangible interfaces for IoT objects. The card set has been evaluated during a workshop with 21 people and the results are discussed.

[1]  Ana Tajadura-Jiménez,et al.  Third workshop on full-body and multisensory experience , 2016, UbiComp Adjunct.

[2]  Hans-Werner Gellersen,et al.  Projected interfaces: enabling serendipitous interaction with smart tangible objects , 2009, Tangible and Embedded Interaction.

[3]  Elise van den Hoven,et al.  Tangible Computing in Everyday Life: Extending Current Frameworks for Tangible User Interfaces with Personal Objects , 2004, EUSAI.

[4]  Joep Frens,et al.  Designing for rich interaction : integrating form, interaction, and function , 2006 .

[5]  Alessandro Soro,et al.  The Messaging Kettle: Prototyping Connection over a Distance between Adult Children and Older Parents , 2015, CHI.

[6]  Monica Divitini,et al.  RapIoT Toolkit: Rapid Prototyping of Collaborative Internet of Things Applications , 2016, 2016 International Conference on Collaboration Technologies and Systems (CTS).

[7]  Andreas Möller,et al.  Cognitive Objects for human-computer interaction and human-robot interaction , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[8]  Yutaka Matsuo,et al.  Earthquake shakes Twitter users: real-time event detection by social sensors , 2010, WWW '10.

[9]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[10]  J. G. Tanenbaum,et al.  The Reading Glove: designing interactions for object-based tangible storytelling , 2010, AH.

[11]  Zhou Cheng,et al.  Overview of the Internet of Things , 2011 .

[12]  Hiroshi Ishii,et al.  ZeroN: mid-air tangible interaction enabled by computer controlled magnetic levitation , 2011, UIST.

[13]  Andrii Matviienko,et al.  AwareKit: Exploring a Tangible Interaction Paradigm for Digital Calendars , 2017, CHI Extended Abstracts.

[14]  Stephan Wensveen,et al.  Interaction frogger: a design framework to couple action and function through feedback and feedforward , 2004, DIS '04.

[15]  Jun Hu,et al.  Semantic connections: Exploring and manipulating connections in smart spaces , 2010, The IEEE symposium on Computers and Communications.

[16]  Hiroshi Ishii,et al.  Mechanical constraints as computational constraints in tabletop tangible interfaces , 2007, CHI.

[17]  Abigail Sellen,et al.  Design for Privacy in Ubiquitous Computing Environments , 1993, ECSCW.

[18]  Tuck Wah Leong,et al.  Internet of things: a review of literature and products , 2013, OZCHI.

[19]  Luis Orozco-Barbosa,et al.  Easy Smart-Home Environment to Assist Patients with Mobility Impairment , 2015, REHAB.

[20]  Christophe Kolski,et al.  Management of Distributed RFID Surfaces: A Cooking Assistant for Ambient Computing in Kitchen , 2014, ANT/SEIT.

[21]  Vlad Trifa,et al.  Towards the Web of Things: Web Mashups for Embedded Devices , 2009 .

[22]  Pierre Dragicevic,et al.  Opportunities and Challenges for Data Physicalization , 2015, CHI.

[23]  David Thaler,et al.  Architectural Considerations in Smart Object Networking , 2015, RFC.

[24]  Koen van Boerdonk,et al.  Moving Tangible Interaction Systems to the Next Level , 2013, Computer.

[25]  Lida Xu,et al.  The internet of things: a survey , 2014, Information Systems Frontiers.

[26]  Gordana Dodig-Crnkovic,et al.  Cognitively Sustainable ICT with Ubiquitous Mobile Services - Challenges and Opportunities , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[27]  Hiroshi Ishii,et al.  inFORM: dynamic physical affordances and constraints through shape and object actuation , 2013, UIST.

[28]  William Buxton,et al.  Graspable user interfaces , 1996 .

[29]  Carlo Maria Medaglia,et al.  The IoT Design Deck: A Tool for the Co-design of Connected Products , 2017, AHFE.

[30]  Tuck Wah Leong,et al.  Approaching a human-centred internet of things , 2013, OZCHI.

[31]  Emmi Harjuniemi,et al.  IoT owl: soft tangible user interface for detecting the presence of people , 2016, MobileHCI Adjunct.

[32]  Simon Scott-Harden.  Active forms for responsive environments , 2012, IUI '12.

[33]  Simon Mayer,et al.  User interfaces for smart things -- A generative approach with semantic interaction descriptions , 2014, TCHI.

[34]  Albrecht Schmidt,et al.  Embedded Interaction: Interacting with the Internet of Things , 2010, IEEE Internet Computing.

[35]  Hiroshi Ishii,et al.  Emerging frameworks for tangible user interfaces , 2000, IBM Syst. J..

[36]  Miguel Bruns Alonso,et al.  Balancing User and System Control in Shape-Changing Interfaces: a Designerly Exploration , 2016, Tangible and Embedded Interaction.

[37]  Alessandro Soro,et al.  The Messaging Kettle: It’s IoTea time , 2015 .

[38]  Ali Mazalek,et al.  Exploring the design space of gestural interaction with active tokens through user-defined gestures , 2014, CHI.

[39]  Austin S. Lee,et al.  Expressing Intent: An Exploration of Rich Interactions , 2016, Tangible and Embedded Interaction.

[40]  Elise van den Hoven,et al.  Peripheral interaction: characteristics and considerations , 2014, Personal and Ubiquitous Computing.

[41]  Hiroshi Ishii,et al.  Radical atoms: beyond tangible bits, toward transformable materials , 2012, INTR.

[42]  Eva Hornecker,et al.  Beyond affordance: tangibles' hybrid nature , 2012, TEI.

[43]  Elise van den Hoven,et al.  Embodied metaphors in tangible interaction design , 2011, Personal and Ubiquitous Computing.

[44]  Elena Mugellini,et al.  Gesturing on the Steering Wheel: a User-elicited taxonomy , 2014, AutomotiveUI.

[45]  Majken Kirkegaard Rasmussen,et al.  Shape-changing interfaces: a review of the design space and open research questions , 2012, CHI.

[46]  Yvonne Rogers,et al.  Getting quizzical about physical: observing experiences with a tangible questionnaire , 2015, UbiComp.

[47]  J. Ioannidis,et al.  The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration , 2009, Annals of Internal Medicine [serial online].

[48]  Yoann Lebrun,et al.  A model for managing interactions between tangible and virtual agents on an RFID interactive tabletop: Case study in traffic simulation , 2015, J. Comput. Syst. Sci..

[49]  W. Keith Edwards,et al.  Intelligibility and Accountability: Human Considerations in Context-Aware Systems , 2001, Hum. Comput. Interact..

[50]  Luca Benini,et al.  Evolving tuis with smart objects for multi-context interaction , 2008, CHI Extended Abstracts.

[51]  Spyros Lalis,et al.  Graspable and resource-flexible applications for pervasive computing at home , 2013, IEEE Communications Magazine.

[52]  Monica Divitini,et al.  Tiles: An Inventor Toolkit for Interactive Objects , 2016, AVI.

[53]  Hiroshi Ishii,et al.  Topobo: a constructive assembly system with kinetic memory , 2004, CHI.

[54]  Arjan Kuijper,et al.  Capacitive near-field communication for ubiquitous interaction and perception , 2014, UbiComp.

[55]  Yvonne Rogers,et al.  Physikit: Data Engagement Through Physical Ambient Visualizations in the Home , 2016, CHI.

[56]  Angelini Leonardo,et al.  Interaction with the Internet of Tangible Things (IoTT) , 2017 .

[57]  Giulio Jacucci,et al.  Supporting the Serendipitous Use of Domestic Technologies , 2016, IEEE Pervasive Computing.

[58]  Elena Mugellini,et al.  Internet of Tangibles: Exploring the Interaction-Attention Continuum , 2018, Tangible and Embedded Interaction.

[59]  Monica Divitini,et al.  Tiles: A Card-based Ideation Toolkit for the Internet of Things , 2017, Conference on Designing Interactive Systems.

[60]  Michael Schmitz,et al.  Concepts for life-like interactive objects , 2010, TEI.

[61]  R. Day.  Remediation: Understanding new media , 1999 .

[62]  Donald A. Norman,et al.  Natural user interfaces are not natural , 2010, INTR.

[63]  Christophe Kolski,et al.  Interaction Between Tangible and Virtual Agents on Interactive Tables: Principles and Case Study , 2013, ANT/SEIT.

[64]  Pierre Dragicevic,et al.  Embedded Data Representations , 2017, IEEE Transactions on Visualization and Computer Graphics.

[65]  Patrizia Marti,et al.  The subtle body , 2014 .

[66]  Margot Brereton,et al.  Invisible connections: investigating older people's emotions and social relations around objects , 2014, CHI.

[67]  Saskia Bakker,et al.  The Interaction-Attention Continuum: Considering Various Levels of Human Attention in Interaction Design , 2016 .

[68]  Florian Evéquoz,et al.  Creating people-aware IoT applications by combining design thinking and user-centered design methods , 2015, 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT).

[69]  Eva Hornecker,et al.  Creative idea exploration within the structure of a guiding framework: the card brainstorming game , 2010, TEI '10.

[70]  Ann Light,et al.  Designing Connected Products: UX for the Consumer Internet of Things , 2015 .

[71]  Monica Divitini,et al.  RapIoT toolkit: Rapid prototyping of collaborative Internet of Things applications , 2019, Future Gener. Comput. Syst..

[72]  Albrecht Schmidt,et al.  Interacting with the ubiquitous computer: towards embedding interaction , 2005, sOc-EUSAI '05.

[73]  M. Heidegger,et al.  Being and time : a translation of Sein und Zeit , 1996 .

[74]  Paul Dourish,et al.  Where the action is , 2001 .

[75]  Timo Arnall,et al.  Investigating an “Internet of Hybrid Products”: Assembling Products, Interactions, Services, and Networks through Design , 2011 .

[76]  Robert Kowalski,et al.  cubble: a multi-device hybrid approach supporting communication in long-distance relationships , 2013, TEI '13.

[77]  Jonathan Chapman,et al.  Emotionally Durable Design: Objects, Experiences and Empathy , 2015 .

[78]  Alessandro Soro,et al.  Technology Individuation: The Foibles of Augmented Everyday Objects , 2017, CHI.

[79]  Jacob Buur,et al.  Getting a grip on tangible interaction: a framework on physical space and social interaction , 2006, CHI.

[80]  Austin S. Lee,et al.  Iyagi: An Immersive Storytelling Tool for Healthy Bedtime Routine , 2017, TEI.

[81]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.