Review: Development and Technical Design of Tangible User Interfaces in Wide-Field Areas of Application

A tangible user interface or TUI connects physical objects and digital interfaces. It is more interactive and interesting for users than a classic graphic user interface. This article presents a descriptive overview of TUI’s real-world applications sorted into ten main application areas—teaching of traditional subjects, medicine and psychology, programming, database development, music and arts, modeling of 3D objects, modeling in architecture, literature and storytelling, adjustable TUI solutions, and commercial TUI smart toys. The paper focuses on TUI’s technical solutions and a description of technical constructions that influences the applicability of TUIs in the real world. Based on the review, the technical concept was divided into two main approaches: the sensory technical concept and technology based on a computer vision algorithm. The sensory technical concept is processed to use wireless technology, sensors, and feedback possibilities in TUI applications. The image processing approach is processed to a marker and markerless approach for object recognition, the use of cameras, and the use of computer vision platforms for TUI applications.

[1]  Stavros N. Demetriadis,et al.  Tangible versus graphical user interfaces for robot programming: exploring cross-age children’s preferences , 2013, Personal and Ubiquitous Computing.

[2]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[3]  Ali Mazalek,et al.  Exploring the design space of gestural interaction with active tokens through user-defined gestures , 2014, CHI.

[4]  Joyce Ma,et al.  Using a Tangible Versus a Multi-touch Graphical User Interface to Support Data Exploration at a Museum Exhibit , 2015, TEI.

[5]  Bertrand Schneider,et al.  Benefits of a Tangible Interface for Collaborative Learning and Interaction , 2011, IEEE Transactions on Learning Technologies.

[6]  Jeha Ryu,et al.  ARtalet: Tangible User Interface Based Immersive Augmented Reality Authoring Tool for Digilog Book , 2010, 2010 International Symposium on Ubiquitous Virtual Reality.

[7]  Patrick Jermann,et al.  Task Performance vs. Learning Outcomes: A Study of a Tangible User Interface in the Classroom , 2010, EC-TEL.

[8]  Jay Lee,et al.  Bottles as a minimal interface to access digital information , 2001, CHI Extended Abstracts.

[9]  Hiroshi Ishii,et al.  Illuminating clay: a 3-D tangible interface for landscape analysis , 2002, CHI.

[10]  Hiroshi Ishii,et al.  Tangible user interfaces in context and theory , 2007, CHI Extended Abstracts.

[11]  Hyunseung Choo,et al.  Vibration Based Tangible Tokens for Intuitive Pairing Among Smart Devices , 2016, HCI.

[12]  Luca Benini,et al.  Hidden Markov Models Implementation for Tangible Interfaces , 2009, INTETAIN.

[13]  L. Garber.  Tangible User Interfaces: Technology You Can Touch , 2012, Computer.

[14]  Thomas Spyrou,et al.  Spyractable: A Tangible User Interface Modular Synthesizer , 2014, HCI.

[15]  Y Wang,et al.  Bringing Clay and Sand into Digital Design — Continuous Tangible user Interfaces , 2004 .

[16]  E. Whiting,et al.  TangibleCircuits: An Interactive 3D Printed Circuit Education Tool for People with Visual Impairments , 2020, CHI.

[17]  Xavier Alamán,et al.  Tangible Interfaces and Virtual Worlds: A New Environment for Inclusive Education , 2013, UCAmI.

[18]  Sara Diamond,et al.  A tangible user interface for interactive data visualization , 2015, CASCON.

[19]  Yulong Bian,et al.  A new storytelling genre: combining handicraft elements and storytelling via mixed reality technology , 2020, The Visual Computer.

[20]  Kari Pulli,et al.  Real-time computer vision with OpenCV , 2012, Commun. ACM.

[21]  Wafa Almukadi,et al.  BlackBlocks: Tangible Interactive System for Children to Learn 3-Letter Words and Basic Math , 2015, ITS.

[22]  Wan-Ting Tseng,et al.  Use pattern-recognition-based technology to explore a new interactive system on smart dining table , 2013, 2013 1st International Conference on Orange Technologies (ICOT).

[23]  Hiroshi Kawakami,et al.  Tangible gimmick for programming education using RFID systems , 2016 .

[24]  Ricardo Langner,et al.  CubeQuery: Tangible Interface for Creating and Manipulating Database Queries , 2014, ITS '14.

[25]  Steven K. Feiner,et al.  Opportunistic Tangible User Interfaces for Augmented Reality , 2010, IEEE Transactions on Visualization and Computer Graphics.

[26]  Jae Yeol Lee,et al.  Tangible user interface of digital products in multi-displays , 2012 .

[27]  Barbara Cutler,et al.  Interpreting Physical Sketches as Architectural Models , 2010, AAG.

[28]  Pedro F. Campos,et al.  Designing Augmented Reality Tangible Interfaces for Kindergarten Children , 2011, HCI.

[29]  Luis A. Guerrero,et al.  Kiteracy: a kit of tangible objects to strengthen literacy skills in children with down syndrome , 2015, IDC.

[30]  Olga Sorkine-Hornung,et al.  Tangible and modular input device for character articulation , 2014, SIGGRAPH '14.

[31]  Enrico Pontelli,et al.  MICOO (multimodal interactive cubes for object orientation): a tangible user interface for the blind and visually impaired , 2011, ASSETS '11.

[32]  Stephen A. Brewster,et al.  Clutching at straws: using tangible interaction to provide non-visual access to graphs , 2010, CHI.

[33]  Barbara Cutler,et al.  Evaluation of user interaction with daylighting simulation in a tangible user interface , 2013 .

[34]  Hans-Werner Gellersen,et al.  SPATA: Spatio-Tangible Tools for Fabrication-Aware Design , 2015, TEI.

[35]  Kristian Gohlke,et al.  Physical Construction Toys for Rapid Sketching of Tangible User Interfaces , 2015, Tangible and Embedded Interaction.

[36]  Lijuan Liu,et al.  ModBot: A Tangible and Modular Making Toolkit for Children to Create Underwater Robots , 2020, CHI Extended Abstracts.

[37]  Barbara Cutler,et al.  Interactive selection of optimal fenestration materials for schematic architectural daylighting design , 2008 .

[38]  Bárbara Paola Muro Haro,et al.  Developing reading skills in children with Down syndrome through tangible interfaces , 2012, MexIHC '12.

[39]  Marina Umaschi Bers,et al.  “I want my robot to look for food”: Comparing Kindergartner’s programming comprehension using tangible, graphic, and hybrid user interfaces , 2015 .

[40]  David Reid,et al.  Multimodal Algebra Learning: From Math Manipulatives to Tangible User Interfaces , 2018, i-com.

[41]  Amaan Nathoo,et al.  Exploring the Use of Tangible User Interfaces for Teaching Basic Java Programming Concepts: A Usability Study , 2019, 2019 Conference on Next Generation Computing Applications (NextComp).

[42]  Tomoko Hashida,et al.  floatio: Floating Tangible User Interface Based on Animacy Perception , 2016, UIST.

[43]  Victor M. Ruiz Penichet,et al.  Educational games based on distributed and tangible user interfaces to stimulate cognitive abilities in children with ADHD , 2015, Br. J. Educ. Technol..

[44]  Amaan Nathoo,et al.  Using tangible user interfaces for teaching concepts of internet of things , 2020, Interact. Technol. Smart Educ..

[45]  Hannes Kaufmann,et al.  ACTO: A Modular Actuated Tangible User Interface Object , 2014, ITS '14.

[46]  Orit Shaer,et al.  Smart Blocks: a tangible mathematical manipulative , 2007, TEI.

[47]  Bertrand Schneider,et al.  Preparing for Future Learning with a Tangible User Interface: The Case of Neuroscience , 2013, IEEE Transactions on Learning Technologies.

[48]  Jae Yeol Lee,et al.  Tangible authoring of 3D virtual scenes in dynamic augmented reality environment , 2011, Comput. Ind..

[49]  María Luisa Martín Ruiz,et al.  Foundations of a smart toy development for the early detection of motoric impairments at childhood , 2015 .

[50]  Giovanni Siano,et al.  The Activity Board 1.0: RFID-NFC WI-FI Multitags Desktop Reader for Education and Rehabilitation Applications , 2017, WorldCIST.

[51]  Emily Sun,et al.  Sifteo cubes , 2012, CHI Extended Abstracts.

[52]  Juan R. Velasco,et al.  Smart Toys Designed for Detecting Developmental Delays , 2016, Sensors.

[53]  Torben Wallbaum,et al.  Towards a Tangible Storytelling Kit for Exploring Emotions with Children , 2017, ACM Multimedia.

[54]  Pattie Maes,et al.  Siftables: towards sensor network user interfaces , 2007, TEI.

[55]  Hirokazu Kato,et al.  Collaborative augmented reality , 2002, CACM.

[56]  Rattapoom Waranusast,et al.  Interactive tangible user interface for music learning , 2013, 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013).

[57]  Elise van den Hoven,et al.  Towards Preserving Indigenous Oral Stories Using Tangible Objects , 2011, 2011 Second International Conference on Culture and Computing.

[58]  M. M. Tentzeris,et al.  Novel “smart cube” wireless sensors with embedded processing/communication/power core for “smart skins” applications , 2012, 2012 IEEE Sensors.

[59]  Keyur Sorathia,et al.  Learning and Experience: Teaching Tangible Interaction & Edutainment , 2012 .

[60]  Rachael C. Schindler,et al.  Interactive Block Games for Assessing Children's Cognitive Skills: Design and Preliminary Evaluation , 2018, Front. Pediatr..

[61]  Ali Kashif Bashir,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2013, ICIRA 2013.

[62]  Abdullah Al Mahmud,et al.  POMA: A tangible user interface to improve social and cognitive skills of Sri Lankan children with ASD , 2020, Int. J. Hum. Comput. Stud..

[63]  Eiman Kanjo,et al.  TangToys: smart toys to communicate and improve children's wellbeing , 2020, UbiComp/ISWC Adjunct.

[64]  Christophe Kolski,et al.  RFID-based tangible and touch tabletop for dual reality in crisis management context , 2021, Journal on Multimodal User Interfaces.

[65]  Bertrand Schneider,et al.  Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning , 2016, ACM Trans. Comput. Hum. Interact..

[66]  Eva Hornecker,et al.  Exploring Social Interaction With a Tangible Music Interface , 2016, Interact. Comput..

[67]  Paula O. Fernandes,et al.  Proposal of a Tangible User Interface to Enhance Accessibility in Geological Exhibitions and the Experience of Museum Visitors , 2016 .

[68]  Hongan Wang,et al.  T-Maze: a tangible programming tool for children , 2011, IDC.

[69]  Hiroshi Kawakami,et al.  Improvement of P-CUBE: Algorithm education tool for visually impaired persons , 2014, 2014 IEEE Symposium on Robotic Intelligence in Informationally Structured Space (RiiSS).

[70]  Hiroshi Ishii,et al.  The tangible user interface and its evolution , 2008, CACM.

[71]  Christophe Kolski,et al.  RFID-driven situation awareness on TangiSense, a table interacting with tangible objects , 2011, Personal and Ubiquitous Computing.

[72]  Patrick Jermann,et al.  An empirical evaluation of touch and tangible interfaces for tabletop displays , 2010, TEI '10.

[73]  Matej Zajc,et al.  Design-based research on the use of a tangible user interface for geometry teaching in an inclusive classroom , 2013, Br. J. Educ. Technol..

[74]  Lilia B. Villafuerte,et al.  Acquisition of social abilities through musical tangible user interface: children with autism spectrum condition and the reactable , 2012, CHI Extended Abstracts.

[75]  Girish Bekaroo,et al.  TangiNet: A Tangible User Interface System for Teaching the Properties of Network Cables , 2019, 2019 Conference on Next Generation Computing Applications (NextComp).

[76]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[77]  Hiroshi Ishii,et al.  Sensetable: a wireless object tracking platform for tangible user interfaces , 2001, CHI.

[78]  Ulrich Leopold,et al.  A geospatial tangible user interface to support stakeholder participation in urban planning , 2015, 2015 1st International Conference on Geographical Information Systems Theory, Applications and Management (GISTAM).

[79]  Zhiyi Rong,et al.  CodeRhythm: A Tangible Programming Toolkit for Visually Impaired Students , 2020, AsianCHI@CHI.

[80]  Dan Morris,et al.  Dynamic mapping of physical controls for tabletop groupware , 2009, CHI.