A Unified Design & Development Framework for Mixed Interactive Systems

Mixed reality, natural user interfaces and the internet of things are complementary computing paradigms. They converge towards news form of interactive systems named mixed interactive systems. Because of their exploding complexity, mixed interactive systems induce new challenges for designers and developers. We need new abstractions of these systems in order to describe their real-virtual interplay. We also need to break mixed interactive systems down into pieces in order to segment their complexity into comprehensible subsystems. This paper presents a framework to enhance the design and development of these systems. We propose a model unifying the paradigms of mixed reality, natural user interfaces and the internet of things. Our model decomposes a mixed interactive system into a graph of mixed entities. Our framework implements this model, which facilitates interactions between users, mixed reality devices and connected objects. In order to demonstrate our approach, we present how designers and developers can use this framework to develop a mixed interactive system dedicated to smart building occupants.

[1]  Luigi Atzori,et al.  The Virtual Object as a Major Element of the Internet of Things: A Survey , 2016, IEEE Communications Surveys & Tutorials.

[2]  Donald A. Norman,et al.  Natural user interfaces are not natural , 2010, INTR.

[3]  Markus Funk,et al.  The virtual twin: controlling smart factories using a spatially-correct augmented reality representation , 2017, IOT.

[4]  Martin Wagner,et al.  Herding Sheep: Live System Development for Distributed Augmented Reality , 2003 .

[5]  Bruno Arnaldi,et al.  Versatile Scenario Guidance for Collaborative Virtual Environments , 2015, GRAPP.

[6]  Christophe Bortolaso,et al.  An MDE-based framework to support the development of Mixed Interactive Systems , 2014, Sci. Comput. Program..

[7]  D. Carrington,et al.  Comparing the MVC and PAC architectures: a formal perspective , 1997, IEE Proc. Softw. Eng..

[8]  Edward Griffor,et al.  Cyber-physical systems and internet of things , 2019 .

[9]  Éric Marchand,et al.  Pose Estimation for Augmented Reality: A Hands-On Survey , 2016, IEEE Transactions on Visualization and Computer Graphics.

[10]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[11]  Stefan Poslad,et al.  Ubiquitous Computing: Smart Devices, Environments and Interactions , 2009 .

[12]  Damla Turgut,et al.  A Systematic Review of the Convergence of Augmented Reality, Intelligent Virtual Agents, and the Internet of Things , 2019, Artificial Intelligence in IoT.

[13]  Hugh F. Durrant-Whyte,et al.  Simultaneous map building and localization for an autonomous mobile robot , 1991, Proceedings IROS '91:IEEE/RSJ International Workshop on Intelligent Robots and Systems '91.

[14]  Kangsoo Kim,et al.  Does a Digital Assistant Need a Body? The Influence of Visual Embodiment and Social Behavior on the Perception of Intelligent Virtual Agents in AR , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[15]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[16]  Michael J. McGuffin,et al.  Enlarging a Smartphone with AR to Create a Handheld VESAD (Virtually Extended Screen-Aligned Display) , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[17]  Weiyuan Liu,et al.  Natural user interface- next mainstream product user interface , 2010, 2010 IEEE 11th International Conference on Computer-Aided Industrial Design & Conceptual Design 1.

[18]  Nadine Pfeiffer-Leßmann,et al.  Virtual Prototyping of Mixed Reality Interfaces with Internet of Things (IoT) Connectivity , 2018, i-com.

[19]  P. Seeling,et al.  An off-the-shelf wearable HUD system for support in indoor environments , 2012, MUM.

[20]  Antonio Iera,et al.  Understanding the Internet of Things: definition, potentials, and societal role of a fast evolving paradigm , 2017, Ad Hoc Networks.

[21]  Elodie Bouzekri,et al.  A Generic Software and Hardware Architecture for Hybrid Interactive Systems (EICS 2018 - Workshop on Heterogeneous Models and Modeling Approaches for Engineering of Interactive Systems, Paris, France, 19/06/2018-19/06/2018) , 2020 .

[22]  Laurence Nigay,et al.  The Engineering of Mixed Reality Systems , 2009, Human-Computer Interaction Series.

[23]  Edward H. Glaessgen,et al.  The Digital Twin Paradigm for Future NASA and U.S. Air Force Vehicles , 2012 .

[24]  Bruno Arnaldi,et al.  #FIVE : High-level components for developing collaborative and interactive virtual environments , 2015, 2015 IEEE 8th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS).

[25]  Michael W. Grieves.  Virtually Intelligent Product Systems: Digital and Physical Twins , 2019, Complex Systems Engineering: Theory and Practice.

[26]  Yu Yuan,et al.  All Reality: Virtual, Augmented, Mixed (X), Mediated (X, Y), and Multimediated Reality , 2018, ArXiv.

[27]  Pattie Maes,et al.  SixthSense: a wearable gestural interface , 2009, SIGGRAPH ASIA Art Gallery & Emerging Technologies.

[28]  Pan Hui,et al.  Ubii: Physical World Interaction Through Augmented Reality , 2017, IEEE Transactions on Mobile Computing.

[29]  Damith Chinthana Ranasinghe,et al.  Adding sense to the Internet of Things , 2011, Personal and Ubiquitous Computing.

[30]  Didier Stricker,et al.  Continuous natural user interface: Reducing the gap between real and digital world , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[31]  Henrik Edlund,et al.  [POSTER] AR as a User Interface for The Internet of Things—Comparing Three Interaction Models , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[32]  Vera Stavroulaki,et al.  Virtualization and Cognitive Management of Real World Objects in the Internet of Things , 2012, 2012 IEEE International Conference on Green Computing and Communications.

[33]  Jérémy Lacoche,et al.  Model and Tools for Integrating IoT into Mixed Reality Environments: Towards a Virtual-Real Seamless Continuum , 2019, ICAT-EGVE.

[34]  Céline Coutrix,et al.  Mixed reality: a model of mixed interaction , 2006, AVI '06.