The Reality-Virtuality Interaction Cube

There has recently been an explosion of work in the humanrobot interaction (HRI) community on the use of mixed, augmented, and virtual reality. In this paper, we present a novel conceptual framework to characterize and cluster work in this new area and identify gaps for future research. We begin by introducing the Plane of Interaction: a framework for characterizing interactive technologies in a 2D space informed by the Model-View-Controller design pattern. We then describe how Interaction Design Elements that contribute to the interactivity of a technology can be characterized within this space and present a taxonomy of Mixed-Reality Interaction Design Elements. We then discuss how these elements may be rendered onto both realityand virtuality-based environments using a variety of hardware devices and introduce the Reality-Virtuality Interaction Cube: a three-dimensional continuum representing the design space of interactive technologies formed by combining the Plane of Interaction with the Reality-Virtuality Continuum. Finally, we demonstrate the feasibility and utility of this framework by clustering and analyzing the set of papers presented at the recent 2018 VAM-HRI Workshop.

[1]  Nhan Tran.  A Hands-Free Virtual-Reality Teleoperation Interface for Wizard-of-Oz Control , 2018 .

[2]  J. Geoffrey Chase,et al.  Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design , 2008 .

[3]  Wendy Ju,et al.  VR-OOM: Virtual Reality On-rOad driving siMulation , 2018, CHI.

[4]  Stefanie Tellex,et al.  Testing Robot Teleoperation using a Virtual Reality Interface with ROS Reality , 2018 .

[5]  Sven Wachsmuth,et al.  Improving Human-Robot Handover Research by Mixed Reality Techniques , 2018, HRI 2018.

[6]  Holly A. Yanco,et al.  Remotely Teleoperating a Humanoid Robot to Perform Fine Motor Tasks with Virtual Reality – 18446 , 2018 .

[7]  Elizabeth A. Croft,et al.  Robot Programming Through Augmented Trajectories in Augmented Reality , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  Subbarao Kambhampati,et al.  Projection-Aware Task Planning and Execution for Human-in-the-Loop Operation of Robots in a Mixed-Reality Workspace , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Philippe Coiffet,et al.  Virtual Reality Technology , 2003, Presence: Teleoperators & Virtual Environments.

[10]  Philippe Fuchs,et al.  An Immersive Virtual Reality System for Semi-autonomous Driving Simulation: A Comparison between Realistic and 6-DoF Controller-based Interaction , 2017, ICCAE '17.

[11]  Tathagata Chakraborti,et al.  Virtual, Augmented, and Mixed Reality for Human-Robot Interaction (VAM-HRI) , 2019, HRI.

[12]  Subbarao Kambhampati,et al.  v2v Communication for Augmenting Reality Enabled Smart HUDs to Increase Situational Awareness of Drivers , 2018 .

[13]  Stefanie Tellex,et al.  Learning from Crowdsourced Virtual Reality Demonstrations , 2018 .

[14]  Neil T. Dantam,et al.  Augmented, Mixed, and Virtual Reality Enabling of Robot Deixis , 2018, HCI.

[15]  Daniel Szafir,et al.  Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[16]  Thomas B. Moeslund,et al.  Projecting robot intentions into human environments , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[17]  Ramviyas Parasuraman,et al.  VR Based Robot Teleoperation Interface for Virtual Tour , 2018 .

[18]  Shumin Zhai,et al.  Applications of augmented reality for human-robot communication , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).

[19]  Björn Hein,et al.  Implementation of Augmented Reality in Autonomous Warehouses: Challenges and Opportunities , 2018, HRI 2018.

[20]  Jeremy A. Marvel,et al.  Towards Augmented Reality Interfaces for Human-Robot Interaction in Manufacturing Environments | NIST , 2018 .

[21]  Mark Billinghurst,et al.  A Survey of Augmented Reality , 2015, Found. Trends Hum. Comput. Interact..

[22]  Jacek Malec,et al.  Remote Operation of Unmanned Surface Vessel through Virtual Reality , 2018, HRI 2018.

[23]  Michael Wooldridge,et al.  The Belief-Desire-Intention Model of Agency , 1998, ATAL.

[24]  Eike Langbehn,et al.  Detection Thresholds for Rotation and Translation Gains in 360° Video-Based Telepresence Systems , 2018, IEEE Transactions on Visualization and Computer Graphics.

[25]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[26]  Stephen Travis Pope,et al.  A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System , 1998 .

[27]  Jingxin Zhang.  A 360 ◦ Video-based Robot Platform for Telepresent Redirected Walking , 2018 .

[28]  Frank Steinicke,et al.  Excuse me! Perception of Abrupt Direction Changes Using Body Cues and Paths on Mixed Reality Avatars , 2018, HRI.

[29]  Elizabeth Boyle,et al.  Mixed Reality Deictic Gesture for Multi-Modal Robot Communication , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[30]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[31]  Chris Rogers.  Towards an Augmented Reality Framework for K-12 Robotics Education , 2018 .

[32]  Moritz Tenorth,et al.  KNOWROB — knowledge processing for autonomous personal robots , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  William R. Sherman,et al.  Understanding Virtual Reality , 2003 .

[34]  Hansen,et al.  Collaborative Meat Processing in Virtual Reality , 2018 .

[35]  Tom Williams,et al.  A Framework for Robot-Generated Mixed-Reality Deixis , 2018 .

[36]  David Whitney,et al.  Comparing Robot Grasping Teleoperation Across Desktop and Virtual Reality with ROS Reality , 2017, ISRR.

[37]  Fangkai Yang,et al.  A Virtual Poster Presenter Using Mixed Reality , 2017, IVA.

[38]  M. Bennett.  Investigating Interactions with Teleoperated and Autonomous Humanoids Using a Suit-Based VR Teleoperation Interface , 2018 .

[39]  Heather M. Ross,et al.  Better Teaming Through Visual Cues: How Projecting Imagery in a Workspace Can Improve Human-Robot Collaboration , 2018, IEEE Robotics & Automation Magazine.

[40]  Matthias Scheutz,et al.  Differences in interaction patterns and perception for teleoperated and autonomous humanoid robots , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[41]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..