Mediating Human-Robot Interactions with Virtual, Augmented, and Mixed Reality

Effective human-robot interaction (HRI) remains a critical barrier to the successful and widespread deployment of robotic technologies. Fundamentally, HRI problems represent breakdowns in communication, where poor information exchange between people and robots leads to disfluencies, incorrect mental models, poorly calibrated trust, inadequate situational awareness, etc. In this paper, I argue that the emergence of a new generation of virtual, augmented, and mixed reality (VAMR) technologies is creating an exciting new design space in which to address these problems in communication. To support this argument, I present the results of three experiments demonstrating the value in using modern VAMR technologies to mediate human-robot interactions and discuss various challenges and opportunities in this emerging space at the intersection of several communities, including robotics, VR, graphics, and human-computer interaction.

[1]  Matthias Scheutz,et al.  A Dempster-Shafer Theoretic Approach to Understanding Indirect Speech Acts , 2014, IBERAMIA.

[2]  Grigore C. Burdea,et al.  Virtual reality and robotics in medicine , 1996, Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN'96 TSUKUBA.

[3]  Shumin Zhai,et al.  Telerobotic virtual control system , 1992, Other Conferences.

[4]  Justine Cassell,et al.  Embodied Conversational Agents: Representation and Intelligence in User Interfaces , 2001, AI Mag..

[5]  Siddhartha S. Srinivasa,et al.  Predicting User Intent Through Eye Gaze for Shared Autonomy , 2016, AAAI Fall Symposia.

[6]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[7]  Lawrence W. Stark,et al.  Visual enhancements in pick-and-place tasks: Human operators controlling a simulated cylindrical manipulator , 1987, IEEE Journal on Robotics and Automation.

[8]  Bilge Mutlu,et al.  Communication of Intent in Assistive Free Flyers , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[9]  Bilge Mutlu,et al.  Communicating Directionality in Flying Robots , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[10]  Anthony D. Andre,et al.  Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue , 2005 .

[11]  Jessie Y. C. Chen,et al.  Human Performance Issues and User Interface Design for Teleoperated Robots , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[12]  Pei-Luen Patrick Rau,et al.  A Cross-cultural Study: Effect of Robot Appearance and Task , 2010, Int. J. Soc. Robotics.

[13]  Siddhartha S. Srinivasa,et al.  Legibility and predictability of robot motion , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[14]  Kentaro Ishii,et al.  Designing Laser Gesture Interface for Robot Control , 2009, INTERACT.

[15]  Maja J. Matarić,et al.  A Survey of Nonverbal Signaling Methods for Non-Humanoid Robots , 2018, Found. Trends Robotics.

[16]  Won S. Kim,et al.  The phantom robot: predictive displays for teleoperation with time delay , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[17]  R.J. Stone.  Advanced human-system interfaces for telerobotics using virtual reality and telepresence technologies , 1991, Fifth International Conference on Advanced Robotics 'Robots in Unstructured Environments.

[18]  B. Scassellati,et al.  Social eye gaze in human-robot interaction , 2017, J. Hum. Robot Interact..

[19]  Ana Paiva,et al.  Automatic analysis of affective postures and body motion to detect engagement with a game companion , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[20]  Thomas B. Moeslund,et al.  Projecting robot intentions into human environments , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[21]  Paul Milgram,et al.  Telerobotic control using augmented reality , 1995, Proceedings 4th IEEE International Workshop on Robot and Human Communication.

[22]  Eckhard Freund,et al.  Projective virtual reality: bridging the gap between virtual reality and robotics , 1999, IEEE Trans. Robotics Autom..

[23]  David Whitney,et al.  ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Sjoerd C. de Vries,et al.  Steering a simulated unmanned aerial vehicle using a head-slaved camera and HMD , 1997, Defense, Security, and Sensing.

[25]  Hooman Hedayati,et al.  Improving Collocated Robot Teleoperation with Augmented Reality , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[26]  Shin Sato,et al.  A human-robot interface using an interactive hand pointer that projects a mark in the real work space , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[27]  J. Geoffrey Chase,et al.  Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design , 2008 .

[28]  Gregory M. P. O'Hare,et al.  Using Mixed Reality Agents as Social Interfaces for Robots , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[29]  Bilge Mutlu,et al.  Anticipatory robot control for efficient human-robot collaboration , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[30]  Jennifer Lee,et al.  Communicating Robot Motion Intent with Augmented Reality , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[31]  Stephanie Rosenthal,et al.  Enhancing human understanding of a mobile robot's state and actions using expressive lights , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[32]  Martijn J. Schuemie,et al.  Research on Presence in Virtual Reality: A Survey , 2001, Cyberpsychology Behav. Soc. Netw..

[33]  J. R. Spofford,et al.  Machine vision augmented displays for teleoperation , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[34]  Jonathan P. How,et al.  MAR-CPS: Measurable Augmented Reality for Prototyping Cyber-Physical Systems , 2015 .

[35]  Donald A. Norman,et al.  Stages and Levels in Human-Machine Interaction , 1984, Int. J. Man Mach. Stud..

[36]  Mark Billinghurst,et al.  A Survey of Augmented Reality , 2015, Found. Trends Hum. Comput. Interact..

[37]  Roger A. Browse,et al.  The effectiveness of real-time graphic simulation in telerobotics , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[38]  Michael A. Goodrich,et al.  Ecological Interfaces for Improving Mobile Robot Teleoperation , 2007, IEEE Transactions on Robotics.

[39]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[40]  Terrence Fong,et al.  VEVI: A Virtual Environment Teleoperations Interface for Planetary Exploration , 1995 .

[41]  Steven K. Feiner,et al.  Windows on the world: 2D windows for 3D augmented reality , 1993, UIST '93.

[42]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[43]  Abraham G. Campbell,et al.  MiRA - Mixed Reality Agents , 2011, Int. J. Hum. Comput. Stud..

[44]  Grigore C. Burdea,et al.  Invited review: the synergy between virtual reality and robotics , 1999, IEEE Trans. Robotics Autom..

[45]  Ross A. Knepper,et al.  Asking for Help Using Inverse Semantics , 2014, Robotics: Science and Systems.

[46]  Zhijun Zhang,et al.  Human–Robot Interaction by Understanding Upper Body Gestures , 2014, PRESENCE: Teleoperators and Virtual Environments.

[47]  Paul Milgram,et al.  Telerobotic control with stereoscopic augmented reality , 1996, Electronic Imaging.

[48]  Bilge Mutlu,et al.  Using gaze patterns to predict task intent in collaboration , 2015, Front. Psychol..

[49]  Atsushi Watanabe,et al.  Communicating robotic navigational intentions , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[50]  Ana Paiva,et al.  Expressive Lights for Revealing Mobile Service Robot State , 2015, ROBOT.

[51]  Alexander H. Waibel,et al.  Natural human-robot interaction using speech, head pose and gestures , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[52]  Donald A. Norman,et al.  Cognitive artifacts , 1991 .

[53]  I. Ince,et al.  Virtuality and reality: a video/graphics environment for teleoperation , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[54]  Nadine Miner,et al.  An interactive virtual reality simulation system for robot control and operator training , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[55]  P. Mojiri Forooshani,et al.  From ROS to unity: Leveraging robot and virtual environment middleware for immersive teleoperation , 2014, 2014 IEEE International Conference on Information and Automation (ICIA).

[56]  Abraham G. Campbell,et al.  Why, when and how to use augmented reality agents (AuRAs) , 2013, Virtual Reality.

[57]  Nicholas R. Gans,et al.  A Multi-view camera-projector system for object detection and robot-human feedback , 2013, 2013 IEEE International Conference on Robotics and Automation.

[58]  Hooman Hedayati,et al.  Robot Teleoperation with Augmented Reality Virtual Surrogates , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[59]  Alan Natapoff,et al.  Influence of perspective-taking and mental rotation abilities in space teleoperation , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[60]  J. Gregory Trafton,et al.  Enabling effective human-robot interaction using perspective-taking in robots , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[61]  Ravi Teja Chadalavada,et al.  That's on my mind! robot to human intention communication through on-board projection on shared floor space , 2015, 2015 European Conference on Mobile Robots (ECMR).

[62]  Daniel Szafir,et al.  Improving Object Disambiguation from Natural Language using Empirical Models , 2018, ICMI.

[63]  Hua Li,et al.  Review and analysis of avionic helmet-mounted displays , 2013 .

[64]  Shumin Zhai,et al.  Applications of augmented reality for human-robot communication , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).

[65]  Rachid Alami,et al.  A methodological approach relating the classification of gesture to identification of human intent in the context of human-robot interaction , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[66]  Magdalena D. Bugajska,et al.  Building a Multimodal Human-Robot Interface , 2001, IEEE Intell. Syst..

[67]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[68]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[69]  Allison Sauppé,et al.  Robot Deictics: How Gesture and Context Shape Referential Communication , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[70]  Ivan E. Sutherland,et al.  The Ultimate Display , 1965 .