Enhancing telepresence during the teleoperation of road vehicles using HMD-based mixed reality

A lack of telepresence is one of the main challenges of vehicle teleoperation, which degrades the task performance of the human operator. This paper introduces a novel human-machine interface (HMI) using a head-mounted display (HMD) to improve the situation awareness and, consequently, the telepresence of the human operator. The proposed HMI concept uses the transmitted data of the camera as well as LiDAR sensors of the remote vehicle to illustrate the 360° vehicle surroundings as a mixture of the real and virtual environments to the human operator. The resulting system provides the possibility to precisely control the remote vehicle with a low additional load to the transmitted data. The developed concept is evaluated by experienced operators within different test scenarios. The results of the test drives show a significant improvement of the task performance as well as a reduction of the workload of the human operators using the proposed HMI concept during control of the teleoperated vehicle.

[1]  David B. Kaber,et al.  Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation , 2004 .

[2]  Linda R. Elliott,et al.  Robotic Telepresence: Perception, Performance, and User Experience , 2012 .

[3]  Terrence Fong,et al.  Vehicle Teleoperation Interfaces , 2001, Auton. Robots.

[4]  Mica R. Endsley,et al.  Design and Evaluation for Situation Awareness Enhancement , 1988 .

[5]  Tang Chen,et al.  Methods for Improving the Control of Teleoperated Vehicles , 2015 .

[6]  Naoji Shiroma,et al.  Hybrid head mounted/surround display for telexistence/telepresence and behavior navigation , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[7]  Ian Oakley,et al.  Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots , 2014, Robotica.

[8]  Thomas B. Sheridan,et al.  Musings on Telepresence and Virtual Presence , 1992, Presence: Teleoperators & Virtual Environments.

[9]  B. Roberts,et al.  A virtual environment for learning to pilot remotely operated vehicles , 1997, Proceedings. International Conference on Virtual Systems and MultiMedia VSMM '97 (Cat. No.97TB100182).

[10]  T. Bock,et al.  Validation of the Vehicle in the Loop (VIL); A milestone for the simulation of driver assistance systems , 2007, 2007 IEEE Intelligent Vehicles Symposium.

[11]  Steve Benford,et al.  The augurscope: a mixed reality interface for outdoors , 2002, CHI.

[12]  Markus Lienkamp,et al.  Interactive path planning for teleoperated road vehicles in urban environments , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[13]  Terrence Fong,et al.  Advanced Interfaces for Vehicle Teleoperation: Collaborative Control, Sensor Fusion Displays, and Remote Driving Tools , 2001, Auton. Robots.

[14]  Mengmeng Yu,et al.  A visual parking guidance for surround view monitoring system , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[15]  B. Krenik.  4G wireless technology: When will it happen? What does it offer? , 2008, 2008 IEEE Asian Solid-State Circuits Conference.

[16]  Giovanni Muscato,et al.  Stereo Viewing and Virtual Reality Technologies in Mobile Robot Teleguide , 2009, IEEE Transactions on Robotics.

[17]  Tomasz Mazuryk,et al.  Virtual Reality History , Applications , Technology and Future , 1999 .

[18]  Chao Li,et al.  Tele-AR System Based on Real-time Camera Tracking , 2012, 2012 International Conference on Virtual Reality and Visualization.

[19]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[20]  Markus Lienkamp,et al.  A System Design for Teleoperated Road Vehicles , 2013, ICINCO.

[21]  Alexander Ferrein,et al.  Intuitive visual teleoperation for UGVs using free-look augmented reality displays , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).