Improved Situational Awareness in ROS Using Panospheric Vision and Virtual Reality

One of the main difficulties in teleoperated systems is providing an operator with sufficient Situational Awareness (SA). This paper introduces three open-source packages that improve the operator's SA using the Robot Operating System (ROS). The first package-rviz_textured_sphere-allows rendering panospheric camera outputs as spherical images in the ROS visualization software RViz. A system where the visualization of this spherical data using an open-source virtual reality (OSVR) headset in the ROS framework is achieved with the second package: rviz_plugin_osvr. Finally, the third package-pointcloud_painter-projects spherical data onto a 3D depth cloud scan of the scene generated from a rotating lidar. This package outputs a XYZRGB pointcloud that can be visualized either in RViz or using the virtual reality headset. Together, these technologies address the wider issue of limited SA in robotics and represent a substantial advancement in the environment visualization capabilities available to open-source robotics developers.

[1]  Karl Kruusamäe,et al.  TeMoto: Intuitive Multi-Range Telerobotic System with Natural Gestural and Verbal Instruction Interface , 2018, Robotics.

[2]  Dean Brown,et al.  Decentering distortion of lenses , 1966 .

[3]  Martin L. Yarmush,et al.  Adaptive Kinematic Control of a Robotic Venipuncture Device Based on Stereo Vision, Ultrasound, and Force Guidance , 2017, IEEE Transactions on Industrial Electronics.

[4]  William D. Smart,et al.  Programming Robots with ROS: A Practical Introduction to the Robot Operating System , 2015 .

[5]  Miguel A. Olivares-Méndez,et al.  Multi-Robot Interfaces and Operator Situational Awareness: Study of the Impact of Immersion and Prediction , 2017, Sensors.

[6]  Kai-Tai Song,et al.  Virtual reality control of a robotic camera holder for minimally invasive surgery , 2017, 2017 11th Asian Control Conference (ASCC).

[7]  Jean Scholtz,et al.  Analysis of human–robot interaction at the DARPA Robotics Challenge Finals , 2017, Int. J. Robotics Res..

[8]  A. E. Conrady.  Decentred Lens-Systems , 1919 .

[9]  José García Rodríguez,et al.  Three-dimensional planar model estimation using multi-constraint knowledge based on k-means and RANSAC , 2015, Appl. Soft Comput..

[10]  Stefan Hrabar.  An evaluation of stereo and laser-based range sensing for rotorcraft unmanned aerial vehicle obstacle avoidance , 2012, J. Field Robotics.

[11]  Rodrigo Ventura,et al.  Immersive 3-D teleoperation of a search and rescue robot using a head-mounted display , 2009, 2009 IEEE Conference on Emerging Technologies & Factory Automation.

[12]  Kazuya Yoshida,et al.  Emergency response to the nuclear accident at the Fukushima Daiichi Nuclear Power Plants using mobile rescue robots , 2013, J. Field Robotics.

[13]  Samuel A. Burden,et al.  The human should be part of the control loop? , 2016, 2016 Resilience Week (RWS).

[14]  D. Nguyen,et al.  Automated 3D architecture reconstruction from photogrammetric structure-and-motion: A case study of the One Pilla pagoda, Hanoi, Vienam , 2015 .

[15]  Russell M. Taylor,et al.  OSVR: An open-source virtual reality platform for both industry and academia , 2015, 2015 IEEE Virtual Reality (VR).

[16]  Luis Payá,et al.  A State-of-the-Art Review on Mapping and Localization of Mobile Robots Using Omnidirectional Vision Sensors , 2017, J. Sensors.

[17]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[18]  Mica R. Endsley,et al.  Measurement of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[19]  S. L. Bogner.  Application of panospheric imaging to an armoured vehicle viewing system , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[20]  Jaroslaw Jankowski,et al.  Usability Evaluation of VR Interface for Mobile Robot Teleoperation , 2015, Int. J. Hum. Comput. Interact..

[21]  Sven G. Kratz,et al.  Immersed remotely: Evaluating the use of Head Mounted Devices for remote collaboration in robotic telepresence , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[22]  Gu Fang,et al.  Autonomous weld seam identification and localisation using eye-in-hand stereo vision for robotic arc welding , 2013 .

[23]  Benjamin Watson,et al.  Using texture maps to correct for optical distortion in head-mounted displays , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[24]  M. Westoby,et al.  ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications , 2012 .