Multi-robot Systems, Virtual Reality and ROS: Developing a New Generation of Operator Interfaces

This chapter describes a series of works developed in order to integrate ROS-based robots with Unity-based virtual reality interfaces. The main goal of this integration is to develop immersive monitoring and commanding interfaces, able to improve the operator’s situational awareness without increasing its workload. In order to achieve this, the available technologies and resources are analyzed and multiple ROS packages and Unity assets are applied, such as \(multimaster\_fkie\), \(rosbridge\_suite\), RosBridgeLib and SteamVR. Moreover, three applications are presented: an interface for monitoring a fleet of drones, another interface for commanding a robot manipulator and an integration of multiple ground and aerial robots. Finally, some experiences and lessons learned, useful for future developments, are reported.

[1]  Antonio Barrientos,et al.  A Multirobot System for Distributed Area Coverage and Signal Searching in Large Outdoor Scenarios * , 2016, J. Field Robotics.

[2]  Tomás Svoboda,et al.  TRADR Project: Long-Term Human-Robot Teaming for Robot Assisted Disaster Response , 2015, KI - Künstliche Intelligenz.

[3]  Giovanni Muscato,et al.  UAV/UGV cooperation for surveying operations in humanitarian demining , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[4]  Ehud Sharlin,et al.  Flying Frustum: A Spatial Interface for Enhancing Human-UAV Awareness , 2015, HAI.

[5]  Christopher D. Wickens,et al.  Mission Control of Multiple Unmanned Aerial Vehicles: A Workload Analysis , 2005, Hum. Factors.

[6]  Antonio Barrientos,et al.  Using ROS in Multi-robot Systems: Experiences and Lessons Learned from Real-World Field Tests , 2017 .

[7]  A. O. Dick,et al.  Operator Workload: Comprehensive Review and Evaluation of Operator Workload Methodologies , 1989 .

[8]  Jeffrey T. Hansberger,et al.  Development of the Next Generation of Adaptive Interfaces , 2015 .

[9]  Martin Molina,et al.  A flexible and dynamic mission planning architecture for UAV swarm coordination , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[10]  N. Moray.  Mental workload : its theory and measurement , 1979 .

[11]  Antonio Barrientos,et al.  An Air-Ground Wireless Sensor Network for Crop Monitoring , 2011, Sensors.

[12]  MengChu Zhou,et al.  Optimizing Operator–Agent Interaction in Intelligent Adaptive Interface Design: A Conceptual Framework , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[13]  Antonio Barrientos,et al.  Heterogeneous Multi-Robot System for Mapping Environmental Variables of Greenhouses , 2016, Sensors.

[14]  Daniel Serrano,et al.  The EU-ICARUS project: Developing assistive robotic tools for search and rescue operations , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[15]  Odest Chadwicke Jenkins,et al.  Rosbridge: ROS for Non-ROS Users , 2011, ISRR.

[16]  Katja Hölttä-Otto,et al.  How users adjust trust in automation , 2016 .

[17]  Steven M. LaValle,et al.  RRT-connect: An efficient approach to single-query path planning , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[18]  Mykola Pechenizkiy,et al.  Stress detection from speech and Galvanic Skin Response signals , 2013, Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems.

[19]  Mary L. Cummings,et al.  Boredom and Distraction in Multiple Unmanned Vehicle Supervisory Control , 2013, Interact. Comput..

[20]  Antonio Barrientos,et al.  Integrating 3D Reconstruction and Virtual Reality: A New Approach for Immersive Teleoperation , 2017, ROBOT.

[21]  David Baran,et al.  Application of Multi-Robot Systems to Disaster-Relief Scenarios with Limited Communication , 2015, FSR.

[22]  S. Kavitha,et al.  Development of automatic speech recognition system for voice activated Ground Control system , 2015, 2015 International Conference on Trends in Automation, Communications and Computing Technology (I-TACT-15).

[23]  Kostas J. Kyriakopoulos,et al.  Multi-robot multiple hypothesis tracking for pedestrian tracking , 2011, 2011 19th Mediterranean Conference on Control & Automation (MED).

[24]  Antonio Barrientos,et al.  An Aerial–Ground Robotic System for Navigation and Obstacle Mapping in Large Outdoor Areas , 2013, Sensors.

[25]  Jean Scholtz,et al.  Evaluation of human-robot interaction awareness in search and rescue , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[26]  Miguel A. Olivares-Méndez,et al.  Analyzing and improving multi-robot missions by using process mining , 2018, Auton. Robots.

[27]  Fernando Jaureguizar,et al.  New generation of human machine interfaces for controlling UAV through depth-based gesture recognition , 2014, Defense + Security Symposium.

[28]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[29]  Greg Mori,et al.  Integrating multi-modal interfaces to command UAVs , 2014, HRI.

[30]  Antonio Franchi,et al.  Decentralized simultaneous multi-target exploration using a connected network of multiple robots , 2015, Auton. Robots.

[31]  Dmitry Kalyadin,et al.  Sea Robot-Assisted Inspection , 2011, IEEE Robotics & Automation Magazine.

[32]  Lydia E. Kavraki,et al.  The Open Motion Planning Library , 2012, IEEE Robotics & Automation Magazine.

[33]  Howie Choset,et al.  Leap-Frog Path Design for Multi-Robot Cooperative Localization , 2009, FSR.

[34]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[35]  P. Mojiri Forooshani,et al.  From ROS to unity: Leveraging robot and virtual environment middleware for immersive teleoperation , 2014, 2014 IEEE International Conference on Information and Automation (ICIA).

[36]  Miguel A. Olivares-Méndez,et al.  Multi-Robot Interfaces and Operator Situational Awareness: Study of the Impact of Immersion and Prediction , 2017, Sensors.

[37]  Soon-Geul Lee,et al.  Complete coverage path planning for multi-robots based on , 2011, 2011 11th International Conference on Control, Automation and Systems.

[38]  Hendrik Wöhrle,et al.  An Intelligent Man-Machine Interface—Multi-Robot Control Adapted for Task Engagement Based on Single-Trial Detectability of P300 , 2016, Front. Hum. Neurosci..

[39]  Thibault Gateau,et al.  A distributed architecture for supervision of autonomous multi-robot missions , 2016, Auton. Robots.

[40]  Jessie Y. C. Chen,et al.  Human Performance Issues and User Interface Design for Teleoperated Robots , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[41]  Mica R. Endsley,et al.  Situation awareness global assessment technique (SAGAT) , 1988, Proceedings of the IEEE 1988 National Aerospace and Electronics Conference.

[42]  Daniel Carbonell,et al.  Robotnik—Professional Service Robotics Applications with ROS , 2016 .

[43]  Domenico Prattichizzo,et al.  Cooperative human-robot haptic navigation , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[44]  Li Li,et al.  The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research , 2017, i-Perception.

[45]  Corinna E. Lathan,et al.  The Effects of Operator Spatial Perception and Sensory Feedback on Human-Robot Teleoperation Performance , 2002, Presence: Teleoperators & Virtual Environments.

[46]  Daniel McDuff,et al.  Remote measurement of cognitive stress via heart rate variability , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[47]  Luca Maria Gambardella,et al.  Human Control of UAVs using Face Pose Estimates and Hand Gestures , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[48]  Mica R. Endsley,et al.  Design and Evaluation for Situation Awareness Enhancement , 1988 .

[49]  Michael A. Goodrich,et al.  Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface Approaches , 2006 .

[50]  Elias B. Kosmatopoulos,et al.  Real-time adaptive multi-robot exploration with application to underwater map construction , 2016, Auton. Robots.

[51]  A. Ollero,et al.  Immersive displays for building spatial knowledge in multi-UAV operations , 2015, 2015 International Conference on Unmanned Aircraft Systems (ICUAS).

[52]  K Foit.  Mixed Reality as a Tool Supporting Programming of the Robot , 2014 .

[53]  Neville Stanton,et al.  Situation awareness measurement: a review of applicability for C4i environments. , 2006, Applied ergonomics.

[54]  Jean Scholtz,et al.  Awareness in human-robot interactions , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[55]  Sergi Hernández Juan,et al.  Multi-master ROS systems , 2015 .