High-Level Teleoperation System for Aerial Exploration of Indoor Environments

Exploration of challenging indoor environments is a demanding task. While automation with aerial robots seems a promising solution, fully autonomous systems still struggle with high-level cognitive tasks and intuitive decision making. To facilitate automation, we introduce a novel teleoperation system with an aerial telerobot that is capable of handling all demanding low-level tasks. Motivated by the typical structure of indoor environments, the system creates an interactive scene topology in real-time that reduces scene details and supports affordances. Thus, difficult high-level tasks can be effectively supervised by a human operator. To elaborate on the effectiveness of our system during a real-world exploration mission, we conducted a user study. Despite being limited by real-world constraints, results indicate that our system better supports operators with indoor exploration, compared to a baseline system with traditional joystick control.

[1]  Dario Floreano,et al.  FlyJacket: An Upper Body Soft Exoskeleton for Immersive Drone Control , 2018, IEEE Robotics and Automation Letters.

[2]  Hooman Hedayati,et al.  Improving Collocated Robot Teleoperation with Augmented Reality , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[3]  Thomas B. Sheridan,et al.  Human and Computer Control of Undersea Teleoperators , 1978 .

[4]  Kosuke Sato,et al.  FlyingHand: extending the range of haptic feedback on virtual hand using drone-based object recognition , 2018, SIGGRAPH Asia Technical Briefs.

[5]  A. R. Jha,et al.  UNMANNED AERIAL VEHICLES FOR MILITARY APPLICATIONS , 2016 .

[6]  Michael A. Goodrich,et al.  Sliding Autonomy for UAV Path-Planning: Adding New Dimensions to Autonomy Management , 2015, AAMAS.

[7]  Lakshminarayanan Subramanian,et al.  WiLDNet: Design and Implementation of High Performance WiFi Based Long Distance Networks , 2007, NSDI.

[8]  Guowei Cai,et al.  A Survey of Small-Scale Unmanned Aerial Vehicles: Recent Advances and Future Development Trends , 2014 .

[9]  Dieter Schmalstieg,et al.  Drone-Augmented Human Vision: Exocentric Control for Drones Exploring Hidden Areas , 2018, IEEE Transactions on Visualization and Computer Graphics.

[10]  R. G. Steinhauer,et al.  The final frontier. , 2000, JEMS : a journal of emergency medical services.

[11]  Jan-Louis Kruger,et al.  Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2 , 2014 .

[12]  Philip David,et al.  Ascending stairway modeling from dense depth imagery for traversability analysis , 2013, 2013 IEEE International Conference on Robotics and Automation.

[13]  James L. Adams An Investigation of the Effects of the Time Lag due to Long Transmission Distances Upon Remote Control. Phase 1; Tracking Experiments , 1961 .

[14]  David Kortenkamp,et al.  Cognitive maps for mobile robots: A representation for mapping and navigation , 1993 .

[15]  Edwin Olson,et al.  AprilTag: A robust and flexible visual fiducial system , 2011, 2011 IEEE International Conference on Robotics and Automation.

[16]  Donald D. Dudenhoeffer,et al.  Dynamic-Autonomy for Urban Search and Rescue , 2002, AAAI Mobile Robot Competition.

[17]  Robin R. Murphy,et al.  Situation Awareness and Task Performance in Robot-Assisted Technical Search: Bujold Goes to Bridg , 2004 .

[18]  Ke Wang,et al.  Visual Semantic Navigation Based on Deep Learning for Indoor Mobile Robots , 2018, Complex..

[19]  Richard M. Murray,et al.  A Mathematical Introduction to Robotic Manipulation , 1994 .

[20]  Reza Bairam Zadeh,et al.  Autonomy and Unmanned Vehicles: Augmented Reactive Mission and Motion Planning Architecture , 2018 .

[21]  James L. Adams AN INVESTIGATION OF THE EFFECTS OF THE TIME LAG DUE TO LONG TRANSMISSION DISTANCES UPON REMOTE CONTROL. PHASE II - VEHICLE EXPERIMENTS. PHASE III - CONCLUSIONS. , 1961 .

[22]  Mica R. Endsley,et al.  Level of Automation Forms a Key Aspect of Autonomy Design , 2018 .

[23]  Bilge Mutlu,et al.  An Autonomous Dynamic Camera Method for Effective Remote Teleoperation , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[24]  Michael Beetz,et al.  RG Mapping : Building Object-Oriented Representations of Structured Human Environments , .

[25]  Marcello Chiaberge,et al.  Multipurpose UAV for search and rescue operations in mountain avalanche events , 2017 .

[26]  Terrence Fong,et al.  Multi-robot remote driving with collaborative control , 2003, IEEE Trans. Ind. Electron..

[27]  Massimo Satler,et al.  Towards Smart Farming and Sustainable Agriculture with Drones , 2015, 2015 International Conference on Intelligent Environments.

[28]  Subodh Bhandari,et al.  Autonomous Navigation of a Quadrotor in Indoor Environments for Surveillance and Reconnaissance , 2015 .

[29]  Dieter Fox,et al.  RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments , 2012, Int. J. Robotics Res..

[30]  Angelos Antonopoulos,et al.  Towards Autonomous Modular UAV Missions: The Detection, Geo-Location and Landing Paradigm , 2016, Sensors.

[31]  Michael Burmester,et al.  Teleoperating Assistive Robots: A Novel User Interface Relying on Semi-Autonomy and 3D Environment Mapping , 2017, J. Robotics Mechatronics.

[32]  Christopher D. Wickens,et al.  The Future of Air Traffic Control: Human Operators and Automation , 1998 .

[33]  Kwangsu Cho,et al.  Fly a Drone Safely: Evaluation of an Embodied Egocentric Drone Controller Interface , 2016, Interact. Comput..

[34]  Hooman Hedayati,et al.  Robot Teleoperation with Augmented Reality Virtual Surrogates , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[35]  Lydia E. Kavraki,et al.  Probabilistic Roadmaps for Robot Path Planning , 1998 .

[36]  Thomas B. Sheridan,et al.  Space teleoperation through time delay: review and prognosis , 1993, IEEE Trans. Robotics Autom..

[37]  Keiji Nagatani,et al.  Topological simultaneous localization and mapping (SLAM): toward exact localization without explicit localization , 2001, IEEE Trans. Robotics Autom..

[38]  Norman M. Wereley,et al.  The final frontier? Exploring organ transportation by drone , 2018, American journal of transplantation : official journal of the American Society of Transplantation and the American Society of Transplant Surgeons.

[39]  Yunseong Lee,et al.  Design and implementation of autonomous wireless charging station for rotary-wing UAVs , 2016 .

[40]  Thomas B. Sheridan,et al.  Telerobotics, Automation, and Human Supervisory Control , 2003 .

[41]  James Everett Young,et al.  Monocle: Interactive detail-in-context using two pan-and-tilt cameras to improve teleoperation effectiveness , 2017, 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[42]  Roland Siegwart,et al.  Cognitive maps for mobile robots - an object based approach , 2007, Robotics Auton. Syst..

[43]  Alberto Valero-Gomez,et al.  Impact of Two Adjustable-Autonomy Models on the Scalability of Single-Human/Multiple-Robot Teams for Exploration Missions , 2011, Hum. Factors.

[44]  Jean-Arcady Meyer,et al.  Incremental vision-based topological SLAM , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[45]  F. Michaud,et al.  Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation , 2013, IEEE Transactions on Robotics.

[46]  Thecla Schiphorst,et al.  Gathering and Applying Guidelines for Mobile Robot Design for Urban Search and Rescue Application , 2017, HCI.

[47]  Sebastian Zug,et al.  User study on remotely controlled UAVs with focus on interfaces and data link quality , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[48]  Wolfram Burgard,et al.  OctoMap: an efficient probabilistic 3D mapping framework based on octrees , 2013, Autonomous Robots.

[49]  Vijay Kumar,et al.  Towards a swarm of agile micro quadrotors , 2012, Autonomous Robots.

[50]  Huiyang Li,et al.  Human Performance Consequences of Stages and Levels of Automation , 2014, Hum. Factors.

[51]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[52]  Ludger Schmidt,et al.  Design and Evaluation of a Natural User Interface for Piloting an Unmanned Aerial Vehicle , 2018, i-com.

[53]  Karl Kruusamäe,et al.  TeMoto: Intuitive Multi-Range Telerobotic System with Natural Gestural and Verbal Instruction Interface , 2018, Robotics.

[54]  David Scaradozzi,et al.  Design of a Bio-Inspired Autonomous Underwater Robot , 2018, J. Intell. Robotic Syst..

[55]  W. Z. Wan Hasan,et al.  Efficient Charging Pad for Unmanned Aerial Vehicle Based on Direct Contact , 2018, 2018 IEEE 5th International Conference on Smart Instrumentation, Measurement and Application (ICSIMA).

[56]  Jonathan Kelly,et al.  Hummingbird: An Open-Source Dual-Rotor Tail-Sitter Platform for Research and Education , 2018, ArXiv.

[57]  Jean Scholtz,et al.  Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition , 2004, Hum. Comput. Interact..

[58]  Autonomous Navigation , 2017, Encyclopedia of GIS.

[59]  Mica R. Endsley,et al.  From Here to Autonomy , 2017, Hum. Factors.

[60]  Mohit Shridhar,et al.  XPose: Reinventing User Interaction with Flying Cameras , 2017, Robotics: Science and Systems.

[61]  Richard Hildebrant A framework for autonomy , 2006, SPIE Optics East.

[62]  Lei Wang,et al.  Research Progress of Nuclear Emergency Response Robot , 2018 .

[63]  Yiannis Aloimonos,et al.  GapFlyt: Active Vision Based Minimalist Structure-Less Gap Detection For Quadrotor Flight , 2018, IEEE Robotics and Automation Letters.

[64]  R. Held,et al.  Adaptation to displaced and delayed visual feedback from the hand. , 1966 .

[65]  Jörg Stückler,et al.  Adjustable autonomy for mobile teleoperation of personal service robots , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[66]  Markus Vincze,et al.  Segmentation of unknown objects in indoor environments , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[67]  Luca Save,et al.  Designing Human-Automation Interaction : a new level of Automation Taxonomy , 2013 .

[68]  James S. Albus,et al.  Autonomy levels for unmanned systems (ALFUS) framework: an update , 2005, SPIE Defense + Commercial Sensing.

[69]  Otmar Hilliges,et al.  Airways: Optimization-Based Planning of Quadrotor Trajectories according to High-Level User Goals , 2016, CHI.

[70]  Bilge Mutlu,et al.  Designing planning and control interfaces to support user collaboration with flying robots , 2017, Int. J. Robotics Res..

[71]  E. G. Johnsen Teleoperators and Human Augmentation , 1967 .

[72]  Alexander Domahidi,et al.  Real-time planning for automated multi-view drone cinematography , 2017, ACM Trans. Graph..

[73]  Wenzhe Li,et al.  Room segmentation: Survey, implementation, and analysis , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[74]  Pamela J. Hinds,et al.  Autonomy and Common Ground in Human-Robot Interaction: A Field Study , 2007, IEEE Intelligent Systems.

[75]  António Espingardeiro,et al.  Human Performance in Telerobotics Operations , 2011 .

[76]  François Michaud,et al.  Online global loop closure detection for large-scale multi-session graph-based SLAM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[77]  Aida Mustapha,et al.  Adjustable autonomy: a systematic literature review , 2017, Artificial Intelligence Review.

[78]  Elena R. Messina,et al.  A Framework For Autonomy Levels For Unmanned Systems (ALFUS) , 2005 .

[79]  Giuseppe Loianno,et al.  Human Gaze-Driven Spatial Tasking of an Autonomous MAV , 2019, IEEE Robotics and Automation Letters.

[80]  Roland Siegwart,et al.  Monocular‐SLAM–based navigation for autonomous micro helicopters in GPS‐denied environments , 2011, J. Field Robotics.

[81]  Davide Scaramuzza,et al.  Aggressive quadrotor flight through narrow gaps with onboard sensing and computing using active vision , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[82]  S. Eckstein Ethical principles for medical research involving human subjects. , 2001, European journal of emergency medicine : official journal of the European Society for Emergency Medicine.

[83]  Wojciech Giernacki,et al.  Crazyflie 2.0 quadrotor as a platform for research and education in robotics and control engineering , 2017, 2017 22nd International Conference on Methods and Models in Automation and Robotics (MMAR).

[84]  Alonso H. Vera,et al.  No AI Is an Island: The Case for Teaming Intelligence , 2019, AI Mag..

[85]  M R Endsley,et al.  Level of automation effects on performance, situation awareness and workload in a dynamic control task. , 1999, Ergonomics.

[86]  S Stefan Lichiardopol,et al.  A Survey on Teleoperation , 2007 .

[87]  Liping Yang,et al.  Generation of navigation graphs for indoor space , 2015, Int. J. Geogr. Inf. Sci..

[88]  Christos Papachristos,et al.  Uncertainty-aware receding horizon exploration and mapping using aerial robots , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[89]  Axel Wendt,et al.  Automatic Room Segmentation From Unstructured 3-D Data of Indoor Environments , 2017, IEEE Robotics and Automation Letters.

[90]  Kiyoshi Kiyokawa,et al.  A Comparison of Adaptive View Techniques for Exploratory 3D Drone Teleoperation , 2019, ACM Trans. Interact. Intell. Syst..

[91]  Stefanie Tellex,et al.  Flight, Camera, Action! Using Natural Language and Mixed Reality to Control a Drone , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[92]  M. Goodrich,et al.  Supporting Remote Manipulation with an Ecological Augmented Virtuality Interface , 2009 .

[93]  Khelifa Baizid,et al.  What Maps and What Displays for Remote Situation Awareness and ROV Localization? , 2011, HCI.

[94]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[95]  El-Ghazali Talbi,et al.  Real-time motion planning , 1991 .

[96]  Ronald L. Boring,et al.  Shared understanding for collaborative control , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[97]  Vijay Kumar,et al.  Estimation, Control, and Planning for Aggressive Flight With a Small Quadrotor With a Single Camera and IMU , 2017, IEEE Robotics and Automation Letters.

[98]  Zsolt Toth,et al.  Ontology based navigation model of the ILONA system , 2017, 2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI).

[99]  Kazuya Yoshida,et al.  Emergency response to the nuclear accident at the Fukushima Daiichi Nuclear Power Plants using mobile rescue robots , 2013, J. Field Robotics.

[100]  Yu Wang,et al.  Human-Robot Interaction Based on Gaze Gestures for the Drone Teleoperation , 2014 .

[101]  Tiffany L. Chen,et al.  Robots for Humanity : A Case Study in Assistive Mobile Manipulation , 2012 .

[102]  Jesse Paterson,et al.  Improving Usability, Efficiency, and Safety of UAV Path Planning through a Virtual Reality Interface , 2019, SUI.

[103]  Edwin G. Johnsen,et al.  Teleoperators and Human Augmentation. An AEC-NASA Technology Survey , 1967 .

[104]  A. D. Alexander A survey study of teleoperators, robotics, and remote systems technology. , 1973 .

[105]  Sebastian Zug,et al.  Exploring Gridmap-based Interfaces for the Remote Control of UAVs under Bandwidth Limitations , 2017, HRI.

[106]  Michael A. Goodrich,et al.  Ecological Interfaces for Improving Mobile Robot Teleoperation , 2007, IEEE Transactions on Robotics.

[107]  Roland Siegwart,et al.  Cognitive Maps for Mobile Robots , 2007 .

[108]  Albert S. Huang,et al.  Natural language command of an autonomous micro-air vehicle , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[109]  Terrence Fong,et al.  Vehicle Teleoperation Interfaces , 2001, Auton. Robots.

[110]  Juan D. Tardós,et al.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras , 2016, IEEE Transactions on Robotics.

[111]  Xiangke Wang,et al.  ROS-based UAV control using hand gesture recognition , 2017, 2017 29th Chinese Control And Decision Conference (CCDC).

[112]  Arthur D. Alexander,et al.  Impacts of telemation on modern society , 1972 .

[113]  B. Faverjon,et al.  Probabilistic Roadmaps for Path Planning in High-Dimensional Con(cid:12)guration Spaces , 1996 .

[114]  William R. Ferrell,et al.  Remote manipulation with transmission delay. , 1965 .

[115]  Johan Stahre,et al.  Levels of automation in manufacturing , 2008 .

[116]  Dario Floreano,et al.  The Foldable Drone: A Morphing Quadrotor That Can Squeeze and Fly , 2019, IEEE Robotics and Automation Letters.

[117]  Board on Human-Systems Integration The Future of Air Traffic Control: Human Operators and Automation , 1998 .

[118]  Patrick Baudisch,et al.  Keeping things in context: a comparative evaluation of focus plus context screens, overviews, and zooming , 2002, CHI.

[119]  Amedeo Cesta,et al.  ROBIN, a Telepresence Robot to Support Older Users Monitoring and Social Inclusion: Development and Evaluation. , 2017, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[120]  Alexander Domahidi,et al.  Real-Time Motion Planning for Aerial Videography With Real-Time With Dynamic Obstacle Avoidance and Viewpoint Optimization , 2017, IEEE Robotics and Automation Letters.

[121]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[122]  François Michaud,et al.  Egocentric and exocentric teleoperation interface using real-time, 3D video projection , 2009, 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[123]  Kiyoshi Kiyokawa,et al.  Adaptive View Management for Drone Teleoperation in Complex 3D Structures , 2017, IUI.

[124]  Takeo Igarashi,et al.  A teleoperating interface for ground vehicles using autonomous flying cameras , 2013, 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT).

[125]  Leila Takayama,et al.  Strategies for human-in-the-loop robotic grasping , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[126]  Sabri Tosunoglu,et al.  A REVIEW OF TELEOPERATION SYSTEM CONTROL , 2003 .

[127]  Shayok Mukhopadhyay,et al.  Autonomous robotic exploration based on multiple rapidly-exploring randomized trees , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[128]  Yuzhong Shen,et al.  Traditional Vs Gesture Based UAV Control , 2018, Advances in Intelligent Systems and Computing.

[129]  Nathaniel I. Durlach,et al.  Telepresence, time delay and adaptation , 1991 .

[130]  Jeffrey D. Anderson,et al.  Managing autonomy in robot teams: Observations from four experiments , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).