Tethered Aerial Visual Assistance

In this paper, an autonomous tethered Unmanned Aerial Vehicle (UAV) is developed into a visual assistant in a marsupial co-robots team, collaborating with a tele-operated Unmanned Ground Vehicle (UGV) for robot operations in unstructured or confined environments. These environments pose extreme challenges to the remote tele-operator due to the lack of sufficient situational awareness, mostly caused by the unstructuredness and confinement, stationary and limited field-of-view and lack of depth perception from the robot's onboard cameras. To overcome these problems, a secondary tele-operated robot is used in current practices, who acts as a visual assistant and provides external viewpoints to overcome the perceptual limitations of the primary robot's onboard sensors. However, a second tele-operated robot requires extra manpower and teamwork demand between primary and secondary operators. The manually chosen viewpoints tend to be subjective and sub-optimal. Considering these intricacies, we develop an autonomous tethered aerial visual assistant in place of the secondary tele-operated robot and operator, to reduce human robot ratio from 2:2 to 1:2. Using a fundamental viewpoint quality theory, a formal risk reasoning framework, and a newly developed tethered motion suite, our visual assistant is able to autonomously navigate to good-quality viewpoints in a risk-aware manner through unstructured or confined spaces with a tether. The developed marsupial co-robots team could improve tele-operation efficiency in nuclear operations, bomb squad, disaster robots, and other domains with novel tasks or highly occluded environments, by reducing manpower and teamwork demand, and achieving better visual assistance quality with trustworthy risk-aware motion.

[1]  Bilge Mutlu,et al.  An Autonomous Dynamic Camera Method for Effective Remote Teleoperation , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[2]  Taylor Murphy.  Within Reach: The Contribution of Dynamic Viewpoint to the Perception of Remote Environments , 2017 .

[3]  Paul S. Schenker,et al.  Visual acts for remote viewing during teleoperation , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[4]  Paolo Robuffo Giordano,et al.  A visual-based shared control architecture for remote telemanipulation , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[5]  Herbert G. Tanner,et al.  Switched UAV-UGV Cooperation Scheme for Target Detection , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[6]  Robin R. Murphy,et al.  Two case studies and gaps analysis of flood assessment for emergency management with small unmanned aerial systems , 2016, 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[7]  Davide Nicolis,et al.  Occlusion-Free Visual Servoing for the Shared Autonomy Teleoperation of Dual-Arm Robots , 2018, IEEE Robotics and Automation Letters.

[8]  Paul S. Schenker,et al.  Human-robot interaction for intelligent assisted viewing during teleoperation , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[9]  Scott Niekum,et al.  Viewpoint selection for visual failure detection , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[10]  Raffaello D'Andrea,et al.  Stabilization of a flying vehicle on a taut tether using inertial sensing , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  James J. Gibson,et al.  The Ecological Approach to Visual Perception: Classic Edition , 2014 .

[12]  Antonio Barrientos,et al.  From Video Games Multiple Cameras to Multi-robot Teleoperation in Disaster Scenarios , 2016, 2016 International Conference on Autonomous Robot Systems and Competitions (ICARSC).

[13]  Robin R. Murphy,et al.  Visual pose stabilization of tethered small unmanned aerial system to assist drowning victim recovery , 2017, 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR).

[14]  Kiyoshi Kiyokawa,et al.  A Comparison of Adaptive View Techniques for Exploratory 3D Drone Teleoperation , 2019, ACM Trans. Interact. Intell. Syst..

[15]  Christopher D. Wickens,et al.  Multiple Resources and Mental Workload , 2008, Hum. Factors.

[16]  J. Craighead,et al.  Use of Tethered Small Unmanned Aerial System at Berkman Plaza II Collapse , 2008, 2008 IEEE International Workshop on Safety, Security and Rescue Robotics.

[17]  Masato Ito,et al.  Optimal viewpoint selection for cooperative visual assistance in multi-robot systems , 2015, 2015 IEEE/SICE International Symposium on System Integration (SII).

[18]  Kiyoshi Kiyokawa,et al.  Adaptive View Management for Drone Teleoperation in Complex 3D Structures , 2017, IUI.

[19]  Paul S. Schenker,et al.  The Visual Acts model for automated camera placement during teleoperation , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[20]  Fumitoshi Matsuno,et al.  Cooperative task excecution of a search and rescue mission by a multi-robot team , 2005, Adv. Robotics.

[21]  Howie Choset,et al.  Locomotive reduction for snake robots , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Vijay Kumar,et al.  Visual servoing of a UGV from a UAV using differential flatness , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[23]  Christian Laugier,et al.  Automatic camera placement for robot vision tasks , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[24]  Sanjay Krishnan,et al.  Learning 2D Surgical Camera Motion From Demonstrations , 2018, 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE).

[25]  Raffaello D'Andrea,et al.  High-speed, steady flight with a quadrocopter in a confined environment using a tether , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Paul S. Schenker,et al.  The visual acts model for automated camera placement: further results and analysis , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[27]  Gerard T. McKee,et al.  Toward the implementation of visual acts for automated viewing during teleoperation , 1995, Other Conferences.

[28]  Luiz Chaimowicz,et al.  Deploying Air-Ground Multi-Robot Teams in Urban Environments , 2005 .

[29]  Robin R. Murphy,et al.  Benchmarking Tether-based UAV Motion Primitives , 2019, 2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[30]  Alexander M. Morison,et al.  Affordances as a Means to Assess Human-Sensor-Robot Performance , 2016 .

[31]  Shigeki Sugano,et al.  Gaze pattern analysis in multi-display systems for teleoperated disaster response robots , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[32]  Robin R. Murphy,et al.  Explicit Motion Risk Representation , 2019, 2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[33]  Shigeki Sugano,et al.  Derivation of an Optimum and Allowable Range of Pan and Tilt Angles in External Sideway Views for Grasping and Placing Tasks in Unmanned Construction Based on Human Object Recognition , 2019, 2019 IEEE/SICE International Symposium on System Integration (SII).

[34]  Bilge Mutlu,et al.  Remote Telemanipulation with Adapting Viewpoints in Visually Complex Environments , 2019, Robotics: Science and Systems.

[35]  H.H.T. Liu,et al.  A cooperative UAV/UGV platform for wildfire detection and fighting , 2008, 2008 Asia Simulation Conference - 7th International Conference on System Simulation and Scientific Computing.

[36]  Robin R. Murphy,et al.  Introduction to AI Robotics , 2000 .

[37]  Robin R. Murphy,et al.  Explicit-risk-aware Path Planning with Reward Maximization , 2019, ArXiv.

[38]  Robin R. Murphy,et al.  Indoor UAV Localization Using a Tether , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[39]  Reid G. Simmons,et al.  Distributed visual servoing with a roving eye , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[40]  Shoichi Maeyama,et al.  View Point Decision Algorithm for an Autonomous Robot to Provide Support Images in the Operability of a Teleoperated Robot , 2016 .

[41]  Anthony Tzes,et al.  The Power-over-Tether system for powering small UAVs: Tethering-line tension control synthesis , 2015, 2015 23rd Mediterranean Conference on Control and Automation (MED).

[42]  Shahin Sirouspour,et al.  Automatic Viewpoint Planning in Teleoperation of a Mobile Robot , 2014, J. Intell. Robotic Syst..

[43]  Gábor J. Székely,et al.  Hierarchical Clustering via Joint Between-Within Distances: Extending Ward's Minimum Variance Method , 2005, J. Classif..

[44]  Abel Gawel,et al.  Aerial-Ground collaborative sensing: Third-Person view for teleoperation , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[45]  Xuesu Xiao,et al.  Robot Risk-Awareness by Formal Risk Reasoning and Planning , 2020, IEEE Robotics and Automation Letters.

[46]  Shigeki Sugano,et al.  Experimental Investigation of Optimum and Allowable Range of Side Views for Teleoperated Digging and Release Works by Using Actual Construction Machinery , 2019, 2019 IEEE/SICE International Symposium on System Integration (SII).

[47]  Giorgio Guglieri,et al.  A Minimum Risk Approach for Path Planning of UAVs , 2011, J. Intell. Robotic Syst..

[48]  Robin R. Murphy,et al.  Active and mediated opportunistic cooperation between an unmanned aerial vehicle and an unmanned ground vehicle , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[49]  Benjamin Grocholsky,et al.  UAV-UGV collaboration with a PackBot UGV and Raven SUAV for pursuit and tracking of a dynamic target , 2008, SPIE Defense + Commercial Sensing.

[50]  Robin R. Murphy,et al.  Autonomous Visual Assistance for Robot Operations Using a Tethered UAV , 2019, ArXiv.

[51]  Reid G. Simmons,et al.  First Results in the Coordination of Heterogeneous Robots for Large-Scale Assembly , 2000, ISER.

[52]  Alexander M. Morison,et al.  Can I Reach that? An Affordance Based Metric of Human-Sensor-Robot System Effectiveness , 2016, HCI.

[53]  Robin R. Murphy,et al.  Cooperative use of unmanned sea surface and micro aerial vehicles at Hurricane Wilma , 2008 .

[54]  Taylor Murphy.  Apprehending Remote Affordances: Assessing Human Sensor Systems and Their Ability to Understand a Distant Environment , 2013 .

[55]  Robin R. Murphy,et al.  Motion Planning for a UAV with a Straight or Kinked Tether , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[56]  Robin R. Murphy,et al.  UAV assisted USV visual navigation for marine mass casualty incident response , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[57]  Mattias Andersson,et al.  View Position Impact on QoE in an Immersive Telepresence System for Remote Operation , 2019, 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX).

[58]  Robin R. Murphy,et al.  Disaster Robotics , 2014, Springer Handbook of Robotics, 2nd Ed..

[59]  Shigeki Sugano,et al.  Inducement of visual attention using augmented reality for multi-display systems in advanced tele-operation , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[60]  Geoffrey A. Hollinger,et al.  Risk‐aware Path Planning for Autonomous Underwater Vehicles using Predictive Ocean Models , 2013, J. Field Robotics.

[61]  O. Meister,et al.  Teaming of an UGV with a VTOL-UAV in urban environments , 2008, 2008 IEEE/ION Position, Location and Navigation Symposium.

[62]  Shigeki Sugano,et al.  A pre-offering view system for teleoperators of heavy machines to acquire cognitive maps , 2017, 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR).

[63]  Paul S. Schenker,et al.  Human-robot cooperation for automated viewing during teleoperation , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[64]  Vijay Kumar,et al.  Experiments in multirobot air-ground coordination , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[65]  Seiga Kiribayashi,et al.  Design and Development of Tether-Powered Multirotor Micro Unmanned Aerial Vehicle System for Remote-Controlled Construction Machine , 2017, FSR.

[66]  Robin R. Murphy,et al.  Visual servoing for teleoperation using a tethered UAV , 2017, 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR).

[67]  A. R. Soltani,et al.  A fuzzy based multi-objective path planning of construction sites , 2004 .

[68]  Kosuke Sekiyama,et al.  Visual support system for remote control by adaptive ROI selection of monitoring robot , 2018 .

[69]  Paul S. Schenker,et al.  Visual acts for goal-directed vision , 1994, Other Conferences.

[70]  Shigeki Sugano,et al.  Performance Analysis in Advanced Tele-operation System Based on Introduction of Danger-Avoidance View , 2015, ICIRA.

[71]  Cecil C. Bridges,et al.  Hierarchical Cluster Analysis , 1966 .

[72]  Kousuke Sekiyama,et al.  Multi-robot visual support system by adaptive ROI selection based on gestalt perception , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).