GPA-Teleoperation: Gaze Enhanced Perception-aware Safe Assistive Aerial Teleoperation

Gaze is an intuitive and direct way to represent the intentions of an individual. However, when it comes to assistive aerial teleoperation which aims to perform operators’ intention, rare attention has been paid to gaze. Existing methods obtain intention directly from the remote controller (RC) input, which is inaccurate, unstable, and unfriendly to nonprofessional operators. Further, most teleoperation works do not consider environment perception which is vital to guarantee safety. In this paper, we present GPA-Teleoperation, a gaze enhanced perception-aware assistive teleoperation framework, which addresses the above issues systematically. We capture the intention utilizing gaze information, and generate a topological path matching it. Then we refine the path into a safe and feasible trajectory which simultaneously enhances the perception awareness to the environment operators are interested in. Additionally, the proposed method is integrated into a customized quadrotor system. Extensive challenging indoor and outdoor real-world experiments and benchmark comparisons verify that the proposed system is reliable, robust and applicable to even unskilled users. We will release the source code of our system to benefit related researches.

[1]  Cosmin Copot,et al.  Application of Visual Servoing and Eye Tracking Glass in Human Robot Interaction: A case study , 2019, 2019 23rd International Conference on System Theory, Control and Computing (ICSTCC).

[2]  Kok Lay Teo,et al.  A computational algorithm for functional inequality constrained optimization problems , 1990, Autom..

[3]  Drivers use active gaze to monitor waypoints during automated driving , 2021, Scientific reports.

[4]  Xin Zhou,et al.  Geometrically Constrained Trajectory Optimization for Multicopters , 2021, ArXiv.

[5]  Assisted Mobile Robot Teleoperation with Intent-aligned Trajectories via Biased Incremental Action Sampling , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Xin Zhou,et al.  EGO-Planner: An ESDF-Free Gradient-Based Local Planner for Quadrotors , 2020, IEEE Robotics and Automation Letters.

[7]  Gaurav S. Sukhatme,et al.  Planning high-speed safe trajectories in confidence-rich maps , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  Jur P. van den Berg,et al.  Automatic collision avoidance for manually tele-operated unmanned aerial vehicles , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Heinrich H. Bülthoff,et al.  Obstacle detection, tracking and avoidance for a teleoperated UAV , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Fei Gao,et al.  RAPTOR: Robust and Perception-Aware Trajectory Replanning for Quadrotor Fast Flight , 2020, IEEE Transactions on Robotics.

[11]  Sven Behnke,et al.  Multimodal obstacle detection and collision avoidance for micro aerial vehicles , 2013, 2013 European Conference on Mobile Robots.

[12]  Fei Gao,et al.  Visibility-aware Trajectory Optimization with Application to Aerial Tracking , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Chao Xu,et al.  Decentralized Spatial-Temporal Trajectory Planning for Multicopter Swarms , 2021, ArXiv.

[14]  Jonathan P. How,et al.  FASTER: Fast and Safe Trajectory Planner for Flights in Unknown Environments , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[15]  Davide Scaramuzza,et al.  Human-Piloted Drone Racing: Visual Processing and Control , 2021, IEEE Robotics and Automation Letters.

[16]  Peter Van Leeuwen,et al.  Differences in Driver Behaviour between Race and Experienced Drivers: A Driving Simulator Study , 2019, VEHITS.

[17]  Xuning Yang,et al.  An Intention Guided Hierarchical Framework for Trajectory-based Teleoperation of Mobile Robots , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Fei Gao,et al.  Robust and Efficient Quadrotor Trajectory Generation for Fast Autonomous Flight , 2019, IEEE Robotics and Automation Letters.

[19]  D. Mestre,et al.  Optokinetic nystagmus is elicited by curvilinear optic flow during high speed curve driving , 2011, Vision Research.

[20]  Teemu H Itkonen,et al.  Beyond the tangent point: gaze targets in naturalistic driving. , 2013, Journal of vision.

[21]  Yasuharu Koike,et al.  Real-Time Control of a Video Game Using Eye Movements and Two Temporal EEG Sensors , 2015, Comput. Intell. Neurosci..

[22]  Erwin R. Boer.  Tangent point oriented curve negotiation , 1996, Proceedings of Conference on Intelligent Vehicles.

[23]  Charles Richter,et al.  Learning to Plan for Visibility in Navigation of Unknown Environments , 2016, ISER.

[24]  Nicu Sebe,et al.  What Are You Looking at? , 2011, International Journal of Computer Vision.

[25]  David N. Lee,et al.  Where we look when we steer , 1994, Nature.

[26]  Zoya Hubenova,et al.  Usage of eye tracker technology in examining attention distribution of operators of unmanned aerial vehicles , 2021 .