Deep Learning based Multi-Modal Sensing for Tracking and State Extraction of Small Quadcopters

This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal data is processed through a thresholding and Kalman filter approach to detect and track the bounding box. Training and testing data are prepared by combining a set of original experiments conducted in a motion capture environment and publicly available UAV image data. The new pipeline compares favorably to existing methods and demonstrates promising tracking and localization capacity of sample experiments.

[1]  Soon Ki Jung,et al.  Tracking Noisy Targets: A Review of Recent Object Tracking Approaches , 2018, ArXiv.

[2]  Rui Caseiro,et al.  High-Speed Tracking with Kernelized Correlation Filters , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Jason J. Ford,et al.  A vision-based sense-and-avoid system tested on a ScanEagle UAV , 2017, 2017 International Conference on Unmanned Aircraft Systems (ICUAS).

[4]  Souma Chowdhury,et al.  Training Detection-Range-Frugal Cooperative Collision Avoidance Models for Quadcopters via Neuroevolution , 2019, ArXiv.

[5]  Charles A. Bouman,et al.  Deep Learning for Moving Object Detection and Tracking from a Single Camera in Unmanned Aerial Vehicles (UAVs) , 2018, IMAWM.

[6]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  D. Alejo,et al.  Optimal Reciprocal Collision Avoidance with mobile and static obstacles for multi-UAV systems , 2014, 2014 International Conference on Unmanned Aircraft Systems (ICUAS).

[8]  Souma Chowdhury,et al.  Learning reciprocal actions for cooperative collision avoidance in quadrotor unmanned aerial vehicles , 2019, Robotics Auton. Syst..

[9]  Dinesh Manocha,et al.  Reciprocal collision avoidance with acceleration-velocity obstacles , 2011, 2011 IEEE International Conference on Robotics and Automation.

[10]  Bohyung Han,et al.  Learning Multi-domain Convolutional Neural Networks for Visual Tracking , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jiri Matas,et al.  A Novel Performance Evaluation Methodology for Single-Target Trackers , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Jing Li,et al.  Multi-target detection and tracking from a single camera in Unmanned Aerial Vehicles (UAVs) , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[14]  R. Olfati-Saber,et al.  Collision avoidance for multiple agent systems , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[15]  Mykel J. Kochenderfer,et al.  Multi-Rotor Aircraft Collision Avoidance using Partially Observable Markov Decision Processes , 2016 .

[16]  Luca Bertinetto,et al.  Fully-Convolutional Siamese Networks for Object Tracking , 2016, ECCV Workshops.

[17]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[18]  S. Shankar Sastry,et al.  Decentralized nonlinear model predictive control of multiple flying robots , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[19]  Michael Felsberg,et al.  Adaptive Color Attributes for Real-Time Visual Tracking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[21]  Rui Caseiro,et al.  Exploiting the Circulant Structure of Tracking-by-Detection with Kernels , 2012, ECCV.

[22]  Wenhan Luo,et al.  Multiple object tracking: A literature review , 2014, Artif. Intell..

[23]  Denis Gillet,et al.  Reciprocal collision avoidance for quadrotors using on-board visual detection , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Leslie Pack Kaelbling,et al.  Collision Avoidance for Unmanned Aircraft using Markov Decision Processes , 2010 .

[25]  Nicholas G. Polson,et al.  Particle Filtering , 2006 .

[26]  Jiri Matas,et al.  Robust scale-adaptive mean-shift for tracking , 2013, Pattern Recognit. Lett..

[27]  K. A. Joshi,et al.  A Survey on Moving Object Detection and Tracking in Video Surveillance System , 2012 .

[28]  Bruce A. Draper,et al.  Visual object tracking using adaptive correlation filters , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Guanghui Wang,et al.  Vision-Based Real-Time Aerial Object Localization and Tracking for UAV Sensing System , 2017, IEEE Access.

[30]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Aaron Mcfadyen,et al.  Decision strategies for automated visual collision avoidance , 2014, 2014 International Conference on Unmanned Aircraft Systems (ICUAS).

[32]  Scott A. Smolka,et al.  A survey on unmanned aerial vehicle collision avoidance systems , 2015, ArXiv.

[33]  Adrien Perkins,et al.  Tracking of Small Unmanned Aerial Vehicles , 2015 .

[34]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Silvio Savarese,et al.  Learning to Track at 100 FPS with Deep Regression Networks , 2016, ECCV.

[36]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Shyang Chang,et al.  A new criterion for automatic multilevel thresholding , 1995, IEEE Trans. Image Process..

[38]  P. Fua,et al.  Detecting Flying Objects Using a Single Moving Camera , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Ryan J Wallace,et al.  Examining Unmanned Aerial System Threats & Defenses: A Conceptual Analysis , 2015 .

[40]  Seokwon Yeom,et al.  Detection and Tracking of Multiple Moving Vehicles with a UAV , 2018, Int. J. Fuzzy Log. Intell. Syst..

[41]  Amir Behjat,et al.  A Neuroevolution-Based Learning of Reciprocal Maneuver for Collision Avoidance in Quadcopters Under Pose Uncertainties , 2019, DAC 2019.