LightDenseYOLO: A Fast and Accurate Marker Tracker for Autonomous UAV Landing by Visible Light Camera Sensor on Drone

Autonomous landing of an unmanned aerial vehicle or a drone is a challenging problem for the robotics research community. Previous researchers have attempted to solve this problem by combining multiple sensors such as global positioning system (GPS) receivers, inertial measurement unit, and multiple camera systems. Although these approaches successfully estimate an unmanned aerial vehicle location during landing, many calibration processes are required to achieve good detection accuracy. In addition, cases where drones operate in heterogeneous areas with no GPS signal should be considered. To overcome these problems, we determined how to safely land a drone in a GPS-denied environment using our remote-marker-based tracking algorithm based on a single visible-light-camera sensor. Instead of using hand-crafted features, our algorithm includes a convolutional neural network named lightDenseYOLO to extract trained features from an input image to predict a marker’s location by visible light camera sensor on drone. Experimental results show that our method significantly outperforms state-of-the-art object trackers both using and not using convolutional neural network in terms of both accuracy and processing time.

[1]  Tianjiang Hu,et al.  Ground Stereo Vision-Based Navigation for Autonomous Take-off and Landing of UAVs: A Chan-Vese Model Approach , 2016 .

[2]  C.-C. Jay Kuo,et al.  A deep learning approach to drone monitoring , 2017, 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[3]  Robert Sutton,et al.  Towards autonomous landing on a moving vessel through fiducial markers , 2017, 2017 European Conference on Mobile Robots (ECMR).

[4]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Jürgen Schmidhuber,et al.  A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots , 2016, IEEE Robotics and Automation Letters.

[6]  J. L. Roux.  An Introduction to the Kalman Filter , 2003 .

[7]  Qiaosong Wang,et al.  Object Recognition in Aerial Images Using Convolutional Neural Networks , 2017, J. Imaging.

[8]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[9]  Bernt Schiele,et al.  How good are detection proposals, really? , 2014, BMVC.

[10]  Angelo Cangelosi,et al.  Autonomous Quadrotor Landing using Deep Reinforcement Learning , 2017, ArXiv.

[11]  Daibing Zhang,et al.  Autonomous landing of a helicopter UAV with a ground-based multisensory fusion system , 2015, Other Conferences.

[12]  Biao Wang,et al.  Use of land＇s cooperative object to estimate UAV＇s pose for autonomous landing , 2013 .

[13]  Prateek Burman.  Quadcopter stabilization with neural network , 2016 .

[14]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[15]  Patrick Doherty,et al.  Deep Learning Quadcopter Control via Risk-Aware Active Learning , 2017, AAAI.

[16]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[17]  Ming-Hsuan Yang,et al.  Visual tracking with online Multiple Instance Learning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Sergio Guadarrama,et al.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Bernt Schiele,et al.  What Makes for Effective Detection Proposals? , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Geoffrey C. Fox,et al.  Real-Time, Cloud-Based Object Detection for Unmanned Aerial Vehicles , 2017, 2017 First IEEE International Conference on Robotic Computing (IRC).

[21]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[22]  Abhinav Gupta,et al.  Learning to fly by crashing , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[25]  Reda ElHakim,et al.  Convolutional Neural Network-Based Deep Urban Signatures with Application to Drone Localization , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[26]  Neelima Chavali,et al.  Object-Proposal Evaluation Protocol is ‘Gameable’ , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Anastasios Tefas,et al.  Human crowd detection for drone flight safety using convolutional neural networks , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[28]  Sven Lange,et al.  A vision based onboard approach for landing and position control of an autonomous multirotor UAV in GPS-denied environments , 2009, 2009 International Conference on Advanced Robotics.

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[31]  Edwin Olson,et al.  AprilTag 2: Efficient and robust fiducial detection , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[32]  Rui Caseiro,et al.  High-Speed Tracking with Kernelized Correlation Filters , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Dan Greenwood,et al.  THE APPLICATION OF NEURAL NETWORKS TO DRONE CONTROL , 1990 .

[34]  Р Ю Чуйков,et al.  Обнаружение транспортных средств на изображениях загородных шоссе на основе метода Single shot multibox Detector , 2017 .

[35]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Yanning Zhang,et al.  A Ground-Based Near Infrared Camera Array System for UAV Auto-Landing in GPS-Denied Environment , 2016, Sensors.

[37]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[38]  Nikolai Smolyanskiy,et al.  Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[39]  Flavio Fontana,et al.  Continuous on-board monocular-vision-based elevation mapping applied to autonomous landing of micro aerial vehicles , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[40]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Xiang Zhou,et al.  Airborne Vision-Based Navigation Method for UAV Accuracy Landing Using Infrared Lamps , 2013, J. Intell. Robotic Syst..

[42]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[43]  Seong-Ook Park,et al.  Drone Classification Using Convolutional Neural Networks With Merged Doppler Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[44]  Matthew A. Garratt,et al.  Monocular vision-based real-time target recognition and tracking for autonomously landing an UAV in a cluttered shipboard environment , 2017, Auton. Robots.

[45]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Yong Zhang,et al.  Research on computer vision-based for UAV autonomous landing on a ship , 2009, Pattern Recognit. Lett..

[47]  Davide Scaramuzza,et al.  Vision-based autonomous quadrotor landing on a moving platform , 2017, 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR).

[48]  Tsuhan Chen,et al.  Deep Neural Network for Real-Time Autonomous Indoor Navigation , 2015, ArXiv.

[49]  Kang Ryoung Park,et al.  Remote Marker-Based Tracking for UAV Landing Using Visible-Light Camera Sensor , 2017, Sensors.

[50]  Nabil Aouf,et al.  Vision Based Autonomous Landing of Multirotor UAV on Moving Platform , 2017, J. Intell. Robotic Syst..

[51]  Michael Blumenstein,et al.  A study on detecting drones using deep convolutional neural networks , 2017, 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[52]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[53]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[54]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[55]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.