A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles

Deep learning is recently showing outstanding results for solving a wide variety of robotic tasks in the areas of perception, planning, localization, and control. Its excellent capabilities for learning representations from the complex data acquired in real environments make it extremely suitable for many kinds of autonomous robotic applications. In parallel, Unmanned Aerial Vehicles (UAVs) are currently being extensively applied for several types of civilian tasks in applications going from security, surveillance, and disaster rescue to parcel delivery or warehouse management. In this paper, a thorough review has been performed on recent reported uses and applications of deep learning for UAVs, including the most relevant developments as well as their performances and limitations. In addition, a detailed explanation of the main deep learning techniques is provided. We conclude with a description of the main challenges for the application of deep learning for UAV-based solutions.

[1]  Ming Liu,et al.  Deep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not , 2016, ArXiv.

[2]  Lei Tian,et al.  Real- time UAV weed scout for selective weed control by adaptive robust control and machine learning algorithm , 2016 .

[3]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jürgen Schmidhuber,et al.  LSTM can Solve Hard Long Time Lag Problems , 1996, NIPS.

[5]  Young-Jun Lee,et al.  Empirical study of drone sound detection in real-life environment with deep neural networks , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[6]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Geoffrey E. Hinton,et al.  Semantic hashing , 2009, Int. J. Approx. Reason..

[8]  Tsuhan Chen,et al.  Deep Neural Network for Real-Time Autonomous Indoor Navigation , 2015, ArXiv.

[9]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[10]  Farid Melgani,et al.  A Convolutional Neural Network Approach for Assisting Avalanche Search and Rescue Operations with UAV Imagery , 2017, Remote. Sens..

[11]  Miguel A. Olivares-Méndez,et al.  Towards an Autonomous Vision-Based Unmanned Aerial System against Wildlife Poachers , 2015, Sensors.

[12]  Kazuhiro Nakadai,et al.  Partially Shared Deep Neural Network in sound source separation and identification using a UAV-embedded microphone array , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Pascual Campoy Cervera,et al.  A Multi-Layered Component-Based Approach for the Development of Aerial Robotic Systems: The Aerostack Framework , 2017, J. Intell. Robotic Syst..

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[16]  Kunihiko Fukushima,et al.  Neocognitron: A hierarchical neural network capable of visual pattern recognition , 1988, Neural Networks.

[17]  Sergey Levine,et al.  Learning deep neural network policies with continuous memory states , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[19]  Carlos Sampedro,et al.  Towards autonomous detection and tracking of electric towers for aerial power line inspection , 2014, 2014 International Conference on Unmanned Aircraft Systems (ICUAS).

[20]  Hyun Myung,et al.  Development of a UAV-type jellyfish monitoring system using deep learning , 2015, 2015 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[21]  Tinne Tuytelaars,et al.  How hard is it to cross the room? - Training (Recurrent) Neural Networks to steer a UAV , 2017, ArXiv.

[22]  Geoffrey C. Fox,et al.  Real-Time, Cloud-Based Object Detection for Unmanned Aerial Vehicles , 2017, 2017 First IEEE International Conference on Robotic Computing (IRC).

[23]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[24]  Shanmuganathan Raman,et al.  Dynamic scene classification using convolutional neural networks , 2015, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[25]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[26]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[27]  Sergey Levine,et al.  Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[28]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[29]  Vassilis Athitsos,et al.  Selective unsupervised feature learning with Convolutional Neural Network (S-CNN) , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[30]  Jin Wei,et al.  Deep learning based doppler radar for micro UAS detection and classification , 2016, MILCOM 2016 - 2016 IEEE Military Communications Conference.

[31]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[32]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[33]  Sebastian Scherer,et al.  3D Convolutional Neural Networks for landing zone detection from LiDAR , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Tanaka Kanji,et al.  Mining visual experience for fast cross-view UAV localization , 2015, 2015 IEEE/SICE International Symposium on System Integration (SII).

[35]  Serge J. Belongie,et al.  Learning deep representations for ground-to-aerial geolocalization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[37]  Gui-Song Xia,et al.  Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery , 2015, Remote. Sens..

[38]  Mar Pujol,et al.  Visual Navigation for UAV with Map References Using ConvNets , 2016 .

[39]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[40]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[41]  Pascual Campoy Cervera,et al.  UBRISTES: UAV-Based Building Rehabilitation with Visible and Thermal Infrared Remote Sensing , 2015, ROBOT.

[42]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[43]  Faruk Kazi,et al.  HMD Vision-based Teleoperating UGV and UAV for Hostile Environment using Deep Learning , 2016, ArXiv.

[44]  Geoffrey E. Hinton,et al.  Using very deep autoencoders for content-based image retrieval , 2011, ESANN.

[45]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[46]  Vijay Kumar,et al.  Counting Apples and Oranges With Deep Learning: A Data-Driven Approach , 2017, IEEE Robotics and Automation Letters.

[47]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Héctor F. Satizábal,et al.  Augmenting a convolutional neural network with local histograms - A case study in crop classification from high-resolution UAV imagery , 2016, ESANN.

[49]  Sergey Levine,et al.  (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image , 2016, Robotics: Science and Systems.

[50]  Weijia Li,et al.  Deep Learning Based Oil Palm Tree Detection and Counting for High-Resolution Remote Sensing Images , 2016, Remote. Sens..

[51]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[53]  Jan Peters,et al.  Reinforcement learning in robotics: A survey , 2013, Int. J. Robotics Res..

[54]  Jürgen Schmidhuber,et al.  A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots , 2016, IEEE Robotics and Automation Letters.

[55]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[56]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[57]  Sergey Levine,et al.  Continuous Deep Q-Learning with Model-based Acceleration , 2016, ICML.

[58]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[60]  Jefersson Alex dos Santos,et al.  Do deep features generalize from everyday objects to remote sensing and aerial scenes domains? , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[61]  Nelson Alves,et al.  Tracking natural trails with swarm‐based visual saliency , 2013, J. Field Robotics.

[62]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Zhe Xu,et al.  Feature Learning Based Approach for Weed Classification Using High Resolution Aerial Images from a Digital Camera Mounted on a UAV , 2014, Remote. Sens..

[64]  Davide Scaramuzza,et al.  Active Autonomous Aerial Exploration for Ground Robot Path Planning , 2017, IEEE Robotics and Automation Letters.

[65]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[66]  Nikolai Vladimirovich Kim,et al.  Situation Control of Unmanned Aerial Vehicles for Road Traffic Monitoring , 2015 .

[67]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[68]  Sergey Levine,et al.  End-to-End Training of Deep Visuomotor Policies , 2015, J. Mach. Learn. Res..

[69]  K. Madhava Krishna,et al.  DeepFly: towards complete autonomous navigation of MAVs with monocular camera , 2016, ICVGIP '16.

[70]  Luca Maria Gambardella,et al.  "On-the-Spot Training" for Terrain Classification in Autonomous Air-Ground Collaborative Teams , 2016, ISER.