Deep learning-based object detection in low-altitude UAV datasets: A survey

Abstract Deep learning-based object detection solutions emerged from computer vision has captivated full attention in recent years. The growing UAV market trends and interest in potential applications such as surveillance, visual navigation, object detection, and sensors-based obstacle avoidance planning have been holding good promises in the area of deep learning. Object detection algorithms implemented in deep learning framework have rapidly became a method for processing of moving images captured from drones. The primary objective of the paper is to provide a comprehensive review of the state of the art deep learning based object detection algorithms and analyze recent contributions of these algorithms to low altitude UAV datasets. The core focus of the studies is low-altitude UAV datasets because relatively less contribution was seen in the literature when compared with standard or remote-sensing based datasets. The paper discusses the following algorithms: Faster RCNN, Cascade RCNN, R-FCN etc. into two-stage, YOLO and its variants, SSD, RetinaNet into one-stage and CornerNet, Objects as Point etc. under advanced stages in deep learning based detectors. Further, one-two and advanced stages of detectors are studied in detail focusing on low-altitude UAV datasets. The paper provides a broad summary of low altitude datasets along with their respective literature in detection algorithms for the potential use of researchers. Various research gaps and challenges for object detection and classification in UAV datasets that need to deal with for improving the performance are also listed.

[1]  Xinkai Wu,et al.  A Hybrid Vehicle Detection Method Based on Viola-Jones and HOG + SVM from UAV Images , 2016, Sensors.

[2]  Zhipeng Zhou,et al.  A Multidimensional Framework for Unmanned Aerial System Applications in Construction Project Management , 2018 .

[3]  Toby P. Breckon,et al.  Real-time people and vehicle detection from UAV imagery , 2011, Electronic Imaging.

[4]  Sheng Wang Vehicle Detection on Aerial Images by Extracting Corner Features for Rotational Invariant Shape Matching , 2011, 2011 IEEE 11th International Conference on Computer and Information Technology.

[5]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[6]  Gaurav Tiwari,et al.  UAV systems for parameter identification in agriculture , 2013, 2013 IEEE Global Humanitarian Technology Conference: South Asia Satellite (GHTC-SAS).

[7]  Wei Liu,et al.  DSSD : Deconvolutional Single Shot Detector , 2017, ArXiv.

[8]  Bernt Schiele,et al.  What Makes for Effective Detection Proposals? , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Farid Melgani,et al.  Automatic Car Counting Method for Unmanned Aerial Vehicle Images , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[11]  Baijian Yang,et al.  PENet: Object Detection using Points Estimation in Aerial Images , 2020, ArXiv.

[12]  Ashish Kapoor,et al.  AirSim-W: A Simulation Environment for Wildlife Conservation with UAVs , 2018, COMPASS.

[13]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Qiong Liu,et al.  Scale adaptive image cropping for UAV object detection , 2019, Neurocomputing.

[15]  Hyung Jeong Yang,et al.  Multiple human tracking in drone image , 2018, Multimedia Tools and Applications.

[16]  Frédéric Jurie,et al.  Vehicle detection in aerial imagery : A small target detection benchmark , 2016, J. Vis. Commun. Image Represent..

[17]  Takayuki Hamamoto,et al.  Enhanced Cascading Classifier Using Multi-Scale HOG for Pedestrian Detection from Aerial Images , 2016, Int. J. Pattern Recognit. Artif. Intell..

[18]  Roland Siegwart,et al.  Detection and characterization of moving objects with aerial vehicles using inertial-optical flow , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Michael C. Nechyba,et al.  A vision system for intelligent mission profiles of micro air vehicles , 2004, IEEE Transactions on Vehicular Technology.

[20]  Xin Yang,et al.  REDBEE: A visual-inertial drone system for real-time moving object detection , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[21]  Junwei Han,et al.  A Survey on Object Detection in Optical Remote Sensing Images , 2016, ArXiv.

[22]  Xindong Wu,et al.  Object Detection With Deep Learning: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Kuan-Ta Chen,et al.  Face Recognition on Drones: Issues and Limitations , 2015, DroNet@MobiSys.

[25]  Samuel Murray,et al.  Okutama-Action: An Aerial View Video Dataset for Concurrent Human Action Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Jihun Park,et al.  A comparison of convolutional object detectors for real-time drone tracking using a PTZ camera , 2017, 2017 17th International Conference on Control, Automation and Systems (ICCAS).

[27]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[28]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[29]  Mubarak Shah,et al.  Person Tracking in UAV Video , 2007, CLEAR.

[30]  Ramakant Nevatia,et al.  Car detection in low resolution aerial images , 2003, Image Vis. Comput..

[31]  Christos-Savvas Bouganis,et al.  DroNet: Efficient convolutional neural network detector for real-time UAV applications , 2018, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[32]  Winston H. Hsu,et al.  Drone-Based Object Counting by Spatially Regularized Regional Proposal Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  I. Colomina,et al.  Unmanned aerial systems for photogrammetry and remote sensing: A review , 2014 .

[35]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Bernt Schiele,et al.  Vision based victim detection from unmanned aerial vehicles , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[37]  Siddharth Swarup Rautaray,et al.  Application of Deep Learning for Object Detection , 2018 .

[38]  Michael Teutsch,et al.  Evaluation of object segmentation to improve moving vehicle detection in aerial videos , 2014, 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[39]  Hao Jiang,et al.  Identification of autonomous landing sign for unmanned aerial vehicle based on faster regions with convolutional neural network , 2017, 2017 Chinese Automation Congress (CAC).

[40]  Larry S. Davis,et al.  AVSS 2011 demo session: A large-scale benchmark dataset for event recognition in surveillance video , 2011, AVSS.

[41]  Hua-Tsung Chen,et al.  Deep learning-based human activity analysis for aerial images , 2017, 2017 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[42]  Quoc V. Le,et al.  DropBlock: A regularization method for convolutional networks , 2018, NeurIPS.

[43]  Frédéric Jurie,et al.  On the use of deep neural networks for the detection of small vehicles in ortho-images , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[44]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Arturo de la Escalera,et al.  Survey of computer vision algorithms and applications for unmanned aerial vehicles , 2018, Expert Syst. Appl..

[46]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[47]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[48]  Yi Li,et al.  R-FCN: Object Detection via Region-based Fully Convolutional Networks , 2016, NIPS.

[49]  Xiaogang Wang,et al.  DeepID-Net: Deformable deep convolutional neural networks for object detection , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Lin Lei,et al.  Vehicle Detection in Aerial Images Based on Region Convolutional Neural Networks and Hard Negative Example Mining , 2017, Sensors.

[51]  Defu Lin,et al.  Adaptive Anchor for Fast Object Detection in Aerial Image , 2020, IEEE Geoscience and Remote Sensing Letters.

[52]  George W. Irwin,et al.  A review on improving the autonomy of unmanned surface vehicles through intelligent collision avoidance manoeuvres , 2012, Annu. Rev. Control..

[53]  Ang Su,et al.  Online cascaded boosting with histogram of orient gradient features for car detection from unmanned aerial vehicle images , 2015 .

[54]  Forrest N. Iandola,et al.  DenseNet: Implementing Efficient ConvNet Descriptor Pyramids , 2014, ArXiv.

[55]  Qinghua Hu,et al.  Vision Meets Drones: A Challenge , 2018, ArXiv.

[56]  Baochang Zhang,et al.  Enhanced Bird Detection from Low-Resolution Aerial Image Using Deep Neural Networks , 2018, Neural Processing Letters.

[57]  Cemal Aker,et al.  Using deep networks for drone detection , 2017, 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[58]  Michal Pechoucek,et al.  Autonomous UAV Surveillance in Complex Urban Environments , 2009, 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology.

[59]  Christoforos Kanellakis,et al.  Survey on Computer Vision for UAVs: Current Developments and Trends , 2017, Journal of Intelligent & Robotic Systems.

[60]  Matti Pietikäinen,et al.  Deep Learning for Generic Object Detection: A Survey , 2018, International Journal of Computer Vision.

[61]  Wang Huiqian,et al.  Vehicle type detection based on deep learning in traffic scene , 2018 .

[62]  Chandan Kumar,et al.  Efficient Object Detection Model for Real-Time UAV Applications , 2019, ArXiv.

[63]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[64]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[65]  Wei Guo,et al.  Bottle Detection in the Wild Using Low-Altitude Unmanned Aerial Vehicles , 2018, 2018 21st International Conference on Information Fusion (FUSION).

[66]  Shifeng Zhang,et al.  Single-Shot Refinement Neural Network for Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[67]  Hei Law,et al.  CornerNet: Detecting Objects as Paired Keypoints , 2018, ECCV.

[68]  Anastasios Tefas,et al.  Human crowd detection for drone flight safety using convolutional neural networks , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[69]  Xingyi Zhou,et al.  Objects as Points , 2019, ArXiv.

[70]  Xiao Xiang Zhu,et al.  R3-Net: A Deep Network for Multioriented Vehicle Detection in Aerial Images and Videos , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[71]  C. Lawrence Zitnick,et al.  Edge Boxes: Locating Object Proposals from Edges , 2014, ECCV.

[72]  Jieping Ye,et al.  Object Detection in 20 Years: A Survey , 2019, Proceedings of the IEEE.

[73]  Fan Yang,et al.  Exploit All the Layers: Fast and Accurate CNN Object Detector with Scale Dependent Pooling and Cascaded Rejection Classifiers , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Erik Blasch,et al.  Clustered Object Detection in Aerial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[75]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Dongyu Zhang,et al.  How to Fully Exploit The Abilities of Aerial Image Detectors , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[77]  Xuelong Li,et al.  Linear SVM classification using boosting HOG features for vehicle detection in low-altitude airborne videos , 2011, 2011 18th IEEE International Conference on Image Processing.

[78]  Jiebo Luo,et al.  DOTA: A Large-Scale Dataset for Object Detection in Aerial Images , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[79]  Shilei Sun,et al.  Vehicle detection from high-resolution aerial images using spatial pyramid pooling-based deep convolutional neural networks , 2016, Multimedia Tools and Applications.

[80]  Qi Tian,et al.  CenterNet: Keypoint Triplets for Object Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[81]  Lin Lei,et al.  Fast vehicle detection in UAV images , 2017, 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP).

[82]  ChangJin Seo,et al.  Vehicle Detection using Images taken by Low-Altitude Unmanned Aerial Vehicles (UAVs) , 2016 .

[83]  Shaohui Sun,et al.  Aerial 3D Building Detection and Modeling From Airborne LiDAR Point Clouds , 2013, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[84]  Patrick Doherty,et al.  Generating UAV communication networks for monitoring and surveillance , 2010, 2010 11th International Conference on Control Automation Robotics & Vision.

[85]  Xiongwei Wu,et al.  Recent Advances in Deep Learning for Object Detection , 2019, Neurocomputing.

[86]  Yun Teng,et al.  CornerNet-Lite: Efficient Keypoint based Object Detection , 2019, BMVC.

[87]  Nuno Vasconcelos,et al.  Cascade R-CNN: Delving Into High Quality Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[88]  Joshua M. Kaster,et al.  Convolutional neural networks on small unmanned aerial systems , 2017, 2017 IEEE National Aerospace and Electronics Conference (NAECON).

[89]  Ebroul Izquierdo,et al.  Dense and Small Object Detection in UAV Vision Based on Cascade Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[90]  Ronan Collobert,et al.  Learning to Refine Object Segments , 2016, ECCV.

[91]  Gianpaolo Francesco Trotta,et al.  Computer vision and deep learning techniques for pedestrian detection and tracking: A survey , 2018, Neurocomputing.

[92]  Jie Cao,et al.  Application of deep learning and unmanned aerial vehicle technology in traffic flow monitoring , 2017, 2017 International Conference on Machine Learning and Cybernetics (ICMLC).

[93]  Bowen Xu,et al.  On the feature detection of nonconforming objects with automated drone surveillance , 2017, ICCIP '17.

[94]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[95]  Cristi Iuga,et al.  Fall monitoring and detection for at-risk persons using a UAV , 2018 .

[96]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[97]  Jun-Wei Hsieh,et al.  CSPNet: A New Backbone that can Enhance Learning Capability of CNN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[98]  Seong Joon Oh,et al.  CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[99]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[100]  Yuning Jiang,et al.  FoveaBox: Beyound Anchor-Based Object Detection , 2019, IEEE Transactions on Image Processing.

[101]  Dimitar Filev,et al.  Intelligent Vehicle Systems:Applications and New Trends , 2008, ICINCO 2008.

[102]  Nasser M. Nasrabadi,et al.  Convolutional Neural Networks for Aerial Multi-Label Pedestrian Detection , 2018, 2018 21st International Conference on Information Fusion (FUSION).

[103]  Xiangyu Zhang,et al.  Light-Head R-CNN: In Defense of Two-Stage Object Detector , 2017, ArXiv.

[104]  Silvio Savarese,et al.  Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes , 2016, ECCV.

[105]  Michael Felsberg,et al.  The Visual Object Tracking VOT2015 Challenge Results , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[106]  W. Grossman,et al.  Autonomous Searching and Tracking of a River using an UAV , 2007, 2007 American Control Conference.

[107]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[108]  Geoffrey E. Hinton,et al.  When Does Label Smoothing Help? , 2019, NeurIPS.

[109]  Menglong Yan,et al.  Semantic Segmentation of Aerial Images With Shuffling Convolutional Neural Networks , 2018, IEEE Geoscience and Remote Sensing Letters.

[110]  Frédéric Jurie,et al.  Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks , 2018, ArXiv.

[111]  Lin Yan,et al.  Landslide mapping from aerial photographs using change detection-based Markov random field , 2016 .

[112]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.