Design and Optimization of Evaluation Metrics in Object Detection and Tracking for Low-Altitude Aerial Video

The combination of Unmanned Aerial Vehicle (UAV) technology and computer vision has become popular in a wide range of applications, such as surveillance and reconnaissance, while popular evaluation measures are sometimes not applicable for specific tasks. In order to evaluate visual object detection and tracking algorithms of low-altitude aerial video properly, we first summarize the evaluation basis of computer vision tasks, including ground truth, prediction-to-ground truth assignment strategy and distance measures between prediction and ground truth. Then, we analyze the advantages and disadvantages of visual object detection and tracking performance measures, including average precision (AP), F-measure, and accuracy. Finally, for the low-altitude (nearly 100 m) surveillance mission of small unmanned aerial vehicles, we discuss the threshold optimization method of popular measures and the design strategy of application measures. Our work provides a reference in the aspect of performance measures design for researchers of UAV vision.

[1]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[2]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Kyunghyun Cho,et al.  Augmentation for small object detection , 2019, 9th International Conference on Advances in Computing and Information Technology (ACITY 2019).

[4]  Ming-Hsuan Yang,et al.  Robust Object Tracking with Online Multiple Instance Learning , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Sven Kosub,et al.  A note on the triangle inequality for the Jaccard distance , 2016, Pattern Recognit. Lett..

[6]  Erik Learned-Miller,et al.  FDDB: A benchmark for face detection in unconstrained settings , 2010 .

[7]  Yi Wu,et al.  Online Object Tracking: A Benchmark , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Pietro Perona,et al.  Pedestrian detection: A benchmark , 2009, CVPR.

[9]  Michael Felsberg,et al.  The Visual Object Tracking VOT2013 Challenge Results , 2013, ICCV 2013.

[10]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[11]  Ming-Hsuan Yang,et al.  Object Tracking Benchmark , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Lei Zhang,et al.  Real-Time Compressive Tracking , 2012, ECCV.

[13]  Qi Tian,et al.  The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking , 2018, ECCV.

[14]  Qinghua Hu,et al.  Vision Meets Drones: A Challenge , 2018, ArXiv.

[15]  Bernard Ghanem,et al.  A Benchmark and Simulator for UAV Tracking , 2016, ECCV.

[16]  Zhenyu He,et al.  The Visual Object Tracking VOT2016 Challenge Results , 2016, ECCV Workshops.

[17]  Yiyu Shi,et al.  DAC-SDC Low Power Object Detection Challenge for UAV Applications , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Michael Felsberg,et al.  The Visual Object Tracking VOT2015 Challenge Results , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[19]  Timo Aila,et al.  Pruning Convolutional Neural Networks for Resource Efficient Inference , 2016, ICLR.

[20]  Michael Felsberg,et al.  The Visual Object Tracking VOT2013 Challenge Results , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[21]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Simone Calderara,et al.  Visual Tracking: An Experimental Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Pengyi Zhang,et al.  SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[25]  Deniz Onural,et al.  Real-Time Detection, Tracking and Classification of Multiple Moving Objects in UAV Videos , 2017, 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI).

[26]  Jiri Matas,et al.  A Novel Performance Evaluation Methodology for Single-Target Trackers , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Michael Felsberg,et al.  The Sixth Visual Object Tracking VOT2018 Challenge Results , 2018, ECCV Workshops.

[28]  Horst Bischof,et al.  Hough-based tracking of non-rigid objects , 2013, Comput. Vis. Image Underst..

[29]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[30]  Ales Leonardis,et al.  Visual Object Tracking Performance Measures Revisited , 2015, IEEE Transactions on Image Processing.

[31]  Yiannis Kompatsiaris,et al.  VisDrone-VDT2018: The Vision Meets Drone Video Detection and Tracking Challenge Results , 2018, ECCV Workshops.

[32]  Jieping Ye,et al.  Object Detection in 20 Years: A Survey , 2019, Proceedings of the IEEE.

[33]  Stefan Roth,et al.  MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking , 2015, ArXiv.