Is my new tracker really better than yours?

The problem of visual tracking evaluation is sporting an abundance of performance measures, which are used by various authors, and largely suffers from lack of consensus about which measures should be preferred. This is hampering the cross-paper tracker comparison and faster advancement of the field. In this paper we provide an overview of the popular measures and performance visualizations and their critical theoretical and experimental analysis. We show that several measures are equivalent from the point of information they provide for tracker comparison and, crucially, that some are more brittle than the others. Based on our analysis we narrow down the set of potential measures to only two complementary ones that can be intuitively interpreted and visualized, thus pushing towards homogenization of the tracker evaluation methodology.

[1]  Lei Zhang,et al.  Real-Time Compressive Tracking , 2012, ECCV.

[2]  Jiri Matas,et al.  P-N learning: Bootstrapping binary classifiers by structural constraints , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Andrea Cavallaro,et al.  A Protocol for Evaluating Video Trackers Under Real-World Conditions , 2013, IEEE Transactions on Image Processing.

[4]  Haibin Ling,et al.  Real time robust L1 tracker using accelerated proximal gradient approach , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Ales Leonardis,et al.  Robust Visual Tracking Using an Adaptive Coupled-Layer Visual Model , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Haibin Ling,et al.  Finding the Best from the Second Bests - Inhibiting Subjective Bias in Evaluation of Visual Tracking Algorithms , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[8]  K. Kao Edward,et al.  An information theoretic approach for tracker performance evaluation , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[9]  Jie Yu,et al.  A Review and Comparison of Measures for Automatic Video Surveillance Systems , 2008, EURASIP J. Image Video Process..

[10]  Yi Wu,et al.  Online Object Tracking: A Benchmark , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Junseok Kwon,et al.  Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive Basin Hopping Monte Carlo sampling , 2009, CVPR.

[12]  Horst Bischof,et al.  Real-Time Tracking via On-line Boosting , 2006, BMVC.

[13]  Matej Kristan,et al.  Closed-world tracking of multiple interacting targets for indoor-sports applications , 2009, Comput. Vis. Image Underst..

[14]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[15]  Ales Leonardis,et al.  A Two-Stage Dynamic Model for Visual Tracking , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Junseok Kwon,et al.  Visual tracking decomposition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Christopher O. Jaynes,et al.  An Open Development Environment for Evaluation of Video Surveillance Systems , 2002 .

[18]  A. Senior,et al.  Performance Evaluation of Surveillance Systems Under Varying Conditions , 2004 .

[19]  Ming-Hsuan Yang,et al.  Robust Object Tracking with Online Multiple Instance Learning , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Ales Leonardis,et al.  An adaptive coupled-layer visual model for robust visual tracking , 2011, 2011 International Conference on Computer Vision.

[21]  Ehud Rivlin,et al.  Robust Fragments-based Tracking using the Integral Histogram , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[22]  Horst Bischof,et al.  Hough-based tracking of non-rigid objects , 2011, 2011 International Conference on Computer Vision.

[23]  James E. Black,et al.  A novel method for video tracking performance evaluation , 2003 .

[24]  Jean-Marc Odobez,et al.  Evaluating Multi-Object Tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[25]  Fatih Porikli,et al.  Performance Evaluation of Object Detection and Tracking Systems , 2006 .

[26]  Ming-Hsuan Yang,et al.  Online visual tracking with histograms and articulating blocks , 2010, Comput. Vis. Image Underst..

[27]  Mathias Kölsch,et al.  Fast 2D Hand Tracking with Flocks of Features and Multi-Cue Integration , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[28]  Jing Zhang,et al.  Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[30]  Patrick Pérez,et al.  Color-Based Probabilistic Tracking , 2002, ECCV.

[31]  Simone Calderara,et al.  Visual Tracking: An Experimental Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ronen Basri,et al.  Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Ming-Hsuan Yang,et al.  An experimental comparison of online object-tracking algorithms , 2011, Optical Engineering + Applications.

[34]  Ming-Hsuan Yang,et al.  Incremental Learning for Robust Visual Tracking , 2008, International Journal of Computer Vision.

[35]  Frank Dellaert,et al.  MCMC-based particle filtering for tracking a variable number of interacting targets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.