UvA-DARE ( Digital Academic Repository ) Visual Tracking : An Experimental Survey Smeulders

There is a large variety of trackers, which have been proposed in the literature during the last two decades with some mixed success. Object tracking in realistic scenarios is difficult problem, therefore it remains a most active area of research in Computer Vision. A good tracker should perform well in a large number of videos involving illumination changes, occlusion, clutter, camera motion, low contrast, specularities and at least six more aspects. However, the performance of proposed trackers have been evaluated typically on less than ten videos, or on the special purpose datasets. In this paper, we aim to evaluate trackers systematically and experimentally on 315 video fragments covering above aspects. We selected a set of nineteen trackers to include a wide variety of algorithms often cited in literature, supplemented with trackers appearing in 2010 and 2011 for which the code was publicly available. We demonstrate that trackers can be evaluated objectively by survival curves, Kaplan Meier statistics, and Grubs testing. We find that in the evaluation practice the F-score is as effective as the object tracking accuracy (OTA) score. The analysis under a large variety of circumstances provides objective insight into the strengths and weaknesses of trackers. Arnold W. M. Smeulders is with Informatics Institute, University of Amsterdam, The Netherlands and with Centrum Wiskunde & Informatica, Amsterdam, The Netherlands. Dung M. Chu is with Informatics Institute, University of Amsterdam, The Netherlands. Rita Cucchiara and Simone Calderara are with Faculty of Engineering of Modena, University of Modena and Reggio Emilia, Italy. Afshin Dehghan and Mubarak Shah are with School of Electric Engineering and Computer Science, University of Florida, USA. * indicates equal contributions. October 18, 2013 DRAFT Digital Object Indentifier 10.1109/TPAMI.2013.230 0162-8828/13/$31.00 © 2013 IEEE This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.

[1]  Xiaoqin Zhang,et al.  Active Contour-Based Visual Tracking by Integrating Colors, Shapes, and Motions , 2013, IEEE Transactions on Image Processing.

[2]  Toufik Ahmed,et al.  On Energy Efficiency in Collaborative Target Tracking in Wireless Sensor Network: A Review , 2013, IEEE Communications Surveys & Tutorials.

[3]  S. Y. Chen,et al.  Kalman Filter for Robot Vision: A Survey , 2012, IEEE Transactions on Industrial Electronics.

[4]  Rita Cucchiara,et al.  People Orientation Recognition by Mixtures of Wrapped Distributions on Random Trees , 2012, ECCV.

[5]  Afshin Dehghan,et al.  GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs , 2012, ECCV.

[6]  Andrea Cavallaro,et al.  Accepted for Publication in Ieee Transactions on Image Processing Adaptive Appearance Modeling for Video Tracking: Survey and Evaluation , 2022 .

[7]  Jaime S. Cardoso,et al.  Filling the gap in quality assessment of video object tracking , 2012, Image Vis. Comput..

[8]  Afshin Dehghan,et al.  Part-based multiple-person tracking with partial occlusion handling , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Kah Phooi Seng,et al.  Lips Contour Detection and Tracking Using Watershed Region-Based Active Contour Model and Modified $H_{\infty}$ , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Andrea Cavallaro,et al.  Adaptive Online Performance Evaluation of Video Trackers , 2012, IEEE Transactions on Image Processing.

[11]  Brian C. Lovell,et al.  Shadow detection: A survey and comparative evaluation of recent methods , 2012, Pattern Recognit..

[12]  Shai Avidan,et al.  Locally Orderless Tracking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  D. Kleinbaum,et al.  Kaplan-Meier Survival Curves and the Log-Rank Test , 2012 .

[14]  Li Bo,et al.  Search on automatic target tracking based on PTZ system , 2011, 2011 International Conference on Image Analysis and Signal Processing.

[15]  Rita Cucchiara,et al.  3DPeS: 3D people dataset for surveillance and forensics , 2011, J-HGBU '11.

[16]  Ales Leonardis,et al.  An adaptive coupled-layer visual model for robust visual tracking , 2011, 2011 International Conference on Computer Vision.

[17]  Huchuan Lu,et al.  Superpixel tracking , 2011, 2011 International Conference on Computer Vision.

[18]  Horst Bischof,et al.  Hough-based tracking of non-rigid objects , 2011, 2011 International Conference on Computer Vision.

[19]  Junseok Kwon,et al.  Tracking by Sampling Trackers , 2011, 2011 International Conference on Computer Vision.

[20]  Rui Caseiro,et al.  Globally optimal solution to multi-object tracking with merged measurements , 2011, 2011 International Conference on Computer Vision.

[21]  Jordi Gonzàlez,et al.  Accurate Moving Cast Shadow Suppression Based on Local Color Constancy Detection , 2011, IEEE Transactions on Image Processing.

[22]  Pascal Fua,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Multiple Object Tracking Using K-shortest Paths Optimization , 2022 .

[23]  Simone Calderara,et al.  Appearance tracking by transduction in surveillance scenarios , 2011, 2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[24]  Li Bai,et al.  Minimum error bounded efficient ℓ1 tracker with occlusion detection , 2011, CVPR 2011.

[25]  Gabriel J. Brostow,et al.  Learning to find occlusion regions , 2011, CVPR 2011.

[26]  Ian D. Reid,et al.  Stable multi-target tracking in real-time surveillance video , 2011, CVPR 2011.

[27]  Gérard G. Medioni,et al.  Context tracker: Exploring supporters and distracters in unconstrained environments , 2011, CVPR 2011.

[28]  Serdar Korukoglu,et al.  A software for performance evaluation and comparison of people detection and tracking methods in video processing , 2011, Multimedia Tools and Applications.

[29]  Armin B. Cremers,et al.  Adaptive real-time video-tracking for arbitrary objects , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  Arnold W. M. Smeulders,et al.  Color Invariant SURF in Discriminative Object Tracking , 2010, ECCV Workshops.

[31]  Marios Savvides,et al.  Automatic Facial Landmark Tracking in Video Sequences Using Kalman Filter Assisted Active Shape Models , 2010, ECCV Workshops.

[32]  Luc Van Gool,et al.  Improving Data Association by Joint Modeling of Pedestrian Trajectories and Groupings , 2010, ECCV.

[33]  Arnold W. M. Smeulders,et al.  Thirteen Hard Cases in Visual Tracking , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[34]  Armando Sousa,et al.  Survey on team tracking techniques applied to sports , 2010, 2010 International Conference on Autonomous and Intelligent Systems, AIS 2010.

[35]  Xuan Song,et al.  An online approach: Learning-Semantic-Scene-by-Tracking and Tracking-by-Learning-Semantic-Scene , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Ramakant Nevatia,et al.  Multi-target tracking by on-line learned discriminative appearance models , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Jiri Matas,et al.  P-N learning: Bootstrapping binary classifiers by structural constraints , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Xuelong Li,et al.  A Review of Active Appearance Models , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[39]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[40]  Jonathan G. Fiscus,et al.  AVSS Multiple Camera Person Tracking Challenge Evaluation Overview , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[41]  Hanqing Lu,et al.  Real-time visual tracking via Incremental Covariance Tensor Learning , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[42]  Jiri Matas,et al.  Online learning of robust object detectors during unstable tracking , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[43]  Takeo Kanade,et al.  Tracking in unstructured crowded scenes , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[44]  Serge J. Belongie,et al.  Visual tracking with online Multiple Instance Learning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Kyoung Mu Lee,et al.  Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive Basin Hopping Monte Carlo sampling , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Jing Zhang,et al.  Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Kyoung Mu Lee,et al.  Visual tracking via geometric particle filtering on the affine group with optimal importance functions , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Junseok Kwon,et al.  Tracking of Abrupt Motion Using Wang-Landau Monte Carlo Estimation , 2008, ECCV.

[49]  Mubarak Shah,et al.  Floor Fields for Tracking in High Density Crowd Scenes , 2008, ECCV.

[50]  Adam Nilski.  An evaluation metric for multiple camera tracking systems: the i-LIDS 5th scenario , 2008, Security + Defence.

[51]  Ramakant Nevatia,et al.  Global data association for multi-object tracking using network flows , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[53]  Aishy Amer,et al.  Performance evaluation for tracking algorithms using object labels , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[54]  B. Ristic,et al.  Tracking systems , 2008, 2008 IEEE Radar Conference.

[55]  Rainer Stiefelhagen,et al.  Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics , 2008, EURASIP J. Image Video Process..

[56]  Simone Calderara,et al.  Bayesian-Competitive Consistent Labeling for People Surveillance , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Yuan Li,et al.  Tracking in Low Frame Rate Video: A Cascade Particle Filter with Discriminative Observers of Different Lifespans , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Ming-Hsuan Yang,et al.  Incremental Learning for Robust Visual Tracking , 2008, International Journal of Computer Vision.

[59]  Ying Chen,et al.  Markov Random Field Modeled Level Sets Method for Object Tracking with Moving Cameras , 2007, ACCV.

[60]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[61]  François Brémond,et al.  ETISEO, performance evaluation for video surveillance systems , 2007, 2007 IEEE Conference on Advanced Video and Signal Based Surveillance.

[62]  Vincent Lepetit,et al.  Fast Keypoint Recognition in Ten Lines of Code , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[63]  Emilio Maggio,et al.  Particle PHD Filtering for Multi-Target Visual Tracking , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[64]  M. Shah,et al.  Object tracking: A survey , 2006, CSUR.

[65]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[66]  Subhash Challa,et al.  Recent Developments in Vision Based Target Tracking for Autonomous Vehicles Navigation , 2006, 2006 IEEE Intelligent Transportation Systems Conference.

[67]  Arnold W. M. Smeulders,et al.  Robust Tracking Using Foreground-Background Texture Discrimination , 2006, International Journal of Computer Vision.

[68]  David A. Forsyth,et al.  Building models of animals from video , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[69]  Ehud Rivlin,et al.  Robust Fragments-based Tracking using the Integral Histogram , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[70]  Fatih Murat Porikli,et al.  Covariance Tracking using Model Update Based on Lie Algebra , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[71]  Mohan M. Trivedi,et al.  Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation , 2006, IEEE Transactions on Intelligent Transportation Systems.

[72]  Fatih Porikli,et al.  Performance Evaluation of Object Detection and Tracking Systems , 2006 .

[73]  Arnold W. M. Smeulders,et al.  Fast occluded object tracking by a robust appearance filter , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[75]  Shai Avidan,et al.  Support vector tracking , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  Performance measures for video object segmentation and tracking , 2004, IEEE Transactions on Image Processing.

[77]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[78]  Branko Ristic,et al.  Beyond the Kalman Filter: Particle Filters for Tracking Applications , 2004 .

[79]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[80]  Rita Cucchiara,et al.  Detecting Moving Objects, Ghosts, and Shadows in Video Streams , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[81]  Gordon Johnston,et al.  Statistical Models and Methods for Lifetime Data , 2003, Technometrics.

[82]  Joost van de Weijer,et al.  Fast Anisotropic Gauss Filtering , 2002, ECCV.

[83]  J. L. Roux.  An Introduction to the Kalman Filter , 2003 .

[84]  Marcel Worring,et al.  Tracking nonparameterized object contours in video , 2002, IEEE Trans. Image Process..

[85]  Stephen Lin,et al.  Diffuse-Specular Separation and Depth Recovery from Image Sequences , 2002, ECCV.

[86]  P. Wayne Power,et al.  Understanding Background Mixture Models for Foreground Segmentation , 2002 .

[87]  Arnold W. M. Smeulders,et al.  Color Invariance , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[88]  Uwe D. Hanebeck,et al.  Template matching using fast normalized cross correlation , 2001, SPIE Defense + Commercial Sensing.

[89]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[90]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[91]  Dorin Comaniciu,et al.  Real-time tracking of non-rigid objects using mean shift , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[92]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[93]  Michael Isard,et al.  A mixed-state condensation tracker with automatic model-switching , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[94]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[95]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..

[96]  D. Collet.  Modelling Survival Data in Medical Research , 2004 .

[97]  Jitendra Malik,et al.  Towards robust automatic traffic scene analysis in real-time , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[98]  Richard Szeliski,et al.  Tracking with Kalman snakes , 1993 .

[99]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[100]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[101]  F. E. Grubbs.  Procedures for Detecting Outlying Observations in Samples , 1969 .

[102]  N. Mantel.  Evaluation of survival data and two new rank order statistics arising in its consideration. , 1966, Cancer chemotherapy reports.

[103]  E. Kaplan,et al.  Nonparametric Estimation from Incomplete Observations , 1958 .