A Review and Comparison of Measures for Automatic Video Surveillance Systems

Today's video surveillance systems are increasingly equipped with video content analysis for a great variety of applications. However, reliability and robustness of video content analysis algorithms remain an issue. They have to be measured against ground truth data in order to quantify the performance and advancements of new algorithms. Therefore, a variety of measures have been proposed in the literature, but there has neither been a systematic overview nor an evaluation of measures for specific video analysis tasks yet. This paper provides a systematic review of measures and compares their effectiveness for specific aspects, such as segmentation, tracking, and event detection. Focus is drawn on details like normalization issues, robustness, and representativeness. A software framework is introduced for continuously evaluating and documenting the performance of video surveillance systems. Based on many years of experience, a new set of representative measures is proposed as a fundamental part of an evaluation framework.

[1]  J. Crowley,et al.  CAVIAR Context Aware Vision using Image-based Active Recognition , 2005 .

[2]  Jorge S. Marques,et al.  Performance evaluation of object detection algorithms for video surveillance , 2006, IEEE Transactions on Multimedia.

[3]  Dmitry B. Goldgof,et al.  Performance Evaluation of Object Detection and Tracking in Video , 2006, ACCV.

[4]  Andrew J. Chosak,et al.  OVVV: Using Virtual Worlds to Design and Evaluate Surveillance Systems , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  i-LIDS Team,et al.  Imagery Library for Intelligent Detection Systems (i-LIDS); A Standard for Testing Video Based Detection Systems , 2006, Proceedings 40th Annual 2006 International Carnahan Conference on Security Technology.

[6]  Jean-Marc Odobez,et al.  Evaluating Multi-Object Tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[7]  N. Lazarevic-McManus,et al.  Performance evaluation in visual surveillance using the F-measure , 2006, VSSN '06.

[8]  Fatih Porikli,et al.  Performance Evaluation of Object Detection and Tracking Systems , 2006 .

[9]  A. Senior,et al.  Performance Evaluation of Surveillance Systems Under Varying Conditions , 2004 .

[10]  A.t NGHIEM,et al.  A New Evaluation Approach for Video Processing Algorithms , 2007, 2007 IEEE Workshop on Motion and Video Computing (WMVC'07).

[11]  Fernando Pereira,et al.  Objective evaluation of relative segmentation quality , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[12]  Tim Ellis.  Performance metrics and methods for tracking in surveillance , 2002 .

[13]  Christopher O. Jaynes,et al.  An Open Development Environment for Evaluation of Video Surveillance Systems , 2002 .

[14]  A. Murat Tekalp,et al.  Performance measures for video object segmentation and tracking , 2003, IEEE Transactions on Image Processing.

[15]  Dimitrios Makris,et al.  Designing evaluation methodologies: The case of motion detection , 2006 .

[16]  François Brémond,et al.  ETISEO, performance evaluation for video surveillance systems , 2007, 2007 IEEE Conference on Advanced Video and Signal Based Surveillance.

[17]  Sharath Pankanti,et al.  Appearance models for occlusion handling , 2006, Image Vis. Comput..

[18]  C. Machy,et al.  Performance Evaluation of Frequent Events Detection Systems , 2006 .

[19]  Robert B. Fisher,et al.  CVML - an XML-based computer vision markup language , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[20]  S. Muller-Schneiders,et al.  Performance evaluation of a real time video surveillance system , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[21]  Andrea Cavallaro,et al.  Performance evaluation of event detection solutions: the CREDS experience , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[22]  Fatih Murat Porikli,et al.  Achieving real-time object detection and tracking under extreme conditions , 2006, Journal of Real-Time Image Processing.

[23]  J.M. Ferryman,et al.  PETS Metrics: On-Line Performance Evaluation Service , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[24]  Roger D. Boyle,et al.  Performance Evaluation Metrics and Statistics for Positional Tracker Evaluation , 2003, ICVS.

[25]  Andrew Zisserman,et al.  Delving into the Whorl of Flower Segmentation , 2007, BMVC.

[26]  Thomas Döring,et al.  Evaluation of object tracking in traffic scenes , 2006 .

[27]  Yan Li,et al.  Evaluating the performance of systems for tracking football players and ball , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[28]  T. List,et al.  Performance evaluating the evaluator , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[29]  Yu Jin Zhang,et al.  A review of recent evaluation methods for image segmentation , 2001, Proceedings of the Sixth International Symposium on Signal Processing and its Applications (Cat.No.01EX467).

[30]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[31]  Sergio A. Velastin,et al.  How close are we to solving the problem of automated visual surveillance? , 2008, Machine Vision and Applications.

[32]  James E. Black,et al.  A novel method for video tracking performance evaluation , 2003 .