An object-based comparative methodology for motion detection based on the F-Measure

The majority of visual surveillance algorithms rely on effective and accurate motion detection. However, most evaluation techniques described in literature do not address the complexity and range of the issues which underpin the design of a good evaluation methodology. In this paper, we explore the problems associated with both the optimising the operating point of any motion detection algorithms and the objective performance comparison of competing algorithms. In particular, we develop an object-based approach based on the F-Measure-a single-valued ROC-like measure which enables a straight-forward mechanism for both optimising and comparing motion detection algorithms. Despite the advantages over pixel-based ROC approaches, a number of important issues associated with parameterising the evaluation algorithm need to be addressed. The approach is illustrated by a comparison of three motion detection algorithms including the well-known Stauffer and Grimson algorithm, based on results obtained on two datasets.

[1]  Tom Fawcett,et al.  Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions , 1997, KDD.

[2]  M. Sugrue,et al.  Motion Distillation for Pedestrian Surveillance , 2006 .

[3]  Touradj Ebrahimi,et al.  Objective evaluation of segmentation quality using spatio-temporal context , 2002, Proceedings. International Conference on Image Processing.

[4]  J. Crowley,et al.  CAVIAR Context Aware Vision using Image-based Active Recognition , 2005 .

[5]  Martin Winter,et al.  Performance evaluation metrics for motion detection and tracking , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[6]  Luigi di Stefano,et al.  Analysis of pixel-level algorithms for video surveillance applications , 2001, Proceedings 11th International Conference on Image Analysis and Processing.

[7]  Qinfen Zheng,et al.  Self-Evaluation for Video Tracking Systems , 2004 .

[8]  D. Thirde,et al.  Evaluation of Motion Segmentation Quality for Aircraft Activity Surveillance , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[9]  Fernando Pereira,et al.  Objective evaluation of relative segmentation quality , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[10]  Paulo Villegas,et al.  Perceptually-weighted evaluation criteria for segmentation masks in video sequences , 2004, IEEE Transactions on Image Processing.

[11]  T. List,et al.  Performance evaluating the evaluator , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[12]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Jorge S. Marques,et al.  New Performance Evaluation Metrics for Object Detection Algorithms , 2004 .

[14]  Christopher O. Jaynes,et al.  An Open Development Environment for Evaluation of Video Surveillance Systems , 2002 .

[15]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[16]  A. Murat Tekalp,et al.  Metrics for performance evaluation of video object segmentation and tracking without ground-truth , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[17]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[18]  Xiang Gao,et al.  Error analysis of background adaption , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[19]  Bülent Sankur,et al.  Performance evaluation metrics for object-based video segmentation , 2000, 2000 10th European Signal Processing Conference.

[20]  James E. Black,et al.  A novel method for video tracking performance evaluation , 2003 .

[21]  Paulo Villegas,et al.  Objective evaluation of segmentation masks in video sequences , 2000, 2000 10th European Signal Processing Conference.

[22]  T. List,et al.  Comparison of target detection algorithms using adaptive background models , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[23]  Jin Hyeong Park,et al.  Performance evaluation of object detection algorithms , 2002, Object recognition supported by user interaction for service robots.

[24]  Jorge S. Marques,et al.  Performance evaluation of object detection algorithms for video surveillance , 2006, IEEE Transactions on Multimedia.

[25]  Sanjit K. Mitra,et al.  Towards Perceptually Driven Segmentation Evaluation Metrics , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[26]  Franco Oberti,et al.  ROC curves for performance evaluation of video sequences processing systems for surveillance applications , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).