Moving Object Detection in Spatial Domain using Background Removal Techniques - State-of-Art

Identifying moving objects is a critical task for many computer vision applications; it provides a classification of the pixels into either foreground or background. A common approach used to achieve such classification is background removal. Even though there exist numerous of background removal algorithms in the literature, most of them follow a simple flow diagram, passing through four major steps, which are pre-processing, background modelling, foreground de- tection and data validation. In this paper, we survey many existing schemes in the literature of background removal, sur- veying the common pre-processing algorithms used in different situations, presenting different background models, and the most commonly used ways to update such models and how they can be initialized. We also survey how to measure the performance of any moving object detection algorithm, whether the ground truth data is available or not, presenting per- formance metrics commonly used in both cases.

[1]  Roland Mech,et al.  A noise robust method for 2D shape estimation of moving objects in video sequences considering a moving camera , 1998, Signal Process..

[2]  Derek R. Magee,et al.  Tracking multiple vehicles using foreground, background and motion models , 2004, Image Vis. Comput..

[3]  Andrew Blake,et al.  A Probabilistic Background Model for Tracking , 2000, ECCV.

[4]  Yee-Hong Yang,et al.  Stationary background generation: An alternative to the difference of two images , 1990, Pattern Recognit..

[5]  Svetha Venkatesh,et al.  Edge evaluation using necessary components , 1992, CVGIP Graph. Model. Image Process..

[6]  Andrew W. Fitzgibbon,et al.  Direct Least Square Fitting of Ellipses , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Siamak Khorram,et al.  The effects of image misregistration on the accuracy of remotely sensed change detection , 1998, IEEE Trans. Geosci. Remote. Sens..

[8]  Robert C. Bolles,et al.  Background modeling for segmentation of video-rate stereo sequences , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[9]  Til Aach,et al.  Bayesian algorithms for adaptive change detection in image sequences using Markov random fields , 1995, Signal Process. Image Commun..

[10]  Terrance E. Boult,et al.  Frame-rate omnidirectional surveillance and tracking of camouflaged and occluded targets , 1999, Proceedings Second IEEE Workshop on Visual Surveillance (VS'99) (Cat. No.98-89223).

[11]  Rudolf Mester,et al.  Detection and description of moving objects by stochastic modelling and analysis of complex scenes , 1996, Signal Process. Image Commun..

[12]  Larry S. Davis,et al.  View-based detection and analysis of periodic motion , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[13]  M. S. Ulstad,et al.  An algorithm for estimating small scale differences between two digital images , 1973, Pattern Recognit..

[14]  Jorge S. Marques,et al.  Performance evaluation of object detection algorithms for video surveillance , 2006, IEEE Transactions on Multimedia.

[15]  I. Haritaoglu,et al.  Background and foreground modeling using nonparametric kernel density estimation for visual surveillance , 2002 .

[16]  Max A. Viergever,et al.  A survey of medical image registration , 1998, Medical Image Anal..

[17]  Til Aach,et al.  Statistical model-based change detection in moving video , 1993, Signal Process..

[18]  Daphna Weinshall,et al.  Motion of disturbances: detection and tracking of multi-body non-rigid motion , 1999, Machine Vision and Applications.

[19]  Michael Harville,et al.  A Framework for High-Level Feedback to Adaptive, Per-Pixel, Mixture-of-Gaussian Background Models , 2002, ECCV.

[20]  Claude Montacié,et al.  Mixture splitting technique and temporal control in a HMM-based recognition system , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[21]  I.E. Abdou,et al.  Quantitative design and evaluation of enhancement/thresholding edge detectors , 1979, Proceedings of the IEEE.

[22]  Chin-Hui Lee,et al.  Title On-line adaptive learning of the correlated continuous densityhidden Markov models for speech recognition , 1998 .

[23]  Kyungnam Kim,et al.  Algorithms and evaluation for object detection and tracking in computer vision , 2005 .

[24]  W. Eric L. Grimson,et al.  Using adaptive tracking to classify and monitor activities in a site , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[25]  Chia-Ling Tsai,et al.  The dual-bootstrap iterative closest point algorithm with application to retinal image registration , 2003, IEEE Transactions on Medical Imaging.

[26]  H. V. Trees Detection, Estimation, And Modulation Theory , 2001 .

[27]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[28]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[29]  Sridha Sridharan,et al.  Real-time adaptive background segmentation , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[30]  Songde Ma,et al.  A Novel Probability Model for Background Maintenance and Subtraction , 2002 .

[31]  Fernando Pereira,et al.  STANDALONE OBJECTIVE EVALUATION OF SEGMENTATION QUALITY , 2001 .

[32]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[33]  Joachim M. Buhmann,et al.  Topology Free Hidden Markov Models: Application to Background Modeling , 2001, ICCV.

[34]  Touradj Ebrahimi,et al.  Change detection based on color edges , 2001, ISCAS 2001. The 2001 IEEE International Symposium on Circuits and Systems (Cat. No.01CH37196).

[35]  Teuvo Kohonen,et al.  Improved versions of learning vector quantization , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[36]  Touradj Ebrahimi,et al.  Robust and illumination invariant change detection based on linear dependence for surveillance application , 2000, 2000 10th European Signal Processing Conference.

[37]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[38]  Massimo Filippi,et al.  Image registration and subtraction to detect active T2 lesions in MS: an interobserver study , 2002, Journal of Neurology.

[39]  Bülent Sankur,et al.  Survey over image thresholding techniques and quantitative performance evaluation , 2004, J. Electronic Imaging.

[40]  Jitendra Malik,et al.  Towards robust automatic traffic scene analysis in real-time , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[41]  Jitendra Malik,et al.  Robust Multiple Car Tracking with Occlusion Reasoning , 1994, ECCV.

[42]  H. Vincent Poor,et al.  An Introduction to Signal Detection and Estimation , 1994, Springer Texts in Electrical Engineering.

[43]  Chandrika Kamath,et al.  Robust techniques for background subtraction in urban traffic video , 2004, IS&T/SPIE Electronic Imaging.

[44]  A. Murat Tekalp,et al.  Performance measures for video object segmentation and tracking , 2003, IEEE Transactions on Image Processing.

[45]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  H. Vincent Poor,et al.  An introduction to signal detection and estimation (2nd ed.) , 1994 .

[47]  Nigel J. B. McFarlane,et al.  Segmentation and tracking of piglets in images , 1995, Machine Vision and Applications.

[48]  Larry S. Davis,et al.  Efficient non-parametric adaptive color modeling using fast Gauss transform , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[49]  Stan Sclaroff,et al.  Segmenting foreground objects from a dynamic textured background via a robust Kalman filter , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[50]  Rita Cucchiara,et al.  Detecting Moving Objects, Ghosts, and Shadows in Video Streams , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  Touradj Ebrahimi,et al.  Video object extraction based on adaptive background and statistical change detection , 2000, IS&T/SPIE Electronic Imaging.

[52]  Sergio A. Velastin,et al.  Automatic congestion detection system for underground platforms , 2001, Proceedings of 2001 International Symposium on Intelligent Multimedia, Video and Speech Processing. ISIMP 2001 (IEEE Cat. No.01EX489).

[53]  Janne Heikkilä,et al.  A real-time system for monitoring of cyclists and pedestrians , 2004, Image Vis. Comput..

[54]  Larry S. Davis,et al.  W/sup 4/: Who? When? Where? What? A real time system for detecting and tracking people , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[55]  Kenneth M. Dawson-Howe Active Surveillance Using Dynamic Background Subtraction , 1996 .

[56]  David Suter,et al.  A Novel Robust Statistical Method for Background Initialization and Visual Surveillance , 2006, ACCV.

[57]  Ramesh C. Jain,et al.  On the Analysis of Accumulative Difference Pictures from Image Sequences of Real World Scenes , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  H. Niemann,et al.  Adaptive change detection for real-time surveillance applications , 2000, Proceedings Third IEEE International Workshop on Visual Surveillance.

[59]  D. Koller,et al.  Towards robust automatic traffic scene analysis in real-time , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[60]  Terrance E. Boult,et al.  Into the woods: visual surveillance of noncooperative and camouflaged targets in complex outdoor settings , 2001, Proc. IEEE.

[61]  Larry S. Davis,et al.  W4S: A real-time system detecting and tracking people in 2 1/2D , 1998, ECCV.

[62]  Quming Zhou,et al.  Tracking and Classifying Moving Objects from Video , 2001 .

[63]  Paul L. Rosin Thresholding for change detection , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[64]  Paul C. Smits,et al.  Toward specification-driven change detection , 2000, IEEE Trans. Geosci. Remote. Sens..

[65]  M. Cristani,et al.  Multi-level background initialization using Hidden Markov Models , 2003, IWVS '03.

[66]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[67]  Aaron F. Bobick,et al.  Fast Lighting Independent Background Subtraction , 2004, International Journal of Computer Vision.

[68]  Stuart J. Russell,et al.  Image Segmentation in Video Sequences: A Probabilistic Approach , 1997, UAI.

[69]  Rangachar Kasturi,et al.  Machine vision , 1995 .

[70]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[71]  Chuan-Yu Chang,et al.  Learning Vector Quantization Neural Networks for LED Wafer Defect Inspection , 2007, Second International Conference on Innovative Computing, Informatio and Control (ICICIC 2007).

[72]  Hanzi Wang,et al.  Background initialization with a new robust statistical approach , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[73]  Robert Pless,et al.  Evaluation of local models of dynamic backgrounds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[74]  Nikos Paragios,et al.  Motion-based background subtraction using adaptive kernel density estimation , 2004, CVPR 2004.

[75]  Les Kitchen,et al.  Edge Evaluation Using Local Edge Coherence , 1981, IEEE Transactions on Systems, Man, and Cybernetics.

[76]  Mari Ostendorf,et al.  HMM topology design using maximum likelihood successive state splitting , 1997, Comput. Speech Lang..

[77]  Vassilios Digalakis,et al.  Speaker adaptation using combined transformation and Bayesian methods , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[78]  Hanumant Singh,et al.  Toward large-area mosaicing for underwater scientific applications , 2003 .

[79]  Larry S. Davis,et al.  Real-time foreground-background segmentation using codebook model , 2005, Real Time Imaging.

[80]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[81]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[82]  Edward S. Deutsch,et al.  On the Quantitative Evaluation of Edge Detection Schemes and their Comparison with Human Performance , 1975, IEEE Transactions on Computers.

[83]  Trevor Darrell,et al.  Background estimation and removal based on range and color , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[84]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[85]  Sean Dougherty,et al.  Edge Detector Evaluation Using Empirical ROC Curves , 2001, Comput. Vis. Image Underst..

[86]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[87]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[88]  Jan J. Gerbrands,et al.  Three-dimensional image segmentation using a split, merge and group approach , 1991, Pattern Recognit. Lett..

[89]  Ramesh C. Jain,et al.  Illumination independent change detection for real world image sequences , 1989, Comput. Vis. Graph. Image Process..

[90]  Gregory D. Hager,et al.  Efficient Region Tracking With Parametric Models of Geometry and Illumination , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[91]  Svetha Venkatesh,et al.  Dynamic Threshold Determination by Local and Global Edge Evaluation , 1995, CVGIP Graph. Model. Image Process..

[92]  Matthew Brand,et al.  An Entropic Estimator for Structure Discovery , 1998, NIPS.

[93]  Olaf Munkelt,et al.  Adaptive Background Estimation and Foreground Detection using Kalman-Filtering , 1995 .

[94]  Michael Harville,et al.  Foreground segmentation using adaptive mixture models in color and depth , 2001, Proceedings IEEE Workshop on Detection and Recognition of Events in Video.

[95]  Larry S. Davis,et al.  Non-parametric Model for Background Subtraction , 2000, ECCV.

[96]  Michael J. Black,et al.  Robust principal component analysis for computer vision , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[97]  Peter J. Rousseeuw,et al.  Robust Regression and Outlier Detection , 2005, Wiley Series in Probability and Statistics.

[98]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[99]  Mubarak Shah,et al.  A hierarchical approach to robust background subtraction using color and gradient information , 2002, Workshop on Motion and Video Computing, 2002. Proceedings..

[100]  Badrinath Roysam,et al.  Image change detection algorithms: a systematic survey , 2005, IEEE Transactions on Image Processing.

[101]  Martin D. Levine,et al.  Dynamic Measurement of Computer Generated Image Segmentations , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[102]  Hamid Aghajan,et al.  Video-based freeway-monitoring system using recursive vehicle tracking , 1995, Electronic Imaging.

[103]  Paul L. Rosin,et al.  Evaluation of global image thresholding for change detection , 2003, Pattern Recognit. Lett..

[104]  Teuvo Kohonen,et al.  Learning vector quantization , 1998 .

[105]  Berna Erol,et al.  A Bayesian framework for Gaussian mixture background modeling , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[106]  Franco Oberti,et al.  ROC curves for performance evaluation of video sequences processing systems for surveillance applications , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[107]  Giovanni Ramponi,et al.  Countering illumination variations in a video surveillance environment , 2001, IS&T/SPIE Electronic Imaging.

[108]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[109]  Azriel Rosenfeld,et al.  Detection and location of people in video images using adaptive fusion of color and edge information , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[110]  Kurt Konolige,et al.  Small Vision Systems: Hardware and Implementation , 1998 .

[111]  Touradj Ebrahimi,et al.  Classification of change detection algorithms for object-based applications , 2003 .

[112]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[113]  I. Jolliffe Principal Component Analysis , 2002 .

[114]  William M. Wells,et al.  Statistical Approaches to Feature-Based Object Recognition , 2004, International Journal of Computer Vision.

[115]  James E. Black,et al.  A novel method for video tracking performance evaluation , 2003 .

[116]  Paulo Villegas,et al.  Objective evaluation of segmentation masks in video sequences , 2000, 2000 10th European Signal Processing Conference.

[117]  Han Wang,et al.  Analysis of gray level corner detection , 1999, Pattern Recognit. Lett..

[118]  Fernando Pereira,et al.  Estimation of video object's relevance , 2000, 2000 10th European Signal Processing Conference.

[119]  Xiang Gao,et al.  Error analysis of background adaption , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[120]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[121]  Til Aach,et al.  Bayesian spatio-temporal motion detection under varying illumination , 2000, 2000 10th European Signal Processing Conference.

[122]  Matthew Brand,et al.  Discovery and Segmentation of Activities in Video , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[123]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[124]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[125]  Shyang Chang,et al.  Statistical change detection with moments under time-varying illumination , 1998, IEEE Trans. Image Process..

[126]  Nikos Paragios,et al.  Background modeling and subtraction of dynamic scenes , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[127]  Heinrich Niemann,et al.  Statistical modeling and performance characterization of a real-time dual camera surveillance system , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[128]  David J. Marchette,et al.  Adaptive mixture density estimation , 1993, Pattern Recognit..

[129]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[130]  Fatih Murat Porikli,et al.  A Bayesian Approach to Background Modeling , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[131]  Larry S. Davis,et al.  A Perturbation Method for Evaluating Background Subtraction Algorithms , 2003 .

[132]  Fernando Pereira,et al.  Objective evaluation of relative segmentation quality , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[133]  Steven J. Nowlan,et al.  Soft competitive adaptation: neural network learning algorithms based on fitting statistical mixtures , 1991 .

[134]  W. Eric L. Grimson,et al.  Background Subtraction Using Markov Thresholds , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[135]  Sergio A. Velastin,et al.  From tracking to advanced surveillance , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[136]  Til Aach,et al.  Illumination-invariant change detection , 2000, 4th IEEE Southwest Symposium on Image Analysis and Interpretation.

[137]  Michael Isard,et al.  Active Contours , 2000, Springer London.

[138]  Daniel P. Lopresti,et al.  Why table ground-truthing is hard , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[139]  J. Filipe,et al.  OBJECTIVE EVALUATION OF VIDEO SEGMENTATION QUALITY , 2009 .

[140]  A. Glodjo,et al.  Efficient On-Line Nonparametric Kernel Density Estimation , 1999, Algorithmica.

[141]  Alexandre R. J. François,et al.  Adaptive Color Background Modeling for Real-Time Segmentation of Video Streams* , 1999 .

[142]  Michael J. Black,et al.  Robust Principal Component Analysis for Computer Vision , 2001, ICCV.

[143]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[144]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[145]  Alessandro Neri,et al.  Automatic moving object and background separation , 1998, Signal Process..

[146]  Andrew K. C. Wong,et al.  A new method for gray-level picture thresholding using the entropy of the histogram , 1985, Comput. Vis. Graph. Image Process..

[147]  Robert L. Lillestrand,et al.  Techniques ror Change Detection , 1972, IEEE Transactions on Computers.

[148]  Larry S. Davis,et al.  Efficient Kernel Density Estimation Using the Fast Gauss Transform with Applications to Color Modeling and Tracking , 2003, IEEE Trans. Pattern Anal. Mach. Intell..