Background Subtraction: Theory and Practice

Background subtraction is a widely used concept for detection of moving objects in videos. In the last two decades there has been a lot of development in designing algorithms for background subtraction, as well as wide use of these algorithms in various important applications, such as visual surveillance, sports video analysis, motion capture, etc. Various statistical approaches have been proposed to model scene backgrounds. The concept of background subtraction also has been extended to detect objects from videos captured from moving cameras. This book reviews the concept and practice of background subtraction. We discuss several traditional statistical background subtraction models, including the widely used parametric Gaussian mixture models and non-parametric models. We also discuss the issue of shadow suppression, which is essential for human motion analysis applications. This book discusses approaches and tradeoffs for background maintenance. This book also reviews many of the recent developments in background subtraction paradigm. Recent advances in developing algorithms for background subtraction from moving cameras are described, including motion-compensation-based approaches and motion-segmentation-based approaches. For links to the videos to accompany this book, please see sites.google.com/a/morganclaypool.com/backgroundsubtraction/ Table of Contents: Preface / Acknowledgments / Figure Credits / Object Detection and Segmentation in Videos / Background Subtraction from a Stationary Camera / Background Subtraction from a Moving Camera / Bibliography / Author's Biography

[1]  G. Johansson,et al.  Configurations in the perception of velocity , 1950 .

[2]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Serge J. Belongie,et al.  What went where , 2003, CVPR 2003.

[4]  Takeo Kanade,et al.  Background Subtraction for Freely Moving Cameras , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[5]  Daniel P. Huttenlocher,et al.  Scene modeling for wide area surveillance and image synthesis , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[6]  Tomaso A. Poggio,et al.  Pedestrian detection using wavelet templates , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Larry S. Davis,et al.  Efficient non-parametric adaptive color modeling using fast Gauss transform , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[8]  Michael J. Black,et al.  Mixture models for optical flow computation , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Thomas Brox,et al.  Higher order motion models and spectral clustering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Kenichi Kanatani,et al.  Motion segmentation by subspace separation and model selection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[11]  Yongmin Li,et al.  On incremental and robust subspace learning , 2004, Pattern Recognit..

[12]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[13]  Lucia Maddalena,et al.  A Self-Organizing Approach to Background Subtraction for Visual Surveillance Applications , 2008, IEEE Transactions on Image Processing.

[14]  Andrew Zisserman,et al.  Geometric invariance in computer vision , 1992 .

[15]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[16]  Andrew Blake,et al.  A Probabilistic Background Model for Tracking , 2000, ECCV.

[17]  Mubarak Shah,et al.  A hierarchical approach to robust background subtraction using color and gradient information , 2002, Workshop on Motion and Video Computing, 2002. Proceedings..

[18]  Dorin Comaniciu,et al.  Mean shift analysis and applications , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  Don R. Hush,et al.  Change detection for target detection and classification in video sequences , 1988, ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing.

[20]  Michael Harville,et al.  A Framework for High-Level Feedback to Adaptive, Per-Pixel, Mixture-of-Gaussian Background Models , 2002, ECCV.

[21]  Hayit Greenspan,et al.  A Probabilistic Framework for Spatio-Temporal Video Representation & Indexing , 2002, ECCV.

[22]  André Guéziec.  Tracking Pitches for Broadcast Television , 2002, Computer.

[23]  Larry S. Davis,et al.  Background Updating for Visual Surveillance , 2005, ISVC.

[24]  F. Xavier Roca,et al.  Exploiting multiple cues in motion segmentation based on background subtraction , 2013, Neurocomputing.

[25]  W. Eric L. Grimson,et al.  Using adaptive tracking to classify and monitor activities in a site , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[26]  Joachim M. Buhmann,et al.  Topology Free Hidden Markov Models: Application to Background Modeling , 2001, ICCV.

[27]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[28]  Azriel Rosenfeld,et al.  Detection and location of people in video images using adaptive fusion of color and edge information , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[29]  Ahmed M. Elgammal,et al.  A Framework for Feature Selection for Background Subtraction , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[30]  K. P. Karmann,et al.  Moving object recognition using an adaptive background memory , 1990 .

[31]  Larry S. Davis,et al.  W/sup 4/: A Real Time System for Detecting and Tracking People , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[32]  Ernest L. Hall,et al.  Computer Image Processing and Recognition , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Joaquim Salvi,et al.  Motion Segmentation: a Review , 2008, CCIA.

[34]  Erik G. Learned-Miller,et al.  Online domain adaptation of a pre-trained cascade of classifiers , 2011, CVPR 2011.

[35]  Yaser Sheikh,et al.  Bayesian modeling of dynamic scenes for object detection , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  P. Anandan,et al.  A Unified Approach to Moving Object Detection in 2D and 3D Scenes , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[38]  D. Koller,et al.  Towards robust automatic traffic scene analysis in real-time , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[39]  Andrew Zisserman,et al.  Learning Layered Motion Segmentations of Video , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[40]  Hans-Hellmut Nagel,et al.  New likelihood test methods for change detection in image sequences , 1984, Comput. Vis. Graph. Image Process..

[41]  Michal Irani,et al.  Computing occluding and transparent motions , 1994, International Journal of Computer Vision.

[42]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[43]  Larry S. Davis,et al.  Non-parametric Model for Background Subtraction , 2000, ECCV.

[44]  Bohyung Han,et al.  Generalized background subtraction based on hybrid inference by belief propagation and Bayesian filtering , 2011, 2011 International Conference on Computer Vision.

[45]  Rita Cucchiara,et al.  Detecting Moving Objects, Ghosts, and Shadows in Video Streams , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[47]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.

[48]  Chin-Seng Chua,et al.  Statistical background modeling for non-stationary camera , 2003, Pattern Recognit. Lett..

[49]  Wei Zhang,et al.  Moving cast shadows detection based on ratio edge , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[50]  L. Davis,et al.  Background and foreground modeling using nonparametric kernel density estimation for visual surveillance , 2002, Proc. IEEE.

[51]  Gregory D. Hager,et al.  Incremental Focus of Attention for Robust Vision-Based Tracking , 1999, International Journal of Computer Vision.

[52]  Nikos Paragios,et al.  Motion-based background subtraction using adaptive kernel density estimation , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[53]  Anup Basu,et al.  Motion Tracking with an Active Camera , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  R. Vidal,et al.  Motion segmentation with missing data using PowerFactorization and GPCA , 2004, CVPR 2004.

[55]  K. Kanatani.  Camera rotation invariance of image characteristics , 1987 .

[56]  Larry S. Davis,et al.  Background modeling and subtraction by codebook construction , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[57]  Edward H. Adelson,et al.  A unified mixture framework for motion segmentation: incorporating spatial coherence and estimating the number of models , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[58]  Richard Szeliski,et al.  An integrated Bayesian approach to layer extraction from image sequences , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[59]  Jan-Olof Eklundh,et al.  Statistical background subtraction for a mobile observer , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[60]  Massimo Piccardi,et al.  Mean-shift background image modelling , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[61]  Larry S. Davis,et al.  Efficient nonparametric kernel density estimation for real time computer vision , 2002 .

[62]  Kurt Keutzer,et al.  Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow , 2010, ECCV.

[63]  Hassan J. Eghbali,et al.  K-S Test for Detecting Changes from Landsat Imagery Data , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[64]  Naoyuki Ichimura.  Motion segmentation based on factorization method and discriminant criterion , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[65]  Larry S. Davis,et al.  W/sup 4/: Who? When? Where? What? A real time system for detecting and tracking people , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[66]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[67]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[68]  Huijun Di,et al.  Background modeling from a free-moving camera by Multi-Layer Homography Algorithm , 2008, 2008 15th IEEE International Conference on Image Processing.

[69]  Dorin Comaniciu,et al.  Nonparametric robust methods for computer vision , 2000 .

[70]  Ramesh C. Jain,et al.  On the Analysis of Accumulative Difference Pictures from Image Sequences of Real World Scenes , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[71]  Thomas B. Moeslund,et al.  Detection and removal of chromatic moving shadows in surveillance scenarios , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[72]  Andrew Zisserman,et al.  MLESAC: A New Robust Estimator with Application to Estimating Image Geometry , 2000, Comput. Vis. Image Underst..

[73]  Takashi Matsuyama,et al.  Appearance sphere: background model for pan-tilt-zoom camera , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[74]  Martin D. Levine,et al.  Vision in Man and Machine , 1985 .

[75]  Yizong Cheng,et al.  Mean Shift, Mode Seeking, and Clustering , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[76]  Stuart J. Russell,et al.  Image Segmentation in Video Sequences: A Probabilistic Approach , 1997, UAI.

[77]  Dariu Gavrila,et al.  Pedestrian Detection from a Moving Vehicle , 2000, ECCV.

[78]  René Vidal,et al.  Sparse subspace clustering , 2009, CVPR.

[79]  Seth J. Teller,et al.  Particle Video: Long-Range Motion Estimation Using Point Trajectories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[80]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[81]  Azriel Rosenfeld,et al.  Tracking Groups of People , 2000, Comput. Vis. Image Underst..

[82]  Stan Sclaroff,et al.  Segmenting foreground objects from a dynamic textured background via a robust Kalman filter , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[83]  Serge J. Belongie,et al.  What went where [motion segmentation] , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[84]  Jörn Ostermann,et al.  Detection of Moving Cast Shadows for Object Segmentation , 1999, IEEE Trans. Multim..

[85]  Mary A. Peterson,et al.  The initial identification of figure-ground relationships: Contributions from shape recognition processes , 1991 .

[86]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[87]  G. Johansson.  Visual perception of biological motion and a model for its analysis , 1973 .

[88]  Y. Weiss,et al.  Multibody factorization with uncertainty and missing data using the EM algorithm , 2004, CVPR 2004.

[89]  Yen-Wei Chen,et al.  Detection of Moving Objects by Independent Component Analysis , 2006, ACCV.

[90]  Asanobu Kitamoto.  The Moments of the Mixel Distribution and Its Application to Statistical Image Classification , 2000, SSPR/SPR.

[91]  James Orwell,et al.  Adaptive eigen-backgrounds for object detection , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[92]  Andrew Blake,et al.  Statistical mosaics for tracking , 1996, Image Vis. Comput..

[93]  David G. Stork,et al.  Pattern Classification , 1973 .

[94]  Bo Thiesson,et al.  Image and Video Segmentation by Anisotropic Kernel Mean Shift , 2004, ECCV.

[95]  Alex Pentland,et al.  A Bayesian Computer Vision System for Modeling Human Interactions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[96]  Jonathan H. Connell,et al.  A Statistical Approach for Real-time Robust Background Subtrac tion and Shadow Detection , 2014 .

[97]  Naokazu Yokoya,et al.  Real-time tracking of multiple moving objects in moving camera image sequences using robust statistics , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[98]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[99]  Danijel Skocaj,et al.  Weighted and robust incremental method for subspace learning , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[100]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[101]  Brendan J. Frey,et al.  Learning flexible sprites in video layers , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[102]  Wilhelm Burger,et al.  Digital Image Processing - An Algorithmic Introduction using Java , 2016, Texts in Computer Science.

[103]  Du-Ming Tsai,et al.  Independent Component Analysis-Based Background Subtraction for Indoor Surveillance , 2009, IEEE Transactions on Image Processing.

[104]  Xiang Gao,et al.  Error analysis of background adaption , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[105]  Vassilios Morellas,et al.  Robust Foreground Detection In Video Using Pixel Layers , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[106]  Jonathon Shlens,et al.  Fast, Accurate Detection of 100,000 Object Classes on a Single Machine , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[107]  Serhat Selcuk Bucak,et al.  Incremental subspace learning via non-negative matrix factorization , 2009, Pattern Recognit..

[108]  Jitendra Malik,et al.  Object Segmentation by Long Term Analysis of Point Trajectories , 2010, ECCV.

[109]  Marc Pollefeys,et al.  A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-rigid, Degenerate and Non-degenerate , 2006, ECCV.

[110]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[111]  Lu Wang,et al.  Background Subtraction using Incremental Subspace Learning , 2007, 2007 IEEE International Conference on Image Processing.

[112]  Yee-Hong Yang,et al.  The background primal sketch: An approach for tracking moving objects , 1992, Machine Vision and Applications.

[113]  S. Gunnar O. Johansson,et al.  Configurations in event perception : an experimental study , 1951 .

[114]  J. Odobez,et al.  Separation of Moving Regions from Background in an Image Sequence Acquired with a Mobil Camera , 1997 .

[115]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.