Algorithms for Visual Tracking of Visitors Under Variable-Lighting Conditions for a Responsive Audio Art Installation

For a responsive audio art installation in a skylit atrium, we developed a single-camera statistical segmentation and tracking algorithm. The algorithm combines statistical background image estimation, per-pixel Bayesian classification, and an approximate solution to the multi-target tracking problem using a bank of Kalman filters and Gale-Shapley matching. A heuristic confidence model enables selective filtering of tracks based on dynamic data. Experiments suggest that our algorithm improves recall and \(F_{2}\)-score over existing methods in OpenCV 2.1. We also find that feedback between the tracking and the segmentation systems improves recall and \(F_{2}\)-score. The system operated effectively for 5–8 h per day for 4 months. Source code and sample data is open source and available in OpenCV.

[1]  Trevor Hastie,et al.  Additive Logistic Regression : a Statistical , 1998 .

[2]  P. KaewTrakulPong,et al.  An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection , 2002 .

[3]  David L. Olson,et al.  Advanced Data Mining Techniques , 2008 .

[4]  James J. Little,et al.  A Boosted Particle Filter: Multitarget Detection and Tracking , 2004, ECCV.

[5]  Naomi Ehrich Leonard,et al.  In the dance studio: Analysis of human flocking , 2012, 2012 American Control Conference (ACC).

[6]  L. S. Shapley,et al.  College Admissions and the Stability of Marriage , 2013, Am. Math. Mon..

[7]  M. Shah,et al.  Object tracking: A survey , 2006, CSUR.

[8]  Luc Van Gool,et al.  An adaptive color-based particle filter , 2003, Image Vis. Comput..

[9]  Paul McKellips,et al.  Are we there yet? , 2014, Lab Animal.

[10]  Gary R. Bradski,et al.  Learning OpenCV - computer vision with the OpenCV library: software that sees , 2008 .

[11]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[13]  Luc Vincent,et al.  Morphological Area Openings and Closings for Grey-scale Images , 1994 .

[14]  Ignacio Parra,et al.  Combination of Feature Extraction Methods for SVM Pedestrian Detection , 2007, IEEE Transactions on Intelligent Transportation Systems.

[15]  Paul A. Viola,et al.  Detecting Pedestrians Using Patterns of Motion and Appearance , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[16]  Josep Vidal,et al.  Kalman tracking for mobile location in NLOS situations , 2003, 14th IEEE Proceedings on Personal, Indoor and Mobile Radio Communications, 2003. PIMRC 2003..

[17]  Amy LaViers,et al.  Style based robotic motion , 2012, 2012 American Control Conference (ACC).

[18]  S. Beucher,et al.  Morphological segmentation , 1990, J. Vis. Commun. Image Represent..

[19]  David Harwood.  A statistical approach for real time robust background subtraction , 1999 .

[20]  Osama Masoud,et al.  A novel method for tracking and counting pedestrians in real-time using a single camera , 2001, IEEE Trans. Veh. Technol..

[21]  Raffaello D'Andrea,et al.  Feed-forward parameter identification for precise periodic quadrocopter motions , 2012, 2012 American Control Conference (ACC).

[22]  P. Wayne Power,et al.  Understanding Background Mixture Models for Foreground Segmentation , 2002 .

[23]  David Beymer,et al.  A real-time computer vision system for vehicle tracking and traffic surveillance , 1998 .

[24]  Jian Yang,et al.  Robust Principal Component Analysis for Recognition , 2013, IScIDE.

[25]  Qi Tian,et al.  Statistical modeling of complex backgrounds for foreground object detection , 2004, IEEE Transactions on Image Processing.

[26]  MalikJitendra,et al.  Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues , 2004 .

[27]  Magnus Egerstedt,et al.  Motion Programs for Puppet Choreography and Control , 2007, HSCC.

[28]  Samuel S. Blackman,et al.  Multiple-Target Tracking with Radar Applications , 1986 .

[29]  Rodrigo F. Cádiz,et al.  Generating music from flocking dynamics , 2012, 2012 American Control Conference (ACC).

[30]  Larry S. Davis,et al.  Fast multiple object tracking via a hierarchical particle filter , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[31]  I. Haritaoglu,et al.  Background and foreground modeling using nonparametric kernel density estimation for visual surveillance , 2002 .

[32]  Trista Pei-chun Chen,et al.  Computer Vision Workload Analysis: Case Study of Video Surveillance Systems , 2005 .

[33]  Ferdinand van der Heijden,et al.  Efficient adaptive density estimation per image pixel for the task of background subtraction , 2006, Pattern Recognit. Lett..

[34]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Luc Vincent,et al.  Morphological grayscale reconstruction in image analysis: applications and efficient algorithms , 1993, IEEE Trans. Image Process..

[36]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Adrian Kaehler,et al.  Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library , 2016 .

[38]  V. Pisarevsky,et al.  Intel's Computer Vision Library: applications in calibration, stereo segmentation, tracking, gesture, face and object recognition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[39]  Qi Tian,et al.  Foreground object detection from videos containing complex background , 2003, MULTIMEDIA '03.

[40]  Manolis I. A. Lourakis,et al.  Binocular Hand Tracking and Reconstruction Based on 2D Shape Matching , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[41]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[42]  John Baillieul,et al.  The control theory of motion-based communication: Problems in teaching robots to dance , 2012, 2012 American Control Conference (ACC).

[43]  Amy LaViers,et al.  The ballet automaton: A formal model for human motion , 2011, Proceedings of the 2011 American Control Conference.

[44]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[45]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[46]  Larry S. Davis,et al.  A Robust Background Subtraction and Shadow Detection , 1999 .