Improved Video Segmentation by Adaptive Combination of Depth Keying and Mixture-of-Gaussians

Video segmentation or matting, the separation of foreground objects from background in video sequences, is a demanding task and is needed for a broad range of applications. The most widespread method for video segmentation is chroma-keying using a known background color for which a controlled environment is required. Recently a different method of keying fore-and background has been proposed in which the chroma-keying is replaced by depth-keying using a Time-of-Flight (ToF) camera. The current ToF-cameras suffer from noise and low resolution sensors, which results in unsatisfying segmentation results. We propose to combine the segmentation of dynamic objects in depth with a segmentation in the color domain using adaptive background models. We weight the two measures depending on the actual depth values using either the variance of the depth images of the ToF-camera or the amplitude image of the ToF-camera as reliability measure. We show that both methods significantly improve the segmentation results.

[1]  Michael F. Cohen,et al.  Image and Video Matting: A Survey , 2007, Found. Trends Comput. Graph. Vis..

[2]  Ryan Crabb,et al.  Real-time foreground segmentation via range and color imaging , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[3]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[4]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[5]  Miao Liao,et al.  Joint depth and alpha matte optimization via fusion of stereo and time-of-flight sensor , 2009, CVPR.

[6]  Trevor Darrell,et al.  Background estimation and removal based on range and color , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[7]  Ruigang Yang,et al.  Real-time video matting using multichannel poisson equations , 2010, Graphics Interface.

[8]  Tim J. Ellis,et al.  Illumination-Invariant Motion Detection Using Colour Mixture Models , 2001, BMVC.

[9]  Reinhard Koch,et al.  Integration of a Time-of-Flight Camera into a Mixed Reality System for Handling Dynamic Scenes, Moving Viewpoints and Occlusions in Real-Time , 2008 .