Efficient video cutout based on adaptive multilevel banded method

Although many efficient image cutout approaches have been proposed, it is still a challenge to cut out foreground objects from a video clip because of the costs of time and memory consumption and the difficulty in maintaining temporal coherence for the cutout results. In this paper, we propose a novel and efficient video cutout approach by extending a multilevel banded image segmentation method to video cutout. In our method, the segmentation results are propagated frame by frame at the lowest resolution level based on optical flow, and are refined in image pyramids level by level through banded graph cuts. We construct an adaptive narrow band to ensure that the segmentation result falls exactly into the narrow band during propagation. Experimental results show that the video cutout approach based on adaptive multilevel bands dramatically reduces time and memory costs, and achieves ideal segmentation results.

[1]  Lizhuang Ma,et al.  Seamless video composition using optimized mean-value cloning , 2010, The Visual Computer.

[2]  Mei Han,et al.  Efficient hierarchical graph-based video segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Hua Huang,et al.  RepSnapping: Efficient Image Cutout for Repeated Scene Elements , 2011, Comput. Graph. Forum.

[4]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[5]  Heung-Yeung Shum,et al.  Paint selection , 2009, SIGGRAPH 2009.

[6]  Hujun Bao,et al.  Robust Bilayer Segmentation and Motion/Depth Estimation with a Handheld Camera , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, SIGGRAPH 2005.

[8]  Luc Vincent,et al.  Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  David Salesin,et al.  Video matting of complex scenes , 2002, SIGGRAPH.

[10]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[11]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[12]  Ralph R. Martin,et al.  Vectorizing Cartoon Animations , 2009, IEEE Transactions on Visualization and Computer Graphics.

[13]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[15]  Guillermo Sapiro,et al.  A Geodesic Framework for Fast Interactive Image and Video Segmentation and Matting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[16]  Harry Shum,et al.  Paint selection , 2009, ACM Trans. Graph..

[17]  A. Criminisi,et al.  Bilayer Segmentation of Live Video , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, SIGGRAPH 2007.

[20]  Jian Sun,et al.  Video object cut and paste , 2005, SIGGRAPH 2005.

[21]  Guillermo Sapiro,et al.  Video SnapCut: robust video object cutout using localized classifiers , 2009, ACM Trans. Graph..

[22]  David A. Forsyth,et al.  Generalizing motion edits with Gaussian processes , 2009, ACM Trans. Graph..

[23]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[24]  Leo Grady,et al.  A multilevel banded graph cuts method for fast image segmentation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[25]  Ralph R. Martin,et al.  Video completion using tracking and fragment merging , 2005, The Visual Computer.

[26]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[27]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..