Motion-Aware Gradient Domain Video Composition

For images, gradient domain composition methods like Poisson blending offer practical solutions for uncertain object boundaries and differences in illumination conditions. However, adapting Poisson image blending to video presents new challenges due to the added temporal dimension. In video, the human eye is sensitive to small changes in blending boundaries across frames and slight differences in motions of the source patch and target video. We present a novel video blending approach that tackles these problems by merging the gradient of source and target videos and optimizing a consistent blending boundary based on a user-provided blending trimap for the source video. Our approach extends mean-value coordinates interpolation to support hybrid blending with a dynamic boundary while maintaining interactive performance. We also provide a user interface and source object positioning method that can efficiently deal with complex video sequences beyond the capabilities of alpha blending.

[1]  Alexei A. Efros,et al.  Photo clip art , 2007, ACM Trans. Graph..

[2]  David Salesin,et al.  Video matting of complex scenes , 2002, SIGGRAPH.

[3]  Mei Han,et al.  Efficient hierarchical graph-based video segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Michael S. Brown,et al.  Matting and compositing of transparent and refractive objects , 2011, TOGS.

[5]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[6]  Michael F. Cohen,et al.  Fourier Analysis of the 2D Screened Poisson Equation for Gradient Domain Problems , 2008, ECCV.

[7]  Harry Shum,et al.  Video object cut and paste , 2005, ACM Trans. Graph..

[8]  Zeev Farbman,et al.  Coordinates for instant image cloning , 2009, ACM Trans. Graph..

[9]  Hujun Bao,et al.  Robust Metric Reconstruction from Challenging Video Sequences , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Yun Zhang,et al.  Video Brush: A Novel Interface for Efficient Video Cutout , 2011, Comput. Graph. Forum.

[11]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[12]  Jian Sun,et al.  Drag-and-drop pasting , 2006, SIGGRAPH 2006.

[13]  Horst Bischof,et al.  Motion estimation with non-local total variation regularization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Michael Gleicher,et al.  Subspace video stabilization , 2011, TOGS.

[15]  Hujun Bao,et al.  Video stabilization based on a 3D perspective camera model , 2009, The Visual Computer.

[16]  Sylvain Paris,et al.  Error-Tolerant Image Compositing , 2010, International Journal of Computer Vision.

[17]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[18]  Alexei A. Efros,et al.  Estimating natural illumination from a single outdoor image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[19]  David A. Forsyth,et al.  Generalizing motion edits with Gaussian processes , 2009, ACM Trans. Graph..

[20]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, ACM Trans. Graph..

[21]  Shi-Min Hu,et al.  Painting patches: Reducing flicker in painterly re-rendering of video , 2011, Science China Information Sciences.

[22]  Jiawen Chen,et al.  The video mesh: A data structure for image-based three-dimensional video editing , 2011, 2011 IEEE International Conference on Computational Photography (ICCP).

[23]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[24]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[25]  Guillermo Sapiro,et al.  Video SnapCut: robust video object cutout using localized classifiers , 2009, SIGGRAPH 2009.

[26]  Zhenjiang Miao,et al.  Video matting via opacity propagation , 2011, The Visual Computer.

[27]  William A. Barrett,et al.  Interactive segmentation of image volumes with Live Surface , 2007, Comput. Graph..

[28]  Lizhuang Ma,et al.  Seamless video composition using optimized mean-value cloning , 2010, The Visual Computer.

[29]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[30]  Leif Kobbelt,et al.  Character animation from 2D pictures and 3D motion data , 2007, TOGS.

[31]  Michael F. Cohen,et al.  GradientShop: A gradient-domain optimization framework for image and video filtering , 2010, TOGS.

[32]  Narendra Ahuja,et al.  Seamless video editing , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[33]  David Salesin,et al.  Video watercolorization using bidirectional texture advection , 2007, SIGGRAPH 2007.

[34]  Yun Zhang,et al.  Environment-Sensitive cloning in images , 2011, The Visual Computer.

[35]  Bruno Lévy,et al.  Geometry-aware direction field processing , 2009, TOGS.

[36]  Micah K. Johnson,et al.  Multi-scale image harmonization , 2010, ACM Trans. Graph..