Editing Object Behaviour in Video Sequences

While there are various commercial‐strength editing tools available today for still images, object‐based manipulation of real‐world video footage is still a challenging problem. In this system paper, we present a framework for interactive video editing. Our focus is on footage from a single, conventional video camera. By relying on spatio‐temporal editing techniques operating on the video cube, we do not need to recover 3D scene geometry. Our framework is capable of removing and inserting objects, object motion editing, non‐rigid object deformations, keyframe interpolation, as well as emulating camera motion. We demonstrate how movie shots with moderate complexity can be persuasively modified during post‐processing.

[1]  Jessica K. Hodgins,et al.  Flow-based video synthesis and editing , 2004, ACM Trans. Graph..

[2]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[3]  Leonard McMillan,et al.  Proscenium: a framework for spatio-temporal video editing , 2003, ACM Multimedia.

[4]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[6]  Frédo Durand,et al.  Motion magnification , 2005, ACM Trans. Graph..

[7]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[8]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[9]  Ning Xu,et al.  Videoshop: A new framework for spatio-temporal video editing in gradient domain , 2005, Graph. Model..

[10]  Jessica K. Hodgins,et al.  Flow-based video synthesis and editing , 2004, SIGGRAPH 2004.

[11]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, ACM Trans. Graph..

[12]  Jian Sun,et al.  Video object cut and paste , 2005, SIGGRAPH 2005.

[13]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[14]  Timothy A. Davis,et al.  Row Modifications of a Sparse Cholesky Factorization , 2005, SIAM J. Matrix Anal. Appl..

[15]  Richard H. Bartels,et al.  Interpolating splines with local tension, continuity, and bias control , 1984, SIGGRAPH.

[16]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  William A. Barrett,et al.  Object-based image editing , 2002, ACM Trans. Graph..

[18]  Maneesh Agrawala,et al.  Soft scissors: an interactive tool for realtime high quality matting , 2007, ACM Trans. Graph..

[19]  Bo Thiesson,et al.  Image and Video Segmentation by Anisotropic Kernel Mean Shift , 2004, ECCV.

[20]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[21]  Maneesh Agrawala,et al.  The cartoon animation filter , 2006, ACM Trans. Graph..

[22]  Dani Lischinski,et al.  Evolving Time Fronts: Spatio-Temporal Video Warping , 2005 .

[23]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[24]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[25]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, ACM Trans. Graph..

[26]  Guillermo Sapiro,et al.  Video inpainting of occluding and occluded objects , 2005, IEEE International Conference on Image Processing 2005.

[27]  A. Criminisi,et al.  Bilayer Segmentation of Live Video , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[28]  Eli Shechtman,et al.  Space-time video completion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[29]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[30]  Harry Shum,et al.  Video object cut and paste , 2005, ACM Trans. Graph..

[31]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[32]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[33]  David Salesin,et al.  Video matting of complex scenes , 2002, SIGGRAPH.

[34]  Keiichi Abe,et al.  Topological structural analysis of digitized binary images by border following , 1985, Comput. Vis. Graph. Image Process..

[35]  Demetri Terzopoulos,et al.  The Computation of Visible-Surface Representations , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Dani Lischinski,et al.  Spatio-temporal video warping: Copyright restrictions prevent ACM from providing the full text for this work. , 2005, International Conference on Computer Graphics and Interactive Techniques.

[37]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, SIGGRAPH 2005.

[38]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.