Video Background Completion Using Motion-Guided Pixel Assignment Optimization

Background completion for consumer videos captured by free-moving cameras is a challenging problem. In this paper, we present a new approach to complete the holes left by removing objects with motion-guided pixels assignment optimization. We first estimate the motion field in the holes by applying a two-step motion propagation method. Then, using estimated motion field as guidance, the missing parts of the video are completed by performing pixels assignment optimization based on the Markov random field, which optimally assigns available pixels from other neighboring video frames to the missing regions. Finally, we present an illumination-adjusting approach to eliminate the illumination inconsistency in the completed holes. We validate our method on a variety of videos captured by free-moving cameras. Compared with previous methods, our method works better to keep the completed background spatiotemporally coherent, to complete video background with much depth discontinuity and to make the illumination consistent in the completed region.

[1]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Tai-Pang Wu,et al.  Video repairing under variable illumination using cyclic motions , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Harry Shum,et al.  Full-frame video stabilization , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Wolfgang Broll,et al.  High-Quality Real-Time Video Inpaintingwith PixMix , 2014, IEEE Transactions on Visualization and Computer Graphics.

[5]  Ming Liu,et al.  Video completion via motion guided spatial-temporal global optimization , 2009, ACM Multimedia.

[6]  Jan Kautz,et al.  Background Inpainting for Videos with Dynamic Objects and a Free-Moving Camera , 2012, ECCV.

[7]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Oliver Grau,et al.  How Not to Be Seen — Object Removal from Videos of Crowded Scenes , 2012, Comput. Graph. Forum.

[9]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[10]  Maneesh Agrawala,et al.  Using Photographs to Enhance Videos of a Static Scene , 2007, Rendering Techniques.

[11]  Jenq-Neng Hwang,et al.  Exemplar-Based Video Inpainting Without Ghost Shadow Artifacts by Maintaining Temporal Continuity , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Qing Zhang,et al.  Video retargeting combining warping and summarizing optimization , 2013, The Visual Computer.

[13]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[14]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, 2006 International Conference on Image Processing.

[15]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[16]  H. H. Chen,et al.  Video Object Extraction via MRF-Based Contour Tracking , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Harry Shum,et al.  A Joint Motion-Image Inpainting Method for Error Concealment in Video Coding , 2006, 2006 International Conference on Image Processing.

[18]  Patrick Pérez,et al.  Video Inpainting of Complex Scenes , 2014, SIAM J. Imaging Sci..

[19]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[20]  Sunil Arya,et al.  ANN: library for approximate nearest neighbor searching , 1998 .

[21]  Lin Chen,et al.  Efficient Shadow Removal Using Subregion Matching Illumination Transfer , 2013, Comput. Graph. Forum.

[22]  Ralph R. Martin,et al.  Video completion using tracking and fragment merging , 2005, The Visual Computer.

[23]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Tai-Pang Wu,et al.  Video repairing: inference of foreground and background under severe occlusion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[25]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  Guillermo Sapiro,et al.  Video inpainting of occluding and occluded objects , 2005, IEEE International Conference on Image Processing 2005.

[27]  Yongwei Nie,et al.  Fast Exact Nearest Patch Matching for Patch-Based Image Editing and Processing , 2011, IEEE Transactions on Visualization and Computer Graphics.

[28]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Xiaochun Cao,et al.  Video Completion for Perspective Camera Under Constrained Motion , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[30]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, SIGGRAPH 2004.

[31]  Edward H. Adelson,et al.  Human-assisted motion annotation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[33]  Yasuyuki Matsushita,et al.  Video Completion by Motion Field Transfer , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[34]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, ACM Trans. Graph..

[35]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Ariel Shamir,et al.  Data-Driven Video Completion , 2014, Eurographics.

[37]  Timothy K. Shih,et al.  Video Inpainting on Digitized Vintage Films via Maintaining Spatiotemporal Continuity , 2011, IEEE Transactions on Multimedia.

[38]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..