Inpainting of Wide-Baseline Multiple Viewpoint Video

We describe a non-parametric algorithm for multiple-viewpoint video inpainting. Uniquely, our algorithm addresses the domain of wide baseline multiple-viewpoint video (MVV) with no temporal look-ahead in near real time speed. A Dictionary of Patches (DoP) is built using multi-resolution texture patches reprojected from geometric proxies available in the alternate views. We dynamically update the DoP over time, and a Markov Random Field optimisation over depth and appearance is used to resolve and align a selection of multiple candidates for a given patch, this ensures the inpainting of large regions in a plausible manner conserving both spatial and temporal coherence. We demonstrate the removal of large objects (e.g., people) on challenging indoor and outdoor MVV exhibiting cluttered, dynamic backgrounds and moving cameras.

[1]  Ralph R. Martin,et al.  Comfort-driven disparity adjustment for stereoscopic video , 2016, Computational Visual Media.

[2]  Ruigang Yang,et al.  Stereoscopic inpainting: Joint color and depth completion from stereo images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Marcus A. Magnor,et al.  Sampling based scene-space video processing , 2015, ACM Trans. Graph..

[4]  Pau Gargallo,et al.  Stereoscopic Image Inpainting: Distinct Depth Maps and Images Inpainting , 2010, 2010 20th International Conference on Pattern Recognition.

[5]  Oliver Grau,et al.  How Not to Be Seen — Object Removal from Videos of Crowded Scenes , 2012, Comput. Graph. Forum.

[6]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[8]  Vicent Caselles,et al.  Exemplar-Based Image Inpainting Using Multiscale Graph Cuts , 2013, IEEE Transactions on Image Processing.

[9]  Jean-Yves Guillemaut,et al.  Moving Camera Registration for Multiple Camera Setups in Dynamic Scenes , 2010, BMVC.

[10]  Brian L. Price,et al.  PatchMatch-Based Content Completion of Stereo Image Pairs , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[11]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[12]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[13]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[14]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[15]  Jean-Yves Guillemaut,et al.  Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications , 2011, International Journal of Computer Vision.

[16]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[17]  Shi-Min Hu,et al.  PatchTable: efficient patch queries for large datasets and applications , 2015, ACM Trans. Graph..

[18]  Eyal Ofek,et al.  DCSH - Matching Patches in RGBD Images , 2013, 2013 IEEE International Conference on Computer Vision.

[19]  Erik Reinhard,et al.  Image Statistics in Visual Computing , 2013 .

[20]  Shi-Min Hu,et al.  PlenoPatch: Patch-Based Plenoptic Image Manipulation , 2017, IEEE Transactions on Visualization and Computer Graphics.

[21]  Narendra Ahuja,et al.  Temporally coherent completion of dynamic video , 2016, ACM Trans. Graph..

[22]  Zhe Zhu,et al.  3D modeling and motion parallax for improved videoconferencing , 2016, Computational Visual Media.

[23]  J. Besag.  On the Statistical Analysis of Dirty Pictures , 1986 .

[24]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[25]  Zhe Zhu,et al.  Faithful Completion of Images of Scenic Landmarks Using Internet Images , 2016, IEEE Transactions on Visualization and Computer Graphics.

[26]  Narendra Ahuja,et al.  Image completion using planar structure guidance , 2014, ACM Trans. Graph..

[27]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[28]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, 2006 International Conference on Image Processing.

[29]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[30]  George Drettakis,et al.  Multi-View Inpainting for Image-Based Scene Editing and Rendering , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[31]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[32]  Nenghai Yu,et al.  Coherent Online Video Style Transfer , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[34]  Filippo Speranza,et al.  Stereoscopic imaging: filling disoccluded areas in depth image-based rendering , 2006, SPIE Optics East.

[35]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2008, Commun. ACM.

[36]  Patrick Pérez,et al.  Video Inpainting of Complex Scenes , 2014, SIAM J. Imaging Sci..

[37]  Ralph R. Martin,et al.  PatchNet: a patch-based image representation for interactive library-driven image editing , 2013, ACM Trans. Graph..