High-Quality Real-Time Video Inpaintingwith PixMix

While image inpainting has recently become widely available in image manipulation tools, existing approaches to video inpainting typically do not even achieve interactive frame rates yet as they are highly computationally expensive. Further, they either apply severe restrictions on the movement of the camera or do not provide a high-quality coherent video stream. In this paper we will present our approach to high-quality real-time capable image and video inpainting. Our PixMix approach even allows for the manipulation of live video streams, providing the basis for real Diminished Reality (DR) applications. We will show how our approach generates coherent video streams dealing with quite heterogeneous background environments and non-trivial camera movements, even applying constraints in real-time.

[1]  Yun Zhang,et al.  Video Brush: A Novel Interface for Efficient Video Cutout , 2011, Comput. Graph. Forum.

[2]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[3]  Tai-Pang Wu,et al.  Video repairing: inference of foreground and background under severe occlusion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[4]  H. Saito,et al.  Diminished Reality using Multiple Handheld Cameras , 2007 .

[5]  Wolfgang Broll,et al.  PixMix: A real-time approach to high-quality Diminished Reality , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  Charlie C. L. Wang,et al.  Fast Query for Exemplar-Based Image Completion , 2010, IEEE Transactions on Image Processing.

[7]  Mubarak Shah,et al.  Motion Layer Based Object Removal in Videos , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[8]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[10]  Xiaochun Cao,et al.  Video Completion for Perspective Camera Under Constrained Motion , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[11]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  T. Chan,et al.  Image inpainting by correspondence maps: A deterministic approach , 2003 .

[13]  Ming Liu,et al.  Video completion via motion guided spatial-temporal global optimization , 2009, ACM Multimedia.

[14]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[15]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[17]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[18]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[19]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[20]  Tai-Pang Wu,et al.  Video repairing under variable illumination using cyclic motions , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Guillermo Sapiro,et al.  Video SnapCut: robust video object cutout using localized classifiers , 2009, SIGGRAPH 2009.

[22]  Guillermo Sapiro,et al.  A Comprehensive Framework for Image Inpainting , 2010, IEEE Transactions on Image Processing.

[23]  Nassir Navab,et al.  Multiview paraperspective projection model for diminished reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[24]  Yasuyuki Matsushita,et al.  Video Completion by Motion Field Transfer , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[25]  Wolfgang Broll,et al.  Advanced self-contained object removal for realizing real-time Diminished Reality in unconstrained environments , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[26]  Ryuuki Sakamoto,et al.  Diminished Reality for Landscape Video Sequences with Homographies , 2010, KES.

[27]  Miika Aittala,et al.  Light-weight marker hiding for augmented reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[28]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[30]  Pieter Peers,et al.  SubEdit: a representation for editing measured heterogeneous subsurface scattering , 2009, SIGGRAPH 2009.

[31]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[32]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, 2006 International Conference on Image Processing.

[33]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[35]  Naokazu Yokoya,et al.  AR marker hiding based on image inpainting and reflection of illumination changes , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[36]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[37]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[38]  Hanno Scharr,et al.  Optimal operators in digital image processing , 2000 .

[39]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[40]  Charlie C. L. Wang,et al.  Gradient based image completion by solving the Poisson equation , 2007, Comput. Graph..