Exemplar-Based Image Inpainting Using an Affine Invariant Similarity Measure

Patch-based approaches are used in state-of-the-art methods for image inpainting. This paper presents a new method for exemplar-based image inpainting using transformed patches. The transformation is determined for each patch in a fully automatic way from a surrounding texture content. We build upon a recent affine invariant patch similarity measure that performs an appropriate patch comparison by automatically adapting the size and shape of the patches. As a consequence, it intrinsically extends the set of available source patches to copy information from. We incorporate this measure into a variational formulation for inpainting and present a numerical algorithm for optimizing it. We show that our method can be applied to complete a perspectively distorted texture as well as to automatically inpaint one view of a scene using other view of the same scene as a source. We present experimental results both for gray and color images, and a comparison with some exemplar-based image inpainting methods.

[1]  E. Nadaraya.  On Estimating Regression , 1964 .

[2]  Patrick Pérez,et al.  Geometrically Guided Exemplar-Based Inpainting , 2011, SIAM J. Imaging Sci..

[3]  Vicent Caselles,et al.  Multiscale Analysis of Similarities between Images on Riemannian Manifolds , 2014, Multiscale Model. Simul..

[4]  Gabriel Peyré,et al.  Manifold models for signals and images , 2009, Comput. Vis. Image Underst..

[5]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[6]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[8]  Guillermo Sapiro,et al.  A Variational Framework for Exemplar-Based Image Inpainting , 2011, International Journal of Computer Vision.

[9]  Tony Lindeberg,et al.  Direct computation of shape cues using scale-adapted spatial derivative operators , 1996, International Journal of Computer Vision.

[10]  T. Chan,et al.  Image inpainting by correspondence maps: A deterministic approach , 2003 .

[11]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[12]  Luc Van Gool,et al.  Transforming Image Completion , 2011, BMVC.

[13]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[14]  Jean-Michel Morel,et al.  Level lines based disocclusion , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[15]  Zhaozhong Wang,et al.  Image Affine Inpainting , 2008, ICIAR.

[16]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[17]  Simon Masnou,et al.  Disocclusion: a variational approach using level lines , 2002, IEEE Trans. Image Process..

[18]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[19]  Gabriele Facciolo,et al.  Linear Multiscale Analysis of Similarities between Images on Riemannian Manifolds: Practical Formula and Affine Covariant Metrics , 2015, SIAM J. Imaging Sci..

[20]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[21]  Leif Kobbelt,et al.  Interactive image completion with perspective correction , 2006, The Visual Computer.

[22]  Tony Lindeberg,et al.  Direct Estimation of Local Surface Shape in a Fixating Binocular Vision System , 1994, ECCV.

[23]  Manuel González,et al.  Affine Invariant Texture Segmentation and Shape from Texture by Variational Methods , 1998, Journal of Mathematical Imaging and Vision.

[24]  Narendra Ahuja,et al.  Image completion using planar structure guidance , 2014, ACM Trans. Graph..

[25]  Saïd Ladjal,et al.  Exemplar-Based Inpainting from a Variational Point of View , 2010, SIAM J. Math. Anal..

[26]  Guy Gilboa,et al.  Nonlocal Operators with Applications to Image Processing , 2008, Multiscale Model. Simul..

[27]  Jonas Gårding,et al.  Shape from texture for smooth curved surfaces in perspective projection , 1992, Journal of Mathematical Imaging and Vision.

[28]  Naokazu Yokoya,et al.  Image Inpainting Considering Brightness Change and Spatial Locality of Textures , 2009, VISAPP.

[29]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[30]  Cordelia Schmid,et al.  Scale & Affine Invariant Interest Point Detectors , 2004, International Journal of Computer Vision.

[31]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[32]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[33]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[34]  Narendra Ahuja,et al.  Transformation guided image completion , 2013, IEEE International Conference on Computational Photography (ICCP).

[35]  Jiri Matas,et al.  Robust wide-baseline stereo from maximally stable extremal regions , 2004, Image Vis. Comput..

[36]  Joachim Weickert,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Generalised Nonlocal Image Smoothing Generalised Nonlocal Image Smoothing , 2022 .

[37]  Giacomo Boracchi,et al.  Foveated self-similarity in nonlocal image filtering , 2012, Electronic Imaging.

[38]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.