Combined frequency and spatial domain-based patch propagation for image completion

Film and photography archives now have an accelerated rate of degradation. Because the preservation of cultural heritage plays an important role in our society, photograph and film restoration has recently drawn a substantial amount of attention. In this paper, an approach that involves exemplar-based inpainting, aimed at determining patch priority and patch matching, is proposed. Different image regions have different levels of importance for vision perception; hence, a priority score must be assigned. Patch priority is calculated by the energy of its distributed cosine transform coefficients (DCT term) and the edge term. The edge term prioritizes the edge patches and the energy of the DCT coefficients of the patch is used as a discriminator for patches with similar edge terms. Patch inpainting is performed by assessing the similarity between the patches in such a manner that the similarity measure is consistent with human visual judgment. Therefore, a structure-based similarity measure is developed. Further, the interpolated missing pixels at the patch are also considered for applying the structure-based patch matching criteria in finding the candidate patch. Experimental results on damaged digitized photographs and natural images are presented, which demonstrate the effectiveness of the image completion framework for tasks such as scratch/text, object removal and image inpainting.

[1]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[2]  Charlie C. L. Wang,et al.  Gradient based image completion by solving the Poisson equation , 2007, Comput. Graph..

[3]  Marcelo Bertalmío,et al.  FLUID DYNAMICS, AND IMAGE AND VIDEO INPAINTING , 2001 .

[4]  Charlie C. L. Wang,et al.  Fast Query for Exemplar-Based Image Completion , 2010, IEEE Transactions on Image Processing.

[5]  John A. Saghri,et al.  Image Quality Measure Based On A Human Visual System Model , 1989 .

[6]  Zhou Wang,et al.  Multi-scale structural similarity for image quality assessment , 2003 .

[7]  Jiaya Jia,et al.  Image completion with structure propagation , 2005, SIGGRAPH 2005.

[8]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[9]  David Tschumperlé,et al.  Fast Anisotropic Smoothing of Multi-Valued Images using Curvature-Preserving PDE's , 2006, International Journal of Computer Vision.

[10]  Onur G. Guleryuz,et al.  Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part II: adaptive algorithms , 2006, IEEE Transactions on Image Processing.

[11]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[12]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[13]  Chuan Zhou,et al.  Gradient Based Image Completion by Solving Poisson Equation , 2005, PCM.

[14]  B. Wandell.  Foundations of vision , 1995 .

[15]  Tony F. Chan,et al.  Nontexture Inpainting by Curvature-Driven Diffusions , 2001, J. Vis. Commun. Image Represent..

[16]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[17]  Nikos Komodakis,et al.  Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning , 2007, IEEE Transactions on Image Processing.

[18]  Zongben Xu,et al.  Image Inpainting by Patch Propagation Using Patch Sparsity , 2010, IEEE Transactions on Image Processing.

[19]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[20]  David Zhang,et al.  Image information restoration based on long-range correlation , 2002, IEEE Trans. Circuits Syst. Video Technol..

[21]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[22]  Onur G. Guleryuz,et al.  Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part I: theory , 2006, IEEE Transactions on Image Processing.

[23]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[24]  Chaofeng Li,et al.  Three-component weighted structural similarity index , 2009, Electronic Imaging.

[25]  Alexander Wong,et al.  A nonlocal-means approach to exemplar-based inpainting , 2008, 2008 15th IEEE International Conference on Image Processing.

[26]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[27]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[28]  Qiuqi Ruan,et al.  Object Removal By Cross Isophotes Exemplar-based Inpainting , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[29]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[30]  Guillermo Sapiro,et al.  Structure and texture filling-in of missing image blocks in wireless transmission and compression applications , 2003, IEEE Trans. Image Process..

[31]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.