Image inpainting based on sparse representations with a perceptual metric

This paper presents an image inpainting method based on sparse representations optimized with respect to a perceptual metric. In the proposed method, the structural similarity (SSIM) index is utilized as a criterion to optimize the representation performance of image data. Specifically, the proposed method enables the formulation of two important procedures in the sparse representation problem, 'estimation of sparse representation coefficients’ and 'update of the dictionary’, based on the SSIM index. Then, using the generated dictionary, approximation of target patches including missing areas via the SSIM-based sparse representation becomes feasible. Consequently, image inpainting for which procedures are totally derived from the SSIM index is realized. Experimental results show that the proposed method enables successful inpainting of missing areas.

[1]  Zhou Wang,et al.  Modern Image Quality Assessment , 2006, Modern Image Quality Assessment.

[2]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[3]  Brendt Wohlberg,et al.  Inpainting with sparse linear combinations of exemplars , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[4]  Miki Haseyama,et al.  Missing Intensity Interpolation Using a Kernel PCA-Based POCS Algorithm and its Applications , 2011, IEEE Transactions on Image Processing.

[5]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[6]  Mehmet Türkan,et al.  Locally linear embedding based texture synthesis for image prediction and error concealment , 2012, 2012 19th IEEE International Conference on Image Processing.

[7]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[8]  Dianne P. O'Leary,et al.  Deblurring Images: Matrices, Spectra, and Filtering (Fundamentals of Algorithms 3) (Fundamentals of Algorithms) , 2006 .

[9]  Marcelo Bertalmío,et al.  Strong-continuation, contrast-invariant inpainting with a third-order optimal PDE , 2006, IEEE Transactions on Image Processing.

[10]  Ling Shao,et al.  Nonlocal Hierarchical Dictionary Learning Using Wavelets for Image Denoising , 2013, IEEE Transactions on Image Processing.

[11]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[12]  Charlie C. L. Wang,et al.  Interactive Image Inpainting Using DCT Based Exemplar Matching , 2009, ISVC.

[13]  Christine Guillemot,et al.  Hierarchical Super-Resolution-Based Inpainting , 2013, IEEE Transactions on Image Processing.

[14]  Mingming Li,et al.  A Novel Inpainting Model for Partial Differential Equation Based on Curvature Function , 2012, J. Multim..

[15]  Robert W. Heath,et al.  Design of Linear Equalizers Optimized for the Structural Similarity Index , 2008, IEEE Transactions on Image Processing.

[16]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[17]  Bernd Girod,et al.  What's wrong with mean-squared error? , 1993 .

[18]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[19]  Christine Guillemot,et al.  Super-Resolution-Based Inpainting , 2012, ECCV.

[20]  Toshiyuki Amano,et al.  Image interpolation using BPLP method on the eigenspace , 2007, Systems and Computers in Japan.

[21]  Zongben Xu,et al.  Image Inpainting by Patch Propagation Using Patch Sparsity , 2010, IEEE Transactions on Image Processing.

[22]  Jean-Michel Morel,et al.  Level lines based disocclusion , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[23]  Alan C. Bovik,et al.  Image information and visual quality , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[24]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[25]  Guangming Shi,et al.  Structure guided fusion for depth map inpainting , 2013, Pattern Recognit. Lett..

[26]  Gerard de Haan,et al.  An Overview and Performance Evaluation of Classification-Based Least Squares Trained Filters , 2008, IEEE Transactions on Image Processing.

[27]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[28]  L. Shao,et al.  From Heuristic Optimization to Dictionary Learning: A Review and Comprehensive Comparison of Image Denoising Algorithms , 2014, IEEE Transactions on Cybernetics.

[29]  Wilson S. Geisler,et al.  Image quality assessment based on a degradation model , 2000, IEEE Trans. Image Process..

[30]  A. Bovik,et al.  Image Quality Assessment , 2012 .

[31]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[32]  Wei Hu,et al.  Image inpainting via sparse representation , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[33]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[34]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[35]  Djemel Ziou,et al.  A global approach for solving evolutive heat transfer for image denoising and inpainting , 2006, IEEE Transactions on Image Processing.

[36]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[37]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[38]  Taku Komura,et al.  Topology matching for fully automatic similarity estimation of 3D shapes , 2001, SIGGRAPH.

[39]  Charlie C. L. Wang,et al.  Fast Query for Exemplar-Based Image Completion , 2010, IEEE Transactions on Image Processing.

[40]  D. Donoho,et al.  Atomic Decomposition by Basis Pursuit , 2001 .

[41]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[42]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[43]  Ze-Nian Li,et al.  Review and Preview: Disocclusion by Inpainting for Image-Based Rendering , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[44]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[45]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[46]  Andrew B. Watson,et al.  Digital images and human vision , 1993 .

[47]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[48]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[49]  Takashi Shibata,et al.  Fast and Structure-Preserving Image Inpainting Based on Probabilistic Structure Estimation , 2012, IEICE Trans. Inf. Syst..

[50]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[51]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[52]  D. Donoho,et al.  Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA) , 2005 .

[53]  Qing Zhang,et al.  Exemplar-Based Image Inpainting Using Color Distribution Analysis , 2012, J. Inf. Sci. Eng..

[54]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[55]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[56]  Bernhard Schölkopf,et al.  Iterative kernel principal component analysis for image modeling , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Miki Haseyama,et al.  Restoration method of missing areas in still images using GMRF model , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[58]  Tony F. Chan,et al.  Nontexture Inpainting by Curvature-Driven Diffusions , 2001, J. Vis. Commun. Image Represent..

[59]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[60]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[61]  T. Takahashi,et al.  Structured matrix rank minimization approach to image inpainting , 2012, 2012 IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS).

[62]  Mehmet Türkan,et al.  Object removal and loss concealment using neighbor embedding methods , 2013, Signal Process. Image Commun..

[63]  Michael Elad,et al.  Sparse Representation for Color Image Restoration , 2008, IEEE Transactions on Image Processing.

[64]  Anil C. Kokaram,et al.  A statistical framework for picture reconstruction using 2D AR models , 2004, Image and Vision Computing.

[65]  Marcel J. T. Reinders,et al.  Edge-based image restoration , 2005, IEEE Transactions on Image Processing.

[66]  Edward R. Vrscay,et al.  SSIM-inspired image restoration using sparse representation , 2012, EURASIP Journal on Advances in Signal Processing.

[67]  King Ngi Ngan,et al.  An efficient framework for image/video inpainting , 2013, Signal Process. Image Commun..