From learning models of natural image patches to whole image restoration

Learning good image priors is of utmost importance for the study of vision, computer vision and image processing applications. Learning priors and optimizing over whole images can lead to tremendous computational challenges. In contrast, when we work with small image patches, it is possible to learn priors and perform patch restoration very efficiently. This raises three questions - do priors that give high likelihood to the data also lead to good performance in restoration? Can we use such patch based priors to restore a full image? Can we learn better patch priors? In this work we answer these questions. We compare the likelihood of several patch models and show that priors that give high likelihood to data perform better in patch restoration. Motivated by this result, we propose a generic framework which allows for whole image restoration using any patch based prior for which a MAP (or approximate MAP) estimate can be calculated. We show how to derive an appropriate cost function, how to optimize it and how to use it to restore whole images. Finally, we present a generic, surprisingly simple Gaussian Mixture prior, learned from a set of natural images. When used with the proposed framework, this Gaussian Mixture Model outperforms all other generic prior methods for image denoising, deblurring and inpainting.

[1]  D. Gabor,et al.  Theory of communication. Part 1: The analysis of information , 1946 .

[2]  E. Kretzmer.  Statistics of television signals , 1952 .

[3]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[4]  G. Matheron.  Random Sets and Integral Geometry , 1976 .

[5]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[6]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[7]  William Bialek,et al.  Statistics of Natural Images: Scaling in the Woods , 1993, NIPS.

[8]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[9]  Donald Geman,et al.  Nonlinear image recovery with half-quadratic regularization , 1995, IEEE Trans. Image Process..

[10]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[11]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[12]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[13]  Daniel L. Ruderman,et al.  Origins of scaling in natural images , 1996, Vision Research.

[14]  Song-Chun Zhu,et al.  Prior Learning and Gibbs Reaction-Diffusion , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[16]  J. V. van Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[17]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[18]  Martin J. Wainwright,et al.  Scale Mixtures of Gaussians and the Statistics of Natural Images , 1999, NIPS.

[19]  Eero P. Simoncelli.  Bayesian Denoising of Visual Images in the Wavelet Domain , 1999 .

[20]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[21]  Bruno A. Olshausen,et al.  PROBABILISTIC FRAMEWORK FOR THE ADAPTATION AND COMPARISON OF IMAGE CODES , 1999 .

[22]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[23]  J L Gallant,et al.  Sparse coding and decorrelation in primary visual cortex during natural vision. , 2000, Science.

[24]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[25]  Miguel Á. Carreira-Perpiñán,et al.  Mode-Finding for Mixtures of Gaussian Distributions , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Joseph W. Goodman,et al.  A mathematical analysis of the DCT coefficient distributions for images , 2000, IEEE Trans. Image Process..

[27]  Aapo Hyvärinen,et al.  Emergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces , 2000, Neural Computation.

[28]  H Barlow,et al.  Redundancy reduction revisited , 2001, Network.

[29]  P. Hoyer,et al.  TOPOGRAPHIC INDEPENDENT COMPONENT ANALYSIS: VISUALIZING THE DEPENDENCE STRUCTURE , 2001 .

[30]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[31]  Assaf Zomet,et al.  Learning to Perceive Transparency from the Statistics of Natural Scenes , 2002, NIPS.

[32]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[33]  Gerhard Lakemeyer,et al.  Exploring artificial intelligence in the new millennium , 2003 .

[34]  Martin J. Wainwright,et al.  Image denoising using scale mixtures of Gaussians in the wavelet domain , 2003, IEEE Trans. Image Process..

[35]  Michael I. Jordan,et al.  Beyond Independent Components: Trees and Clusters , 2003, J. Mach. Learn. Res..

[36]  Eero P. Simoncelli,et al.  On Advances in Statistical Modeling of Natural Images , 2004, Journal of Mathematical Imaging and Vision.

[37]  Konrad Paul Kording,et al.  How are complex cell properties adapted to the statistics of natural stimuli? , 2004, Journal of neurophysiology.

[38]  L. Maler,et al.  Plastic and Nonplastic Pyramidal Cells Perform Unique Roles in a Network Capable of Adaptive Redundancy Reduction , 2004, Neuron.

[39]  David Mumford,et al.  Occlusion Models for Natural Images: A Statistical Study of a Scale-Invariant Dead Leaves Model , 2004, International Journal of Computer Vision.

[40]  Rafael C. González,et al.  Digital image processing using MATLAB , 2006 .

[41]  David J. Field,et al.  How Close Are We to Understanding V1? , 2005, Neural Computation.

[42]  Michael Elad,et al.  K-SVD : DESIGN OF DICTIONARIES FOR SPARSE REPRESENTATION , 2005 .

[43]  Michael S. Lewicki,et al.  A Hierarchical Bayesian Model for Learning Nonlinear Statistical Regularities in Nonstationary Natural Signals , 2005, Neural Computation.

[44]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[45]  M. Bethge.  Factorial coding of natural images: how effective are linear models in removing higher-order dependencies? , 2006, Journal of the Optical Society of America. A, Optics, image science, and vision.

[46]  Carlo Gaetan,et al.  Composite likelihood methods for space-time data , 2006 .

[47]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[48]  Richard E. Turner,et al.  On Sparsity and Overcompleteness in Image Models , 2007, NIPS.

[49]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[50]  William T. Freeman,et al.  What makes a good model of natural images? , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Yiannis Aloimonos,et al.  Who killed the directed model? , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Yacov Hel-Or,et al.  A Discriminative Approach for Wavelet Denoising , 2008, IEEE Transactions on Image Processing.

[53]  Richard Szeliski,et al.  Submitted to Ieee Transactions on Pattern Analysis and Machine Intelligence Automatic Estimation and Removal of Noise from a Single Image , 2022 .

[54]  Karen O. Egiazarian,et al.  Image restoration by sparse 3D transform-domain collaborative filtering , 2008, Electronic Imaging.

[55]  Michael J. Black,et al.  Fields of Experts , 2009, International Journal of Computer Vision.

[56]  Yair Weiss,et al.  The 'tree-dependent components' of natural scenes are edge filters , 2009, NIPS.

[57]  Eero P. Simoncelli,et al.  Nonlinear Extraction of Independent Components of Natural Images Using Radial Gaussianization , 2009, Neural Computation.

[58]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[59]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[60]  Yan Yang,et al.  Image Denoising by Sparse Code Shrinkage , 2009, 2009 5th International Conference on Wireless Communications, Networking and Mobile Computing.

[61]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[62]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[63]  Richard E. Turner,et al.  Occlusive Components Analysis , 2009, NIPS.

[64]  Michael S. Lewicki,et al.  Emergence of complex cell properties by learning to generalize in natural scenes , 2009, Nature.

[65]  Yair Weiss,et al.  Scale invariance and noise in natural images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[66]  Xaq Pitkow,et al.  Exact feature probabilities in images with occlusion. , 2010, Journal of vision.

[67]  Jörg Lücke,et al.  The Maximal Causes of Natural Scenes are Edge Filters , 2010, NIPS.

[68]  Bruno A. Olshausen,et al.  Building a better probabilistic model of images by factorization , 2011, 2011 International Conference on Computer Vision.

[69]  R. Gray,et al.  The Importance of Intrinsically Two-dimensional Image Features in Biological Vision and Picture Coding , 2011 .

[70]  Matthias Bethge,et al.  In All Likelihood, Deep Belief Is Not Enough , 2010, J. Mach. Learn. Res..

[71]  Nicolas Le Roux,et al.  Learning a Generative Model of Images by Factoring Appearance and Shape , 2011, Neural Computation.

[72]  Jascha Sohl-Dickstein,et al.  Hamiltonian Annealed Importance Sampling for partition function estimation , 2012, ArXiv.

[73]  Stéphane Mallat,et al.  Solving Inverse Problems With Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity , 2010, IEEE Transactions on Image Processing.