Accelerating GMM-based patch priors for image restoration: Three ingredients for a 100x speed-up

Image restoration methods aim to recover the underlying clean image from corrupted observations. The Expected Patch Log-likelihood (EPLL) algorithm is a powerful image restoration method that uses a Gaussian mixture model (GMM) prior on the patches of natural images. Although it is very effective for restoring images, its high runtime complexity makes EPLL ill-suited for most practical applications. In this paper, we propose three approximations to the original EPLL algorithm. The resulting algorithm, which we call the fast-EPLL (FEPLL), attains a dramatic speed-up of two orders of magnitude over EPLL while incurring a negligible drop in the restored image quality (less than 0.5 dB). We demonstrate the efficacy and versatility of our algorithm on a number of inverse problems such as denoising, deblurring, super-resolution, inpainting and devignetting. To the best of our knowledge, FEPLL is the first algorithm that can competitively restore a 512×512 pixel image in under 0.5s for all the degradations mentioned above without specialized code optimizations such as CPU parallelization or GPU implementation.

[1]  Arnak S. Dalalyan,et al.  Image denoising with patch based PCA: local versus global , 2011, BMVC.

[2]  Yair Weiss,et al.  The Return of the Gating Network: Combining Generative Models and Discriminative Training in Natural Image Priors , 2015, NIPS.

[3]  Charles Bouveyron,et al.  High-Dimensional Mixture Models for Unsupervised Image Denoising (HDMI) , 2018, SIAM J. Imaging Sci..

[4]  Sebastian Nowozin,et al.  Regression Tree Fields — An efficient, non-parametric approach to image labeling problems , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Jean Ponce,et al.  Task-Driven Dictionary Learning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Karen O. Egiazarian,et al.  Image upsampling via spatially adaptive block-matching filtering , 2008, 2008 16th European Signal Processing Conference.

[9]  Yunjin Chen,et al.  Insights Into Analysis Operator Learning: From Patch-Based Sparse Models to Higher Order MRFs , 2014, IEEE Transactions on Image Processing.

[10]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[11]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Karen O. Egiazarian,et al.  Spatially Adaptive Filtering as Regularization in Inverse Imaging: Compressive Sensing, Super-Resolution, and Upsampling , 2017 .

[13]  Vladimir Katkovnik,et al.  Nonlocal image deblurring: Variational formulation with nonlocal collaborative L0-norm prior , 2009, 2009 International Workshop on Local and Non-Local Approximation in Image Processing.

[14]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[15]  Jean-Michel Morel,et al.  A Nonlocal Bayesian Image Denoising Algorithm , 2013, SIAM J. Imaging Sci..

[16]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[17]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[18]  Wei Yu,et al.  On learning optimized reaction diffusion processes for effective image restoration , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Donald Geman,et al.  Nonlinear image recovery with half-quadratic regularization , 1995, IEEE Trans. Image Process..

[20]  Truong Q. Nguyen,et al.  Adaptive Image Denoising by Mixture Adaptation , 2016, IEEE Transactions on Image Processing.

[21]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[22]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[23]  Nian Cai,et al.  Image denoising via patch-based adaptive Gaussian mixture prior method , 2016, Signal Image Video Process..

[24]  Michael Elad,et al.  Example-Based Image Synthesis via Randomized Patch-Matching , 2016, IEEE Transactions on Image Processing.

[25]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[26]  Leon Hirsch,et al.  Super Resolution From A Single Image , 2016 .

[27]  Shih-Fu Chang,et al.  Discriminative Indexing for Probabilistic Image Patch Priors , 2014, ECCV.

[28]  Ezequiel López-Rubio,et al.  Superresolution from a Single Noisy Image by the Median Filter Transform , 2016, SIAM J. Imaging Sci..

[29]  Michael Elad,et al.  Expected Patch Log Likelihood with a Sparse Prior , 2014, EMMCVPR.

[30]  Karen O. Egiazarian,et al.  Single image super-resolution via BM3D sparse coding , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[31]  Narendra Ahuja,et al.  Super-resolving Noisy Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Robert L. Cook,et al.  Stochastic sampling in computer graphics , 1988, TOGS.

[33]  Michael Elad,et al.  Multi-Scale Patch-Based Image Restoration , 2016, IEEE Transactions on Image Processing.

[34]  Stéphane Mallat,et al.  Solving Inverse Problems With Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity , 2010, IEEE Transactions on Image Processing.

[35]  Stefan Roth,et al.  Shrinkage Fields for Effective Image Restoration , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[37]  Michael Elad,et al.  Generalizing the Nonlocal-Means to Super-Resolution Reconstruction , 2009, IEEE Transactions on Image Processing.

[38]  Jacob Goldberger,et al.  Hierarchical Clustering of a Mixture Model , 2004, NIPS.