Generative Patch Priors for Practical Compressive Image Recovery

In this paper, we propose the generative patch prior (GPP) that defines a generative prior for compressive image recovery, based on patch-manifold models. Unlike learned, image-level priors that are restricted to the range space of a pre-trained generator, GPP can recover a wide variety of natural images using a pre-trained patch generator. Additionally, GPP retains the benefits of generative priors like high reconstruction quality at extremely low sensing rates, while also being much more generally applicable. We show that GPP outperforms several unsupervised and supervised techniques on three different sensing models – linear compressive sensing with known, and unknown calibration settings, and the non-linear phase retrieval problem. Finally, we propose an alternating optimization strategy using GPP for joint calibration-and-reconstruction which performs favorably against several baselines on a real world, uncalibrated compressive sensing dataset. The code and models for GPP are available on github. 1.

[1]  Zhihao Xia,et al.  Training Image Estimators without Image Ground-Truth , 2019, NeurIPS.

[2]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[3]  Robin M Heidemann,et al.  Generalized autocalibrating partially parallel acquisitions (GRAPPA) , 2002, Magnetic resonance in medicine.

[4]  Andreas Krause,et al.  Advances in Neural Information Processing Systems (NIPS) , 2014 .

[5]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[6]  Guangming Shi,et al.  Compressive Sensing via Nonlocal Low-Rank Regularization , 2014, IEEE Transactions on Image Processing.

[7]  Ali Ahmed,et al.  Deep Ptych: Subsampled Fourier Ptychography Using Generative Priors , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Yin Zhang,et al.  An efficient augmented Lagrangian method with applications to total variation minimization , 2013, Computational Optimization and Applications.

[9]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[10]  Richard G. Baraniuk,et al.  Learned D-AMP: Principled Neural Network based Compressive Image Recovery , 2017, NIPS.

[11]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[12]  Hidekata Hontani,et al.  Manifold Modeling in Embedded Space: A Perspective for Interpreting "Deep Image Prior" , 2019, ArXiv.

[13]  Prateek Jain,et al.  Phase Retrieval Using Alternating Minimization , 2013, IEEE Transactions on Signal Processing.

[14]  Michael Elad,et al.  Calibrationless parallel imaging reconstruction based on structured low‐rank matrix completion , 2013, Magnetic resonance in medicine.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Chinmay Hegde,et al.  Algorithmic Guarantees for Inverse Imaging with Untrained Network Priors , 2019, NeurIPS.

[17]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[18]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Lei Tian,et al.  Deep learning approach for Fourier ptychography microscopy. , 2018, Optics express.

[20]  Richard G. Baraniuk,et al.  An Architecture for Compressive Imaging , 2006, 2006 International Conference on Image Processing.

[21]  Chinmay Hegde,et al.  Alternating Phase Projected Gradient Descent with Generative Priors for Solving Compressive Phase Retrieval , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Feng Liu,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries in Wavelet Domain , 2009, 2009 Fifth International Conference on Image and Graphics.

[23]  Richard G. Baraniuk,et al.  From Denoising to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[24]  Pavan K. Turaga,et al.  ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Measurements , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Pavan Turaga,et al.  Convolutional Neural Networks for Noniterative Reconstruction of Compressively Sensed Images , 2017, IEEE Transactions on Computational Imaging.

[26]  A. Robert Calderbank,et al.  Sensitivity to Basis Mismatch in Compressed Sensing , 2011, IEEE Trans. Signal Process..

[27]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[28]  Rama Chellappa,et al.  Example-Driven Manifold Priors for Image Deconvolution , 2011, IEEE Transactions on Image Processing.

[29]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[30]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[31]  Richard G. Baraniuk,et al.  A deep learning approach to structured signal recovery , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[32]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[33]  Richard G. Baraniuk,et al.  prDeep: Robust Phase Retrieval with a Flexible Deep Network , 2018, ICML.

[34]  Amir Beck,et al.  On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes , 2015, SIAM J. Optim..

[35]  Tali Dekel,et al.  SinGAN: Learning a Generative Model From a Single Natural Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[36]  Bernard Ghanem,et al.  ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Amit Ashok,et al.  Information-optimal Scalable Compressive Imaging System , 2014 .

[38]  Gang Wang,et al.  Sparse Phase Retrieval via Truncated Amplitude Flow , 2016, IEEE Transactions on Signal Processing.

[39]  Thomas G. Dietterich,et al.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations , 2018, ICLR.

[40]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[41]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[42]  Rushil Anirudh,et al.  MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking , 2020, International Journal of Computer Vision.

[43]  Karen O. Egiazarian,et al.  Image denoising with block-matching and 3D filtering , 2006, Electronic Imaging.

[44]  Thomas Strohmer,et al.  Self-Calibration and Bilinear Inverse Problems via Linear Least Squares , 2016, SIAM J. Imaging Sci..

[45]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[46]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[47]  Pavan K. Turaga,et al.  Rate-Adaptive Neural Networks for Spatial Multiplexers , 2018, ArXiv.

[48]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[49]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[50]  Paul Hand,et al.  Global Guarantees for Blind Demodulation with Generative Priors , 2019, NeurIPS.

[51]  Thomas Strohmer,et al.  Self-calibration and biconvex compressive sensing , 2015, ArXiv.

[52]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[53]  Chinmay Hegde,et al.  Fast, Sample-Efficient Algorithms for Structured Phase Retrieval , 2017, NIPS.

[54]  Greg Ongie,et al.  Learned Patch-Based Regularization for Inverse Problems in Imaging , 2019, 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[55]  P. Batchelor,et al.  International Society for Magnetic Resonance in Medicine , 1997 .

[56]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Ali Ahmed,et al.  Blind Image Deconvolution Using Deep Generative Priors , 2018, IEEE Transactions on Computational Imaging.

[58]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[59]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[60]  Gabriel Peyré,et al.  Manifold models for signals and images , 2009, Comput. Vis. Image Underst..

[61]  Xiaodong Li,et al.  Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow , 2015, ArXiv.

[62]  Reinhard Heckel,et al.  Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks , 2018, ICLR.

[63]  Vladislav Voroninski,et al.  Phase Retrieval Under a Generative Prior , 2018, NeurIPS.

[64]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[65]  Kaushik Mitra,et al.  Phase retrieval for Fourier Ptychography under varying amount of measurements , 2018, BMVC.

[66]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[67]  William T. Freeman,et al.  The patch transform and its applications to image editing , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.