Generative Patch Priors for Practical Compressive Image Recovery

In this paper, we propose the generative patch prior (GPP) that defines a generative prior for compressive image recovery, based on patch-manifold models. Unlike learned, image-level priors that are restricted to the range space of a pre-trained generator, GPP can recover a wide variety of natural images using a pre-trained patch generator. Additionally, GPP retains the benefits of generative priors like high reconstruction quality at extremely low sensing rates, while also being much more generally applicable. We show that GPP outperforms several unsupervised and supervised techniques on three different sensing models -- linear compressive sensing with known, and unknown calibration settings, and the non-linear phase retrieval problem. Finally, we propose an alternating optimization strategy using GPP for joint calibration-and-reconstruction which performs favorably against several baselines on a real world, un-calibrated compressive sensing dataset.

[1]  Zhihao Xia,et al.  Training Image Estimators without Image Ground-Truth , 2019, NeurIPS.

[2]  Bernard Ghanem,et al.  ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Pavan K. Turaga,et al.  ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Measurements , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Hidekata Hontani,et al.  Manifold Modeling in Embedded Space: A Perspective for Interpreting "Deep Image Prior" , 2019, ArXiv.

[5]  Greg Ongie,et al.  Learned Patch-Based Regularization for Inverse Problems in Imaging , 2019, 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[6]  Prateek Jain,et al.  Phase Retrieval Using Alternating Minimization , 2013, IEEE Transactions on Signal Processing.

[7]  Richard G. Baraniuk,et al.  From Denoising to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[8]  P. Batchelor,et al.  International Society for Magnetic Resonance in Medicine , 1997 .

[9]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[10]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[11]  Reinhard Heckel,et al.  Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks , 2018, ICLR.

[12]  Kaushik Mitra,et al.  Phase retrieval for Fourier Ptychography under varying amount of measurements , 2018, BMVC.

[13]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[14]  Paul Hand,et al.  Global Guarantees for Blind Demodulation with Generative Priors , 2019, NeurIPS.

[15]  Guangming Shi,et al.  Compressive Sensing via Nonlocal Low-Rank Regularization , 2014, IEEE Transactions on Image Processing.

[16]  Thomas Strohmer,et al.  Self-calibration and biconvex compressive sensing , 2015, ArXiv.

[17]  Pavan K. Turaga,et al.  Rate-Adaptive Neural Networks for Spatial Multiplexers , 2018, ArXiv.

[18]  Richard G. Baraniuk,et al.  Learned D-AMP: Principled Neural Network based Compressive Image Recovery , 2017, NIPS.

[19]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[20]  Richard G. Baraniuk,et al.  An Architecture for Compressive Imaging , 2006, 2006 International Conference on Image Processing.

[21]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[22]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[23]  Thomas G. Dietterich,et al.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations , 2018, ICLR.

[24]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[25]  Yin Zhang,et al.  An efficient augmented Lagrangian method with applications to total variation minimization , 2013, Computational Optimization and Applications.

[26]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[27]  Chinmay Hegde,et al.  Algorithmic Guarantees for Inverse Imaging with Untrained Network Priors , 2019, NeurIPS.

[28]  Vladislav Voroninski,et al.  Phase Retrieval Under a Generative Prior , 2018, NeurIPS.

[29]  Richard G. Baraniuk,et al.  prDeep: Robust Phase Retrieval with a Flexible Deep Network , 2018, ICML.

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[32]  A. Robert Calderbank,et al.  Sensitivity to Basis Mismatch in Compressed Sensing , 2011, IEEE Trans. Signal Process..

[33]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Lei Tian,et al.  Deep learning approach for Fourier ptychography microscopy. , 2018, Optics express.

[35]  Gang Wang,et al.  Sparse Phase Retrieval via Truncated Amplitude Flow , 2016, IEEE Transactions on Signal Processing.

[36]  Gabriel Peyré,et al.  Manifold models for signals and images , 2009, Comput. Vis. Image Underst..

[37]  Feng Liu,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries in Wavelet Domain , 2009, 2009 Fifth International Conference on Image and Graphics.

[38]  Robin M Heidemann,et al.  Generalized autocalibrating partially parallel acquisitions (GRAPPA) , 2002, Magnetic resonance in medicine.

[39]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[40]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[41]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[42]  Amir Beck,et al.  On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes , 2015, SIAM J. Optim..

[43]  Karen O. Egiazarian,et al.  Image denoising with block-matching and 3D filtering , 2006, Electronic Imaging.

[44]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[45]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[47]  Chinmay Hegde,et al.  Fast, Sample-Efficient Algorithms for Structured Phase Retrieval , 2017, NIPS.

[48]  Tali Dekel,et al.  SinGAN: Learning a Generative Model From a Single Natural Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  William T. Freeman,et al.  The patch transform and its applications to image editing , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Thomas Strohmer,et al.  Self-Calibration and Bilinear Inverse Problems via Linear Least Squares , 2016, SIAM J. Imaging Sci..

[51]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[52]  Andreas Krause,et al.  Advances in Neural Information Processing Systems (NIPS) , 2014 .

[53]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[54]  Rushil Anirudh,et al.  MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking , 2020, International Journal of Computer Vision.

[55]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[56]  Richard G. Baraniuk,et al.  A deep learning approach to structured signal recovery , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[57]  Amit Ashok,et al.  Information-optimal Scalable Compressive Imaging System , 2014 .

[58]  Chinmay Hegde,et al.  Alternating Phase Projected Gradient Descent with Generative Priors for Solving Compressive Phase Retrieval , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[59]  Ali Ahmed,et al.  Blind Image Deconvolution Using Deep Generative Priors , 2018, IEEE Transactions on Computational Imaging.

[60]  Pavan Turaga,et al.  Convolutional Neural Networks for Noniterative Reconstruction of Compressively Sensed Images , 2017, IEEE Transactions on Computational Imaging.

[61]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[62]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[63]  Ali Ahmed,et al.  Deep Ptych: Subsampled Fourier Ptychography Using Generative Priors , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[64]  Michael Elad,et al.  Calibrationless parallel imaging reconstruction based on structured low‐rank matrix completion , 2013, Magnetic resonance in medicine.

[65]  Xiaodong Li,et al.  Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow , 2015, ArXiv.

[66]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[67]  Rama Chellappa,et al.  Example-Driven Manifold Priors for Image Deconvolution , 2011, IEEE Transactions on Image Processing.