Compressible Latent-Space Invertible Networks for Generative Model-Constrained Image Reconstruction

There remains an important need for the development of image reconstruction methods that can produce diagnostically useful images from undersampled measurements. In magnetic resonance imaging (MRI), for example, such methods can facilitate reductions in data-acquisition times. Deep learning-based methods hold potential for learning object priors or constraints that can serve to mitigate the effects of data-incompleteness on image reconstruction. One line of emerging research involves formulating an optimization-based reconstruction method in the latent space of a generative deep neural network. However, when generative adversarial networks (GANs) are employed, such methods can result in image reconstruction errors if the sought-after solution does not reside within the range of the GAN. To circumvent this problem, in this work, a framework for reconstructing images from incomplete measurements is proposed that is formulated in the latent space of invertible neural network-based generative models. A novel regularization strategy is introduced that takes advantage of the multiscale architecture of certain invertible neural networks, which can result in improved reconstruction performance over classical methods in terms of traditional metrics. The proposed method is investigated for reconstructing images from undersampled MRI data. The method is shown to achieve comparable performance to a state-of-the-art generative model-based reconstruction method while benefiting from a deterministic reconstruction procedure and easier control over regularization parameters.

[1]  J. A. Clarkson,et al.  On definitions of bounded variation for functions of two variables , 1933 .

[2]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[3]  Raja Giryes,et al.  Image-Adaptive GAN based Reconstruction , 2019, AAAI.

[4]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[5]  Eric Nalisnick,et al.  Normalizing Flows for Probabilistic Modeling and Inference , 2019, J. Mach. Learn. Res..

[6]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[7]  Thomas Pock,et al.  Inverse GANs for accelerated MRI reconstruction , 2019, Optical Engineering + Applications.

[8]  D. Donoho For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[9]  M. Lustig,et al.  Compressed Sensing MRI , 2008, IEEE Signal Processing Magazine.

[10]  Michael Unser,et al.  Deep Convolutional Neural Network for Inverse Problems in Imaging , 2016, IEEE Transactions on Image Processing.

[11]  Yonina C. Eldar,et al.  From Theory to Practice: Sub-Nyquist Sampling of Sparse Wideband Analog Signals , 2009, IEEE Journal of Selected Topics in Signal Processing.

[12]  Alexandros G. Dimakis,et al.  Compressed Sensing with Deep Image Prior and Learned Regularization , 2018, ArXiv.

[13]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[14]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[15]  E. Sidky,et al.  Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization , 2008, Physics in medicine and biology.

[16]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[17]  Ben Adcock,et al.  BREAKING THE COHERENCE BARRIER: A NEW THEORY FOR COMPRESSED SENSING , 2013, Forum of Mathematics, Sigma.

[18]  Cynthia Rudin,et al.  PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Mark A. Anastasio,et al.  Deep Learning-Guided Image Reconstruction from Incomplete Data , 2017, ArXiv.

[20]  Jaakko Lehtinen,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Stefano Ermon,et al.  Modeling Sparse Deviations for Compressed Sensing using Generative Models , 2018, ICML.

[22]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[23]  Mark A. Anastasio,et al.  Medical image reconstruction with image-adaptive priors learned by use of generative adversarial networks , 2020, Medical Imaging 2020: Physics of Medical Imaging.

[24]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[25]  Pascal Vincent,et al.  fastMRI: An Open Dataset and Benchmarks for Accelerated MRI , 2018, ArXiv.

[26]  Anders C. Hansen,et al.  On the absence of the RIP in real-world applications of compressed sensing and the RIP in levels , 2014, ArXiv.

[27]  Ivan Kobyzev,et al.  Normalizing Flows: An Introduction and Review of Current Methods , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[29]  Ting Sun,et al.  Single-pixel imaging via compressive sampling , 2008, IEEE Signal Process. Mag..

[30]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[31]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[32]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[33]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[34]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[35]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[36]  Melanie Grunwald,et al.  Foundations Of Image Science , 2016 .

[37]  David Duvenaud,et al.  Invertible Residual Networks , 2018, ICML.

[38]  Jeffrey A. Fessler,et al.  Image Reconstruction: From Sparsity to Data-Adaptive Methods and Machine Learning , 2019, Proceedings of the IEEE.

[39]  Emil Y. Sidky,et al.  Do CNNs Solve the CT Inverse Problem? , 2020, IEEE Transactions on Biomedical Engineering.

[40]  Yoram Bresler,et al.  MR Image Reconstruction From Highly Undersampled k-Space Data by Dictionary Learning , 2011, IEEE Transactions on Medical Imaging.

[41]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[42]  Pieter Abbeel,et al.  Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design , 2019, ICML.

[43]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[44]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[45]  Yoram Bresler,et al.  Transform Learning for Magnetic Resonance Image Reconstruction: From Model-Based Learning to Building Neural Networks , 2019, IEEE Signal Processing Magazine.

[46]  Ali Ahmed,et al.  Invertible generative models for inverse problems: mitigating representation error and dataset bias , 2019, ICML.

[47]  Marlene Behrmann,et al.  Effects of unilateral cortical resection of the visual cortex on bilateral human white matter , 2019, NeuroImage.

[48]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[50]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[51]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[52]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.