Model Adaptation for Inverse Problems in Imaging

Deep neural networks have been applied successfully to a wide variety of inverse problems arising in computational imaging. These networks are typically trained using a forward model that describes the measurement process to be inverted, which is often incorporated directly into the network itself. However, these approaches lack robustness to drift of the forward model: if at test time the forward model varies (even slightly) from the one the network was trained for, the reconstruction performance can degrade substantially. Given a network trained to solve an initial inverse problem with a known forward model, we propose two novel procedures that adapt the network to a perturbed forward model, even without full knowledge of the perturbation. Our approaches do not require access to more labeled data (i.e., ground truth images), but only a small set of calibration measurements. We show these simple model adaptation procedures empirically achieve robustness to changes in the forward model in a variety of settings, including deblurring, super-resolution, and undersampled image reconstruction in magnetic resonance imaging.

[1]  Ben Adcock,et al.  The troublesome kernel: why deep learning for inverse problems is typically unstable , 2020, ArXiv.

[2]  Mathews Jacob,et al.  MoDL: Model-Based Deep Learning Architecture for Inverse Problems , 2017, IEEE Transactions on Medical Imaging.

[3]  Shady Abu Hussein,et al.  Correction Filter for Single Image Super-Resolution: Robustifying Off-the-Shelf Deep Super-Resolvers , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Bo Li,et al.  Improving Robustness of Deep-Learning-Based Image Reconstruction , 2020, ICML.

[5]  Simon R. Arridge,et al.  Solving inverse problems using data-driven models , 2019, Acta Numerica.

[6]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[7]  Jan Kotera,et al.  Convolutional Neural Networks for Direct Text Deblurring , 2015, BMVC.

[8]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[9]  Kalliopi Basioti,et al.  Image Restoration from Parametric Transformations using Generative Models , 2020, ArXiv.

[10]  Jeffrey A. Fessler,et al.  Model-Based Image Reconstruction for MRI , 2010, IEEE Signal Processing Magazine.

[11]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[12]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Carola-Bibiane Schönlieb,et al.  Adversarial Regularizers in Inverse Problems , 2018, NeurIPS.

[15]  Francesco Renna,et al.  On instabilities of deep learning in image reconstruction and the potential costs of AI , 2019, Proceedings of the National Academy of Sciences.

[16]  Ying Fu,et al.  Tuning-free Plug-and-Play Proximal Algorithm for Inverse Imaging Problems , 2020, ICML.

[17]  Afshin Rostamizadeh,et al.  Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling , 2018, ICML.

[18]  Joan-Josep Climent,et al.  A geometrical approach on generalized inverses by Neumann-type series , 2001 .

[19]  Jong Chul Ye,et al.  Deep learning with domain adaptation for accelerated projection‐reconstruction MR , 2018, Magnetic resonance in medicine.

[20]  Rebecca Willett,et al.  Neumann Networks for Linear Inverse Problems in Imaging , 2020, IEEE Transactions on Computational Imaging.

[21]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[22]  Alexandros G. Dimakis,et al.  Deep Learning Techniques for Inverse Problems in Imaging , 2020, IEEE Journal on Selected Areas in Information Theory.

[23]  Morteza Mardani,et al.  Neural Proximal Gradient Descent for Compressive Imaging , 2018, NeurIPS.

[24]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[25]  Rushil Anirudh,et al.  An Unsupervised Approach to Solving Inverse Problems using Generative Adversarial Networks , 2018, ArXiv.