Fast and Painless Image Reconstruction in Deep Image Prior Subspaces

The deep image prior (DIP) is a state-of-the-art unsupervised approach for solving linear inverse problems in imaging. We address two key issues that have held back practical deployment of the DIP: the long computing time needed to train a separate deep network per reconstruction, and the susceptibility to overfitting due to a lack of robust early stopping strategies in the unsupervised setting. To this end, we restrict DIP optimisation to a sparse linear subspace of the full parameter space. We construct the subspace from the principal eigenspace of a set of parameter vectors sampled at equally spaced intervals during DIP pre-training on synthetic task-agnostic data. The low-dimensionality of the resulting subspace reduces DIP's capacity to fit noise and allows the use of fast second order optimisation methods, e.g., natural gradient descent or L-BFGS. Experiments across tomographic tasks of different geometry, ill-posedness and stopping criteria consistently show that second order optimisation in a subspace is Pareto-optimal in terms of optimisation time to reconstruction fidelity trade-off.

[1]  Y. Liu,et al.  Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Zhihui Zhu,et al.  A Validation Approach to Over-parameterized Matrix and Image Recovery , 2022, ArXiv.

[3]  Jos'e Miguel Hern'andez-Lobato,et al.  Bayesian Experimental Design for Computed Tomography with the Linearised Deep Image Prior , 2022, ArXiv.

[4]  Eric T. Nalisnick,et al.  Adapting the Linearised Laplace Model Evidence for Modern Deep Learning , 2022, ICML.

[5]  Yann LeCun,et al.  Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors , 2022, NeurIPS.

[6]  P. Maass,et al.  An Educated Warm Start for Deep Image Prior-Based Micro CT Reconstruction , 2021, IEEE Transactions on Computational Imaging.

[7]  Subhransu Maji,et al.  The Spectral Bias of the Deep Image Prior , 2019, ArXiv.

[8]  Taihui Li,et al.  Early Stopping for Deep Image Prior , 2021, ArXiv.

[9]  Di Guo,et al.  A review on deep learning MRI reconstruction without fully sampled k-space , 2021, BMC Medical Imaging.

[10]  Se Young Chun,et al.  Rethinking Deep Image Prior for Denoising , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Ali Farhadi,et al.  Learning Neural Network Subspaces , 2021, ICML.

[12]  José Miguel Hernández-Lobato,et al.  Bayesian Deep Learning via Subnetwork Inference , 2020, ICML.

[13]  Jong Chul Ye,et al.  Deep learning for tomographic image reconstruction , 2020, Nature Machine Intelligence.

[14]  Shuai Leng,et al.  Low Dose CT Image and Projection Dataset. , 2020, Medical physics.

[15]  Per-Gunnar Martinsson,et al.  Randomized numerical linear algebra: Foundations and algorithms , 2020, Acta Numerica.

[16]  Maximilian Schmidt,et al.  Computed tomography reconstruction using deep image prior and learned reconstruction methods , 2020, Inverse Problems.

[17]  Wangmeng Zuo,et al.  Deep Learning on Image Denoising: An overview , 2019, Neural Networks.

[18]  James Martens,et al.  New Insights and Perspectives on the Natural Gradient Method , 2014, J. Mach. Learn. Res..

[19]  Ulugbek Kamilov,et al.  Image Restoration Using Total Variation Regularized Deep Image Prior , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  Michael Carbin,et al.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , 2018, ICLR.

[21]  Richard G. Baraniuk,et al.  Unsupervised Learning with Stein's Unbiased Risk Estimator , 2018, ArXiv.

[22]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[23]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yiran Chen,et al.  Learning Structured Sparsity in Deep Neural Networks , 2016, NIPS.

[25]  Jan Sijbers,et al.  The ASTRA Toolbox: A platform for advanced algorithm development in electron tomography. , 2015, Ultramicroscopy.

[26]  Roger B. Grosse,et al.  Optimizing Neural Networks with Kronecker-factored Approximate Curvature , 2015, ICML.

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[29]  Shun-ichi Amari,et al.  Information Geometry and Its Applications: Survey , 2013, GSI.

[30]  David P. Woodruff,et al.  Fast approximation of matrix coherence and statistical leverage , 2011, ICML.

[31]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[32]  Massimo Fornasier,et al.  Theoretical Foundations and Numerical Methods for Sparse Recovery , 2010, Radon Series on Computational and Applied Mathematics.

[33]  A. Chambolle,et al.  An introduction to Total Variation for Image Analysis , 2009 .

[34]  Yonina C. Eldar.  Generalized SURE for Exponential Families: Applications to Regularization , 2008, IEEE Transactions on Signal Processing.

[35]  Nicol N. Schraudolph,et al.  Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent , 2002, Neural Computation.

[36]  M. Rattray,et al.  MATRIX MOMENTUM FOR PRACTICAL NATURAL GRADIENT LEARNING , 1999 .

[37]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[38]  Martin T. Hagan,et al.  Gauss-Newton approximation to Bayesian learning , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[39]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[40]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..