Variational Model-Based Deep Neural Networks for Image Reconstruction

In recent years, we have witnessed unprecedented growth of research interests in deep learning approaches to image reconstruction. A majority of these approaches are inspired by the well-developed variational method and associated optimization algorithms for the inverse problem of image reconstruction. These approaches mimic the iterative schemes of the standard optimization algorithms but integrate learnable components to form structured deep neural networks, and employ large amount of observation data to train the networks for the specific reconstruction tasks. They have demonstrated significantly improved empirical performance and requiremuch lower computational cost compared to the classical methods in a variety of applications. We provide the details of the derivations, the network architectures and the training procedures for several typical networks in this field.

[1]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[2]  Yin Zhang,et al.  An efficient augmented Lagrangian method with applications to total variation minimization , 2013, Computational Optimization and Applications.

[3]  Geoffrey E. Hinton,et al.  Application of Deep Belief Networks for Natural Language Understanding , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[4]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[5]  Dong Liang,et al.  Model Learning: Primal Dual Networks for Fast MR imaging , 2019, MICCAI.

[6]  Sundeep Rangan,et al.  AMP-Inspired Deep Networks for Sparse Linear Inverse Problems , 2016, IEEE Transactions on Signal Processing.

[7]  Kari Pulli,et al.  FlexISP , 2014, ACM Trans. Graph..

[8]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[9]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[10]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Otmar Scherzer,et al.  Variational Methods in Imaging , 2008, Applied mathematical sciences.

[12]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[13]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[14]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[15]  Wen Gao,et al.  Maximal Sparsity with Deep Networks? , 2016, NIPS.

[16]  Feng Jiang,et al.  Scalable Convolutional Neural Network for Image Compressed Sensing , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Xiaojing Ye,et al.  Learnable Descent Algorithm for Nonsmooth Nonconvex Image Reconstruction , 2021, SIAM J. Imaging Sci..

[18]  Michael Möller,et al.  Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Bernard Ghanem,et al.  ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  J. Morel,et al.  A multiscale algorithm for image segmentation by variational method , 1994 .

[21]  L. Vese,et al.  A Variational Method in Image Recovery , 1997 .

[22]  Jean-Michel Morel,et al.  A variational method in image segmentation: Existence and approximation results , 1992 .

[23]  Xiaohan Chen,et al.  ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA , 2018, ICLR.

[24]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[25]  Guangcan Liu,et al.  Differentiable Linearized ADMM , 2019, ICML.

[26]  Yoshua Bengio,et al.  Deep Learning for NLP (without Magic) , 2012, ACL.

[27]  Jian Sun,et al.  Deep ADMM-Net for Compressive Sensing MRI , 2016, NIPS.

[28]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[29]  Wuzhen Shi,et al.  Deep networks for compressed image sensing , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[30]  Pavan K. Turaga,et al.  ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Measurements , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Xiaohan Chen,et al.  Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds , 2018, NeurIPS.

[32]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[34]  Michael J. Black,et al.  Fields of Experts , 2009, International Journal of Computer Vision.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Kede Ma,et al.  Waterloo Exploration Database: New Challenges for Image Quality Assessment Models. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[37]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[38]  Sanja Fidler,et al.  Proximal Deep Structured Models , 2016, NIPS.

[39]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Jonas Adler,et al.  Learned Primal-Dual Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[42]  Guillermo Sapiro,et al.  Learning Efficient Sparse and Low Rank Models , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[44]  Lei Zhang,et al.  Waterloo Exploration Database: New Challenges for Image Quality Assessment Models , 2017, IEEE Transactions on Image Processing.

[45]  Xuanqin Mou,et al.  Machine Learning for Tomographic Imaging , 2020 .

[46]  Jeffrey A. Fessler,et al.  Momentum-Net: Fast and Convergent Iterative Neural Network for Inverse Problems , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.