Variational Networks: Connecting Variational Methods and Deep Learning

In this paper, we introduce variational networks (VNs) for image reconstruction. VNs are fully learned models based on the framework of incremental proximal gradient methods. They provide a natural transition between classical variational methods and state-of-the-art residual neural networks. Due to their incremental nature, VNs are very efficient, but only approximately minimize the underlying variational model. Surprisingly, in our numerical experiments on image reconstruction problems it turns out that giving up exact minimization leads to a consistent performance increase, in particular in the case of convex models.

[1]  Dimitri P. Bertsekas,et al.  Incremental proximal methods for large scale convex optimization , 2011, Math. Program..

[2]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[3]  Song-Chun Zhu,et al.  Prior Learning and Gibbs Reaction-Diffusion , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Li Xu,et al.  Unnatural L0 Sparse Representation for Natural Image Deblurring , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  D. Bertsekas,et al.  Convergen e Rate of In remental Subgradient Algorithms , 2000 .

[6]  Jiong Wang,et al.  On the Convergence of Generalized Simultaneous Iterative Reconstruction Algorithms , 2007, IEEE Transactions on Image Processing.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[9]  Tao Zhang,et al.  Active contours for tracking distributions , 2004, IEEE Transactions on Image Processing.

[10]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.

[11]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[12]  Jiaya Jia,et al.  High-quality motion deblurring from a single image , 2008, ACM Trans. Graph..

[13]  Jan Kautz,et al.  Loss Functions for Neural Networks for Image Processing , 2015, ArXiv.

[14]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[15]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[16]  Saeid Nahavandi,et al.  Multi-Residual Networks , 2016, ArXiv.

[17]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[18]  Sebastian Nowozin,et al.  Interleaved Regression Tree Field Cascades for Blind Image Deconvolution , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[19]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[20]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[21]  Vivek S. Borkar,et al.  Distributed Asynchronous Incremental Subgradient Methods , 2001 .

[22]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, CVPR.

[23]  Wei Yu,et al.  On learning optimized reaction diffusion processes for effective image restoration , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[25]  Olvi L. Mangasarian,et al.  Backpropagation Convergence via Deterministic Nonmonotone Perturbed Minimization , 1993, NIPS.

[26]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[27]  Thomas Pock,et al.  Learning joint demosaicing and denoising based on sequential energy minimization , 2016, 2016 IEEE International Conference on Computational Photography (ICCP).

[28]  Serge J. Belongie,et al.  Residual Networks are Exponential Ensembles of Relatively Shallow Networks , 2016, ArXiv.

[29]  Yunjin Chen,et al.  Insights Into Analysis Operator Learning: From Patch-Based Sparse Models to Higher Order MRFs , 2014, IEEE Transactions on Image Processing.

[30]  Guillermo Sapiro,et al.  Morphing Active Contours , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Suvrit Sra,et al.  Scalable nonconvex inexact proximal splitting , 2012, NIPS.

[32]  Wei Yu,et al.  Learning Reaction-Diffusion Models for Image Inpainting , 2015, GCPR.

[33]  Justin Domke,et al.  Generic Methods for Optimization-Based Modeling , 2012, AISTATS.

[34]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[35]  Michael J. Black,et al.  Fields of Experts , 2009, International Journal of Computer Vision.

[36]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[37]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[38]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[39]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[40]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[41]  Karl Kunisch,et al.  A Bilevel Optimization Approach for Parameter Learning in Variational Models , 2013, SIAM J. Imaging Sci..

[42]  Kim Steenstrup Pedersen,et al.  The Nonlinear Statistics of High-Contrast Patches in Natural Images , 2003, International Journal of Computer Vision.

[43]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[44]  Stefan Roth,et al.  Shrinkage Fields for Effective Image Restoration , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Alexandre Boulch.  ShaResNet: reducing residual network parameter number by sharing weights , 2017, ArXiv.

[46]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[47]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[48]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[49]  Thomas Pock,et al.  Learning a Variational Model for Compressed Sensing MRI Reconstruction , 2016 .

[50]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.