Convolutional Proximal Neural Networks and Plug-and-Play Algorithms

In this paper, we introduce convolutional proximal neural networks (cPNNs), which are by construction averaged operators. For filters of full length, we propose a stochastic gradient descent algorithm on a submanifold of the Stiefel manifold to train cPNNs. In case of filters with limited length, we design algorithms for minimizing functionals that approximate the orthogonality constraints imposed on the operators by penalizing the least squares distance to the identity operator. Then, we investigate how scaled cPNNs with a prescribed Lipschitz constant can be used for denoising signals and images, where the achieved quality depends on the Lipschitz constant. Finally, we apply cPNN based denoisers within a Plug-and-Play (PnP) framework and provide convergence results for the corresponding PnP forward-backward splitting algorithm based on an oracle construction.

[1]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[2]  Gabriele Steidl,et al.  First order algorithms in variational image processing , 2014, ArXiv.

[3]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[4]  Karl Kunisch,et al.  Variational Networks: An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration , 2020, Journal of Mathematical Imaging and Vision.

[5]  Karen O. Egiazarian,et al.  BM3D Frames and Variational Image Deblurring , 2011, IEEE Transactions on Image Processing.

[6]  Charles A. Bouman,et al.  Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation , 2015, IEEE Transactions on Computational Imaging.

[7]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[8]  P. L. Combettes,et al.  Compositions and convex combinations of averaged nonexpansive operators , 2014, 1407.5100.

[9]  Gilbert Strang,et al.  Functions of Difference Matrices Are Toeplitz Plus Hankel , 2014, SIAM Rev..

[10]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[11]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  W. R. Mann,et al.  Mean value methods in iteration , 1953 .

[13]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[14]  Andrea Braides.  Γ-convergence for beginners , 2002 .

[15]  Stanley H. Chan,et al.  Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications , 2016, IEEE Transactions on Computational Imaging.

[16]  S. Neumayer,et al.  Stabilizing invertible neural networks using mixture models , 2020, Inverse Problems.

[17]  Patrick L. Combettes,et al.  Monotone operator theory in convex optimization , 2018, Math. Program..

[18]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[19]  Laurent Condat,et al.  Proximal splitting algorithms: Relax them all! , 2019 .

[20]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[21]  Michael Unser,et al.  CNN-Based Projected Gradient Descent for Consistent CT Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[22]  Jean-François Aujol,et al.  Estimation of the Noise Level Function Based on a Nonparametric Detection of Homogeneous Image Regions , 2015, SIAM J. Imaging Sci..

[23]  Audrey Repetti,et al.  Building Firmly Nonexpansive Convolutional Neural Networks , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[25]  Michael Möller,et al.  Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Patrick L. Combettes *.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[27]  Yonina C. Eldar,et al.  Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing , 2021, IEEE Signal Processing Magazine.

[28]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  N. Higham.  Computing the polar decomposition with applications , 1986 .

[30]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[31]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[32]  Masashi Sugiyama,et al.  Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks , 2018, NeurIPS.

[33]  Bernhard Pfahringer,et al.  Regularisation of neural networks by enforcing Lipschitz continuity , 2018, Machine Learning.

[34]  Patrick L. Combettes,et al.  Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators , 2019, SIAM J. Math. Data Sci..

[35]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[36]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[37]  Gabriele Steidl,et al.  Inertial Stochastic PALM and its Application for Learning Student-t Mixture Models , 2020, ArXiv.

[38]  Xianglong Liu,et al.  Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks , 2017, AAAI.

[39]  Philip M. Long,et al.  The Singular Values of Convolutional Layers , 2018, ICLR.

[40]  Dario Bini,et al.  SPECTRAL AND COMPUTATIONAL PROPERTIES OF BAND SYMMETRIC TOEPLITZ MATRICES , 1983 .

[41]  Patrick L. Combettes,et al.  Deep Neural Network Structures Solving Variational Inequalities , 2018, Set-Valued and Variational Analysis.

[42]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[43]  Paul Vicol,et al.  Understanding and mitigating exploding inverses in invertible neural networks , 2020, AISTATS.

[44]  Ralph Byers,et al.  A New Scaling for Newton's Iteration for the Polar Decomposition and its Backward Stability , 2008, SIAM J. Matrix Anal. Appl..

[45]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[46]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[47]  Gabriele Steidl,et al.  Preconditioners for Ill-Conditioned Toeplitz Matrices , 1999 .

[48]  Gabriele Steidl,et al.  Parseval Proximal Neural Networks , 2019, Journal of Fourier Analysis and Applications.

[49]  D. Russell Luke,et al.  Quantitative Convergence Analysis of Iterated Expansive, Set-Valued Mappings , 2016, Math. Oper. Res..

[50]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[51]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[52]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[53]  José M. Bioucas-Dias,et al.  Scene-Adapted plug-and-play algorithm with convergence guarantees , 2017, 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP).

[54]  Shotaro Akaho,et al.  Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold , 2005, Neurocomputing.

[55]  Jun Li,et al.  Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform , 2020, ICLR.

[56]  Shunsuke Ono,et al.  Primal-Dual Plug-and-Play Image Restoration , 2017, IEEE Signal Processing Letters.

[57]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[58]  Nicholas J. Higham,et al.  Functions of matrices - theory and computation , 2008 .

[59]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[60]  Brendt Wohlberg,et al.  An Online Plug-and-Play Algorithm for Regularized Image Reconstruction , 2018, IEEE Transactions on Computational Imaging.

[61]  Patrick L. Combettes,et al.  Proximal Thresholding Algorithm for Minimization over Orthonormal Bases , 2007, SIAM J. Optim..

[62]  Simon Setzer,et al.  Operator Splittings, Bregman Methods and Frame Shrinkage in Image Processing , 2011, International Journal of Computer Vision.

[63]  Michael Moeller,et al.  Energy Dissipation with Plug-and-Play Priors , 2019 .