Building Firmly Nonexpansive Convolutional Neural Networks

Building nonexpansive Convolutional Neural Networks (CNNs) is a challenging problem that has recently gained a lot of attention from the image processing community. In particular, it appears to be the key to obtain convergent Plugand-Play algorithms. This problem, which relies on an accurate control of the the Lipschitz constant of the convolutional layers, has also been investigated for Generative Adversarial Networks to improve robustness to adversarial perturbations. However, to the best of our knowledge, no efficient method has been developed yet to build nonexpansive CNNs. In this paper, we develop an optimization algorithm that can be incorporated in the training of a network to ensure the nonexpansiveness of its convolutional layers. This is shown to allow us to build firmly nonexpansive CNNs. We apply the proposed approach to train a CNN for an image denoising task and show its effectiveness through simulations.

[1]  J.-C. Pesquet,et al.  A Douglas–Rachford Splitting Approach to Nonsmooth Convex Variational Signal Recovery , 2007, IEEE Journal of Selected Topics in Signal Processing.

[2]  Cem Anil,et al.  Sorting out Lipschitz function approximation , 2018, ICML.

[3]  Chun-Liang Li,et al.  One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Bernard Ghanem,et al.  Deep Layers as Stochastic Solvers , 2018, International Conference on Learning Representations.

[5]  Moustapha Cissé,et al.  Parseval Networks: Improving Robustness to Adversarial Examples , 2017, ICML.

[6]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[7]  P. L. Combettes,et al.  Proximity for sums of composite functions , 2010, 1007.3535.

[8]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[9]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Xiaohan Chen,et al.  Plug-and-Play Methods Provably Converge with Properly Trained Denoisers , 2019, ICML.

[11]  Patrick L. Combettes,et al.  Stochastic forward-backward and primal-dual approximation algorithms with application to online image restoration , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[12]  Richard Hans Robert Hahnloser,et al.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit , 2000, Nature.

[13]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[14]  Philip M. Long,et al.  The Singular Values of Convolutional Layers , 2018, ICLR.

[15]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[16]  Jean-Christophe Pesquet,et al.  Deep unfolding of a proximal interior point method for image restoration , 2018, Inverse Problems.

[17]  L. Rosasco,et al.  A stochastic inertial forward–backward splitting algorithm for multivariate monotone inclusions , 2015, 1507.00848.

[18]  Yuichi Yoshida,et al.  Spectral Norm Regularization for Improving the Generalizability of Deep Learning , 2017, ArXiv.

[19]  Yong Cheng,et al.  Comments on "Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering" , 2011, IEEE Trans. Image Process..

[20]  Michael Möller,et al.  Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[22]  Patrick L. Combettes,et al.  Deep Neural Network Structures Solving Variational Inequalities , 2018, Set-Valued and Variational Analysis.