SUNLayer: Stable denoising with generative networks

It has been experimentally established that deep neural networks can be used to produce good generative models for real world data. It has also been established that such generative models can be exploited to solve classical inverse problems like compressed sensing and super resolution. In this work we focus on the classical signal processing problem of image denoising. We propose a theoretical setting that uses spherical harmonics to identify what mathematical properties of the activation functions will allow signal denoising with local methods.

[1]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[2]  Kevin Gimpel,et al.  Generalizing and Improving Weight Initialization , 2016, ArXiv.

[3]  Vladislav Voroninski,et al.  Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk , 2017, IEEE Transactions on Information Theory.

[4]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[5]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[6]  Kevin Gimpel,et al.  Adjusting for Dropout Variance in Batch Normalization and Weight Initialization , 2016 .

[7]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[8]  Amit Singer,et al.  The sample complexity of multi-reference alignment , 2017, SIAM J. Math. Data Sci..

[9]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[10]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[11]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[12]  Thomas Brox,et al.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks , 2016, NIPS.

[13]  Yonina C. Eldar,et al.  Phase Retrieval via Matrix Completion , 2011, SIAM Rev..

[14]  Yonina C. Eldar,et al.  Phase Retrieval via Matrix Completion , 2013, SIAM J. Imaging Sci..

[15]  Amit Singer,et al.  Multireference alignment using semidefinite programming , 2013, ITCS.

[16]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[17]  Quoc V. Le,et al.  Swish: a Self-Gated Activation Function , 2017, 1710.05941.

[18]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[19]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[20]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.