Loss Functions for Image Restoration With Neural Networks

Neural networks are becoming central in several areas of computer vision and image processing and different architectures have been proposed to solve specific problems. The impact of the loss layer of neural networks, however, has not received much attention in the context of image processing: the default and virtually only choice is $\ell _2$. In this paper, we bring attention to alternative choices for image restoration. In particular, we show the importance of perceptually-motivated losses when the resulting image is to be evaluated by a human observer. We compare the performance of several losses, and propose a novel, differentiable error function. We show that the quality of the results improves significantly with better loss functions, even when the network architecture is left unchanged.

[1]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[2]  R. Kress,et al.  Inverse Acoustic and Electromagnetic Scattering Theory , 1992 .

[3]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[4]  Stefan Winkler,et al.  Visibility of noise in natural images , 2004, IS&T/SPIE Electronic Imaging.

[5]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[6]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[7]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[8]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[9]  Alessandro Foi,et al.  Clipped noisy images: Heteroskedastic modeling and practical denoising , 2009, Signal Process..

[10]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[11]  Alessandro Foi,et al.  Cross-color BM3D filtering of noisy raw data , 2009, 2009 International Workshop on Local and Non-Local Approximation in Image Processing.

[12]  Zhou Wang,et al.  Structural Similarity-Based Approximation of Signals and Images Using Orthogonal Bases , 2010, ICIAR.

[13]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Sylvain Paris,et al.  Learning photographic global tonal adjustment with a database of input/output image pairs , 2011, CVPR 2011.

[15]  Zhou Wang,et al.  Information Content Weighting for Perceptual Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[16]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[17]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[18]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Edward R. Vrscay,et al.  SSIM-inspired image restoration using sparse representation , 2012, EURASIP Journal on Advances in Signal Processing.

[20]  David Zhang,et al.  A comprehensive evaluation of full reference image quality assessment algorithms , 2012, 2012 19th IEEE International Conference on Image Processing.

[21]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[22]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[23]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[24]  Ce Liu,et al.  Deep Convolutional Neural Network for Image Deconvolution , 2014, NIPS.

[25]  Yi-Qing Wang,et al.  A multilayer neural network for image demosaicking , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[26]  Lei Zhang,et al.  Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index , 2013, IEEE Transactions on Image Processing.

[27]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Markus H. Gross,et al.  Perceptually based downscaling of images , 2015, ACM Trans. Graph..

[29]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).