Learning wavelet coefficients for face super-resolution

Face image super-resolution imaging is an important technology which can be utilized in crime scene investigations and public security. Modern CNN-based super-resolution produces excellent results in terms of peak signal-to-noise ratio and the structural similarity index (SSIM). However, perceptual quality is generally poor, and the details of the facial features are lost. To overcome this problem, we propose a novel deep neural network to predict the super-resolution wavelet coefficients in order to obtain clearer facial images. Firstly, this paper uses prior knowledge of face images to manually emphases relevant facial features with more attention. Then, a linear low-rank convolution in the network is used. Finally, image edge features from canny detector are applied to enhance super-resolution images during training. The experimental results show that the proposed method can achieve competitive PSNR and SSIM and produces images with much higher perceptual quality.

[1]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[5]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Tieniu Tan,et al.  Wavelet Domain Generative Adversarial Network for Multi-scale Face Hallucination , 2019, International Journal of Computer Vision.

[7]  A. D. Dranoshchuk,et al.  About Perceptual Quality Estimation for Image Compression , 2019, 2019 Wave Electronics and its Application in Information and Telecommunication Systems (WECONF).

[8]  Christian Ledig,et al.  Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Ruofeng Tong,et al.  Robust super resolution of compressed video , 2011, The Visual Computer.

[11]  Geoffrey E. Hinton,et al.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition-' Washington , D . C . , June , 1983 OPTIMAL PERCEPTUAL INFERENCE , 2011 .

[12]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Xing Gao,et al.  A hybrid wavelet convolution network with sparse-coding for image super-resolution , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[14]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[17]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[18]  Mohammad Norouzi,et al.  Pixel Recursive Super Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Jing Tian,et al.  Wavelet-Based Image Interpolation Using a Three-Component Exponential Mixture Model , 2008, 2008 Congress on Image and Signal Processing.

[20]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Amos Storkey,et al.  Advances in Neural Information Processing Systems 20 , 2007 .

[22]  Il Kyu Eom,et al.  Image interpolation based on inter-scale dependency in wavelet domain , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Neil A. Dodgson,et al.  Proceedings Ninth IEEE International Conference on Computer Vision , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[26]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[27]  Neeraj Kumar,et al.  Learning to predict super resolution wavelet coefficients , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[28]  Behzad Shahraray,et al.  Uniform Resampling of Digitized Contours , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Rynson W. H. Lau,et al.  Efficient image super-resolution integration , 2018, The Visual Computer.

[30]  Tieniu Tan,et al.  Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Thomas S. Huang,et al.  Non-Local Recurrent Network for Image Restoration , 2018, NeurIPS.

[32]  Vishal Monga,et al.  Deep Wavelet Prediction for Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Chao Zhang,et al.  Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution , 2018, NeurIPS.

[35]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[36]  Ali Aghagolzadeh,et al.  Single-image super-resolution via patch-based and group-based local smoothness modeling , 2019, The Visual Computer.

[37]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[39]  Xin Yu,et al.  Ultra-Resolving Face Images by Discriminative Generative Networks , 2016, ECCV.

[40]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[42]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[43]  Aggelos K. Katsaggelos,et al.  Digital image restoration , 2012, IEEE Signal Process. Mag..

[44]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[45]  Xiaoou Tang,et al.  Surpassing Human-Level Face Verification Performance on LFW with GaussianFace , 2014, AAAI.

[46]  Il Kyu Eom,et al.  Image Interpolation Based on Statistical Relationship Between Wavelet Subbands , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[47]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..