Multi-scale skip-connection network for image super-resolution

A skip-connection learning framework-based convolution neural network (CNN) has recently achieved great success in image super-resolution (SR). However, most CNN models based on the skip-connection learning framework do not fully make use of potential multi-scale features of images. In this paper, we propose a multi-scale skip-connection network (MSN) to improve the visual quality of the image SR. First, convolution kernels with different sizes are exploited to capture the multi-scale features of LR images. All the feature-maps captured by convolution kernels of the same size are direct input into a multi-scale hybrid group (MHG); second, the convolution layers of each MHG are composed of dilated convolutions and standard convolutions. The hybrid convolutions can fully train feature details obtained from preceding and current scale convolution layers; three, the output of each hybrid convolution layer is fed into subsequent hybrid convolution layers by skip-connections, thus producing dense connections; lastly, the meta-upscale module is used as the upscale module, which can magnify the trained feature maps arbitrary scale factors. By being evaluated on a wide variety of images, the proposed MSN network achieves an advantage over the state-of-the-art methods in terms of both numerical results and visual quality.

[1]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Dacheng Tao,et al.  Single Image Superresolution via Directional Group Sparsity and Directional Features , 2015, IEEE Transactions on Image Processing.

[4]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[5]  Xingyi Li,et al.  Filter shaping for Convolutional Neural Networks , 2016, International Conference on Learning Representations.

[6]  Gwanggil Jeon,et al.  Gated Multiple Feedback Network for Image Super-Resolution , 2019, BMVC.

[7]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[8]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Xuelong Li,et al.  Principal Component 2-D Long Short-Term Memory for Font Recognition on Single Chinese Characters , 2016, IEEE Transactions on Cybernetics.

[12]  Cheolkon Jung,et al.  DCSR: Dilated Convolutions for Single Image Super-Resolution , 2019, IEEE Transactions on Image Processing.

[13]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[18]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Luc Van Gool,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Lei Zhang,et al.  An edge-guided image interpolation algorithm via directional filtering and data fusion , 2006, IEEE Transactions on Image Processing.

[22]  Biyao Shao,et al.  3D Room Layout Estimation From a Single RGB Image , 2020, IEEE Transactions on Multimedia.

[23]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[24]  Zhen Li,et al.  Deep recursive up-down sampling networks for single image super-resolution , 2020, Neurocomputing.

[25]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[26]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[27]  Xuelong Li,et al.  Image Super-Resolution With Sparse Neighbor Embedding , 2012, IEEE Transactions on Image Processing.

[28]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Tieniu Tan,et al.  Meta-SR: A Magnification-Arbitrary Network for Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Kiyoharu Aizawa,et al.  Sketch-based manga retrieval using manga109 dataset , 2015, Multimedia Tools and Applications.

[31]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[33]  Yue Gao,et al.  Deep Multi-View Enhancement Hashing for Image Retrieval , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[35]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[36]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Aline Roumy,et al.  Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding , 2012, BMVC.

[38]  R. Venkatesh Babu,et al.  Image Denoising via CNNs: An Adversarial Approach , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Chun Chen,et al.  Image-based facial sketch-to-photo synthesis via online coupled dictionary learning , 2012, Inf. Sci..

[40]  Xiangjun Zhang,et al.  Image Interpolation by Adaptive 2-D Autoregressive Modeling and Soft-Decision Estimation , 2008, IEEE Transactions on Image Processing.

[41]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Yongdong Zhang,et al.  Depth Image Denoising Using Nuclear Norm and Learning Graph Model , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[43]  Xuelong Li,et al.  Single Image Super-Resolution With Non-Local Means and Steering Kernel Regression , 2012, IEEE Transactions on Image Processing.

[44]  Jiancheng Lv,et al.  Deep learning-based automatic downbeat tracking: a brief review , 2019, Multimedia Systems.

[45]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[46]  Wei Wu,et al.  Feedback Network for Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).