CISRDCNN: Super-resolution of compressed images using deep convolutional neural networks

In recent years, much research has been conducted on image super-resolution (SR). To the best of our knowledge, however, few SR methods were concerned with compressed images. The SR of compressed images is a challenging task due to the complicated compression artifacts, while many images suffer from them in practice. The intuitive solution for this difficult task is to decouple it into two sequential but independent subproblems, i.e., compression artifacts reduction (CAR) and SR. Nevertheless, some useful details may be removed in CAR stage, which is contrary to the goal of SR and makes the SR stage more challenging. In this paper, an end-to-end trainable deep convolutional neural network is designed to perform SR on compressed images (CISRDCNN), which reduces compression artifacts and improves image resolution jointly. Experiments on compressed images produced by JPEG (we take the JPEG as an example in this paper) demonstrate that the proposed CISRDCNN yields state-of-the-art SR performance on commonly used test images and imagesets. The results of CISRDCNN on real low quality web images are also very impressive, with obvious quality enhancement. Further, we explore the application of the proposed SR method in low bit-rate image coding, leading to better rate-distortion performance than JPEG.

[1]  Chunhong Pan,et al.  Fast Image Upsampling via the Displacement Field , 2014, IEEE Transactions on Image Processing.

[2]  Fei Gao,et al.  Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking , 2017, IEEE Transactions on Cybernetics.

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[8]  Moncef Gabbouj,et al.  Image Interpolation Based on Non-local Geometric Similarities and Directional Gradients , 2016, IEEE Trans. Multim..

[9]  Weifeng Liu,et al.  Multiview Canonical Correlation Analysis Networks for Remote Sensing Image Recognition , 2017, IEEE Geoscience and Remote Sensing Letters.

[10]  Lingfeng Wang,et al.  Ensemble based deep networks for image super-resolution , 2017, Pattern Recognit..

[11]  Lei Zhang,et al.  Convolutional Sparse Coding for Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Jun Yu,et al.  Multitask Autoencoder Model for Recovering Human Poses , 2018, IEEE Transactions on Industrial Electronics.

[13]  Xun Wang,et al.  Lane marking detection via deep convolutional neural network , 2017, Neurocomputing.

[14]  Xinfeng Zhang,et al.  Retrieval Compensated Group Structured Sparsity for Image Super-Resolution , 2017, IEEE Transactions on Multimedia.

[15]  Srimanta Mandal,et al.  Noise adaptive super-resolution from single image via non-local mean and sparse representation , 2017, Signal Process..

[16]  Shunsuke Ono,et al.  Optimized JPEG image decompression with super-resolution interpolation using multi-order total variation , 2013, 2013 IEEE International Conference on Image Processing.

[17]  Lei Zhang,et al.  Sparse Representation Based Image Interpolation With Nonlocal Autoregressive Modeling , 2013, IEEE Transactions on Image Processing.

[18]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[19]  Qionghai Dai,et al.  CCR: Clustering and Collaborative Representation for Fast Single Image Super-Resolution , 2016, IEEE Transactions on Multimedia.

[20]  Dacheng Tao,et al.  DehazeNet: An End-to-End System for Single Image Haze Removal , 2016, IEEE Transactions on Image Processing.

[21]  Xiangjun Zhang,et al.  Image Interpolation by Adaptive 2-D Autoregressive Modeling and Soft-Decision Estimation , 2008, IEEE Transactions on Image Processing.

[22]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[23]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[24]  Xiaohai He,et al.  Single Image Super-Resolution via Adaptive Transform-Based Nonlocal Self-Similarity Modeling and Learning-Based Gradient Regularization , 2017, IEEE Transactions on Multimedia.

[25]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[26]  Jong-Ok Kim,et al.  Joint super-resolution and compression artifact reduction based on dual-learning , 2016, 2016 Visual Communications and Image Processing (VCIP).

[27]  Truong Q. Nguyen,et al.  Single Image Super-Resolution Using Local Geometric Duality and Non-Local Similarity , 2016, IEEE Transactions on Image Processing.

[28]  Yu-Chiang Frank Wang,et al.  A Self-Learning Approach to Single Image Super-Resolution , 2013, IEEE Transactions on Multimedia.

[29]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[30]  Xiyuan Hu,et al.  Single image super-resolution in compressed domain based on field of expert prior , 2012, 2012 5th International Congress on Image and Signal Processing.

[31]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Nanning Zheng,et al.  Incorporating image priors with deep convolutional neural networks for image super-resolution , 2016, Neurocomputing.

[33]  Jun Yu,et al.  Coupled Deep Autoencoder for Single Image Super-Resolution , 2017, IEEE Transactions on Cybernetics.

[34]  Fei Zhou,et al.  Consistent Coding Scheme for Single-Image Super-Resolution Via Independent Dictionaries , 2016, IEEE Transactions on Multimedia.

[35]  Yunsong Li,et al.  Hyperspectral image super-resolution using deep convolutional neural network , 2017, Neurocomputing.

[36]  Weiguo Gong,et al.  Combining edge difference with nonlocal self-similarity constraints for single image super-resolution , 2017, Neurocomputing.

[37]  Xuelong Li,et al.  Single Image Super-Resolution With Non-Local Means and Steering Kernel Regression , 2012, IEEE Transactions on Image Processing.

[38]  Meng Wang,et al.  Multimodal Deep Autoencoder for Human Pose Recovery , 2015, IEEE Transactions on Image Processing.

[39]  曹飞龙,et al.  Image Interpolation via Low-Rank Matrix Completion and Recovery , 2015 .

[40]  Wan-Chi Siu,et al.  Fast Image Interpolation via Random Forests , 2015, IEEE Transactions on Image Processing.

[41]  Jane You,et al.  HSAE: A Hessian regularized sparse auto-encoders , 2016, Neurocomputing.

[42]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Chao Ren,et al.  Single image super resolution using local smoothness and nonlocal self-similarity priors , 2016, Signal Process. Image Commun..

[44]  Chia-Hung Yeh,et al.  Learning-Based Joint Super-Resolution and Deblocking for a Highly Compressed Image , 2015, IEEE Transactions on Multimedia.

[45]  Lei Zhang,et al.  Nonlocally Centralized Sparse Representation for Image Restoration , 2013, IEEE Transactions on Image Processing.

[46]  Michael Elad,et al.  Single Image Interpolation Via Adaptive Nonlocal Sparsity-Based Modeling , 2014, IEEE Transactions on Image Processing.

[47]  Zhiwei Xiong,et al.  Robust Web Image/Video Super-Resolution , 2010, IEEE Transactions on Image Processing.

[48]  Kai-Kuang Ma,et al.  Contrast-Guided Image Interpolation , 2013, IEEE Transactions on Image Processing.

[49]  Weifeng Liu,et al.  Canonical correlation analysis networks for two-view image recognition , 2017, Inf. Sci..

[50]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[51]  Li Shang,et al.  Modified sparse representation based image super-resolution reconstruction method , 2017, Neurocomputing.

[52]  Dacheng Tao,et al.  Deep Neural Network for Structural Prediction and Lane Detection in Traffic Scene , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[53]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[54]  Yiguang Liu,et al.  The Euclidean embedding learning based on convolutional neural network for stereo matching , 2017, Neurocomputing.

[55]  Hau-San Wong,et al.  Adaptive activation functions in convolutional neural networks , 2018, Neurocomputing.

[56]  Chih-Yuan Yang,et al.  Learning a No-Reference Quality Metric for Single-Image Super-Resolution , 2016, Comput. Vis. Image Underst..

[57]  Aline Roumy,et al.  Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding , 2012, BMVC.

[58]  Lin Li,et al.  Filtered Mapping-Based Method for Compressed Web Image Super-Resolution , 2017, IEEE Access.

[59]  Truong Q. Nguyen,et al.  Single Image Super-Resolution via Adaptive High-Dimensional Non-Local Total Variation and Adaptive Geometric Feature , 2017, IEEE Transactions on Image Processing.

[60]  Tao Li,et al.  Rotation expanded dictionary-based single image super-resolution , 2016, Neurocomputing.

[61]  Wenhan Yang,et al.  Isophote-Constrained Autoregressive Model With Adaptive Window Extension for Image Interpolation , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[62]  Wen Gao,et al.  CONCOLOR: Constrained Non-Convex Low-Rank Model for Image Deblocking , 2016, IEEE Transactions on Image Processing.

[63]  Muhammad Awais,et al.  Medical image retrieval using deep convolutional neural network , 2017, Neurocomputing.

[64]  Hongyang Chao,et al.  Building Dual-Domain Representations for Compression Artifacts Reduction , 2016, ECCV.

[65]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[66]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[67]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[68]  Quan Pan,et al.  Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[69]  Zhenyue Zhang,et al.  Locally affine patch mapping and global refinement for image super-resolution , 2011, Pattern Recognit..

[70]  Dong Xu,et al.  Example-Based Super-Resolution With Soft Information and Decision , 2013, IEEE Transactions on Multimedia.

[71]  Weifeng Liu,et al.  LMAE: A large margin Auto-Encoders for classification , 2017, Signal Process..

[72]  Mohan S. Kankanhalli,et al.  Online object tracking based on CNN with spatial-temporal saliency guided sampling , 2017, Neurocomputing.

[73]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Lei Zhang,et al.  Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization , 2010, IEEE Transactions on Image Processing.

[75]  Truong Q. Nguyen,et al.  Single Image Superresolution Based on Gradient Profile Sharpness , 2015, IEEE Transactions on Image Processing.