Blind Super-Resolution with Kernel-Aware Feature Refinement

Numerous learning-based super-resolution (SR) methods that assume the blur kernel is known in advance or hand-crafted perform excellently on the synthesized images. While these methods fail to produce satisfactory results when applied to real-world images. Therefore, several blind SR methods are proposed, where the blur kernel is unavailable. However, they only consider the Gaussian blur kernel, which is not enough for real-world images. To address this problem, we propose to synthesize training data with real blur kernels estimated from real sensor images. Further, we find that models trained with the data synthesized with the estimated real blur kernels still cannot perform well enough without awareness of the blur kernel. Thus, we first propose a new kernel predictor (KP) to predict the blur kernel from the input low-resolution image. Then we design the kernel-aware feature refinement (KAFR) module to utilize the predicted blur kernel to facilitate the restoration of the high-resolution image. Extensive experiments on synthesized and real-world images show that the proposed method achieves superior performance against the state-of-the-art (SOTA) methods both on effectiveness and efficiency.

[1]  Richard Hartley,et al.  Phase-Only Image Based Kernel Estimation for Single Image Blind Deblurring , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Xinfeng Zhang,et al.  Multiple Cycle-in-Cycle Generative Adversarial Networks for Unsupervised Image Super-Resolution , 2020, IEEE Transactions on Image Processing.

[3]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Aline Roumy,et al.  Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding , 2012, BMVC.

[5]  Jing Yang,et al.  To learn image super-resolution, use a GAN to learn how to do image degradation first , 2018, ECCV.

[6]  Anat Levin,et al.  Accurate Blur Models vs. Image Priors in Single Image Super-resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[8]  Michal Irani,et al.  Blind Super-Resolution Kernel Estimation using an Internal-GAN , 2019, NeurIPS.

[9]  Sergio Escalera,et al.  Convolutional Neural Network Super Resolution for Face Recognition in Surveillance Monitoring , 2016, AMDO.

[10]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  Chao Dong,et al.  Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Wenguan Wang,et al.  Cascaded Human-Object Interaction Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Siyuan Liu,et al.  Unsupervised Image Super-Resolution Using Cycle-in-Cycle Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Luc Van Gool,et al.  DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[15]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[16]  Bernhard Schölkopf,et al.  EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Wangmeng Zuo,et al.  Learning a Single Convolutional Super-Resolution Network for Multiple Degradations , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Wangmeng Zuo,et al.  Blind Super-Resolution With Iterative Kernel Correction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Ling Shao,et al.  Simultaneous Super-Resolution and Cross-Modality Synthesis of 3D Medical Images Using Weakly-Supervised Joint Convolutional Sparse Coding , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[25]  Ming-Hsuan Yang,et al.  Motion Blur Kernel Estimation via Deep Learning , 2018, IEEE Transactions on Image Processing.

[26]  Luc Van Gool,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Ling Shao,et al.  Motion-Attentive Transition for Zero-Shot Video Object Segmentation , 2020, AAAI.

[28]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[30]  Sabine Süsstrunk,et al.  Kernel Modeling Super-Resolution on Real Low-Resolution Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).