Attention-based Multi-Reference Learning for Image Super-Resolution

This document presents additional details on the architecture of the Super-Resolution (SR) module and on the training implementation of AMRSR (Section 1). A pseudocode of AMRSR is then proposed with a legend summarizing the notation used in the main paper (Section 2) . Ablation study results are added by training AMRSR with all the losses (visual-oriented) for different number of references and for different configurations of the attention mapping (Section 3). Another ablation study is conducted by changing the number of parts that the LR input and the reference images are divided into, with a focus also on the inference time (Section 3). More visual comparisons between AMRSR and the other approaches cited in the experimental results of the original paper are illustrated (Section 4). Further results of the comparison between AMRSR and CIMR [1] are finally presented (Section 5).

[1]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[2]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[3]  Daniel Cremers,et al.  A Super-Resolution Framework for High-Accuracy Multiview Reconstruction , 2013, International Journal of Computer Vision.

[4]  Chih-Yuan Yang,et al.  Single-Image Super-Resolution: A Benchmark , 2014, ECCV.

[5]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[7]  Chao Dong,et al.  Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Matthias Nießner,et al.  Shading-based refinement on volumetric signed distance functions , 2015, ACM Trans. Graph..

[9]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Yu Qiao,et al.  RankSRGAN: Generative Adversarial Networks With Ranker for Image Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Torsten Sattler,et al.  A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ashok Veeraraghavan,et al.  Improving resolution and depth-of-field of light field cameras using a hybrid imaging system , 2014, 2014 IEEE International Conference on Computational Photography (ICCP).

[14]  Lu Fang,et al.  Learning Cross-scale Correspondence and Patch-based Synthesis for Reference-based Super-Resolution , 2017, BMVC.

[15]  Radu Timofte,et al.  3D Appearance Super-Resolution With Deep Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Baining Guo,et al.  Learning Texture Transformer Network for Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[19]  Marc Pollefeys,et al.  Learned Multi-View Texture Super-Resolution , 2019, 2019 International Conference on 3D Vision (3DV).

[20]  Jiaya Jia,et al.  MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Jie Liu,et al.  Residual Feature Distillation Network for Lightweight Image Super-Resolution , 2020, ECCV Workshops.

[22]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Lu Fang,et al.  CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping , 2018, ECCV.

[24]  David J. Brady,et al.  CrossNet++: Cross-Scale Large-Parallax Warping for Reference-Based Super-Resolution , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Shuguang Cui,et al.  Towards Content-Independent Multi-Reference Super-Resolution: Adaptive Pattern Matching and Feature Aggregation , 2020, ECCV.

[26]  Ning Xu,et al.  Wide Activation for Efficient and Accurate Image Super-Resolution , 2018, ArXiv.

[27]  Radomír Mech,et al.  Event-Specific Image Importance , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Shenghao Yang,et al.  Survey of single image super-resolution reconstruction , 2020, IET Image Process..

[29]  ZhouQian-Yi,et al.  Color map optimization for 3D reconstruction with consumer depth cameras , 2014 .

[30]  Steven C. H. Hoi,et al.  Deep Learning for Image Super-Resolution: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Vladlen Koltun,et al.  Color map optimization for 3D reconstruction with consumer depth cameras , 2014, ACM Trans. Graph..

[32]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Hairong Qi,et al.  Image Super-Resolution by Neural Texture Transfer , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Xiaoyan Sun,et al.  Landmark Image Super-Resolution by Retrieving Web Images , 2013, IEEE Transactions on Image Processing.

[35]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[36]  Thomas S. Huang,et al.  Image Super-Resolution With Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Umesh Ghanekar,et al.  A compendious study of super-resolution techniques by single image , 2018 .

[38]  Olaf Hellwich,et al.  SyB3R: A Realistic Synthetic Benchmark for 3D Reconstruction from Images , 2016, ECCV.

[39]  O. Hellwich,et al.  SyB 3 R : A Realistic Synthetic Benchmark for 3 D Reconstruction from Images , 2016 .

[40]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[41]  In So Kweon,et al.  Robust Reference-Based Super-Resolution With Similarity-Aware Deformable Convolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Yongwoo Kim,et al.  Ultra Lightweight Image Super-Resolution with Multi-Attention Layers , 2020, ArXiv.

[43]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.