Stereo Cross Global Learnable Attention Module for Stereo Image Super-Resolution

Stereo super-resolution is a technique that utilizes corresponding information from multiple viewpoints to enhance the texture of low-resolution images. In recent years, numerous impressive works have advocated attention mechanisms based on epipolar constraints to boost the performance of stereo super-resolution. However, techniques that exclusively depend on epipolar constraint attention are insufficient to recover realistic and natural textures for heavily corrupted low-resolution images. We noticed that global self-similarity features within the image and across the views can proficiently fix the texture details of low-resolution images that are severely damaged. Therefore, in the current paper, we propose a stereo cross global learnable attention module (SCGLAM), aiming to improve the performance of stereo super-resolution. The experimental outcomes show that our approach outperforms others when dealing with heavily damaged low-resolution images. The relevant code is made available on this link as open source.

[1]  Thomas Bo Schön,et al.  NTIRE 2023 Challenge on Stereo Image Super-Resolution: Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  T. Zeng,et al.  PFT-SSR: Parallax Fusion Transformer for Stereo Image Super-Resolution , 2023, ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Jianmei Su,et al.  Global Learnable Attention for Single Image Super-Resolution , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Xiuzhuang Zhou,et al.  SwiniPASSR: Swin Transformer based Parallax Attention Network for Stereo Image Super-Resolution , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  Xiaojie Chu,et al.  NAFSSR: Stereo Image Super-Resolution Using NAFNet , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Zachary Teed,et al.  RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching , 2021, 2021 International Conference on 3D Vision (3DV).

[7]  T. Zeng,et al.  Transformer for Single Image Super-Resolution , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Luc Van Gool,et al.  SwinIR: Image Restoration Using Swin Transformer , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[9]  Ying Shan,et al.  Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[10]  Qiaosi Yi,et al.  Feedback Network for Mutually Boosted Stereo Image Super-Resolution and Disparity Estimation , 2021, ACM Multimedia.

[11]  Yuchen Fan,et al.  Image Super-Resolution with Non-Local Sparse Attention , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Luc Van Gool,et al.  Designing a Practical Degradation Model for Deep Blind Image Super-Resolution , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Yulan Guo,et al.  Symmetric Parallax Attention for Stereo Image Super-Resolution , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Weimin Tan,et al.  Disparity-Aware Domain Adaptation in Stereo Image Restoration , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Thomas S. Huang,et al.  Image Super-Resolution With Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Kwanghoon Sohn,et al.  Stereoscopic Image Super-Resolution with Stereo Consistent Feature , 2020, AAAI.

[17]  Weidong Sheng,et al.  A Stereo Attention Module for Stereo Image Super-Resolution , 2020, IEEE Signal Processing Letters.

[18]  Mohamed El-Sharkawy,et al.  Thin MobileNet: An Enhanced MobileNet Architecture , 2019, 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON).

[19]  Radu Timofte,et al.  Unsupervised Learning for Real-World Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[20]  Yu Qiao,et al.  RankSRGAN: Generative Adversarial Networks With Ranker for Image Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Shu-Tao Xia,et al.  Second-Order Attention Network for Single Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Wei An,et al.  Flickr1024: A Large-Scale Dataset for Stereo Image Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[23]  Wei An,et al.  Learning Parallax Attention for Stereo Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Hairong Qi,et al.  Image Super-Resolution by Neural Texture Transfer , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[26]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[27]  Seung-Hwan Baek,et al.  Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[35]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[36]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[39]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[41]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Raanan Fattal,et al.  Image and video upscaling from local self-examples , 2011, TOGS.

[43]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[44]  Mehran Ebrahimi,et al.  Solving the Inverse Problem of Image Zooming Using "Self-Examples" , 2007, ICIAR.

[45]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).