Geometry Enhanced Reference-based Image Super-resolution

With the prevalence of smartphones equipped with a multi-camera system comprising multiple cameras with different field-of-view (FoVs), images captured by two or three cameras now share a portion of the FoV that are compatible with reference-based super-resolution (RefSR). In this work, we propose a novel RefSR model that utilizes geometric matching methods to enhance its performance in two aspects. First, we integrate geometric matching maps to improve feature fusion. Second, we train the matching modules equipped in the RefSR models under the supervision of accurate geometric matching maps to increase their robustness. Our experimental results demonstrate the effectiveness and state-of-the-art performance of the proposed method.

[1]  Siheng Chen,et al.  Task Decoupled Framework for Reference-based Super-Resolution , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Sunghyun Cho,et al.  Reference-based Video Super-Resolution Using Multi-Camera Video Triplets , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  L. Gool,et al.  Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Luc Van Gool,et al.  PDC-Net+: Enhanced Probabilistic Dense Correspondence Network , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Qifeng Chen,et al.  Dual-Camera Super-Resolution with Aligned Attention Modules , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Chen Change Loy,et al.  Robust Reference-based Super-Resolution via C2-Matching , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Luc Van Gool,et al.  Warp Consistency for Unsupervised Learning of Dense Correspondences , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Luc Van Gool,et al.  GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network , 2020, NeurIPS.

[9]  Mingjie Sun,et al.  Feature Representation Matters: End-to-End Learning for Reference-Based Image Super-Resolution , 2020, ECCV.

[10]  Jean Ponce,et al.  Learning to Compose Hypercolumns for Visual Correspondence , 2020, ECCV.

[11]  Baining Guo,et al.  Learning Texture Transformer Network for Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Jia Deng,et al.  RAFT: Recurrent All-Pairs Field Transforms for Optical Flow , 2020, ECCV.

[13]  Martin Danelljan,et al.  GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Josef Sivic,et al.  Convolutional Neural Network Architecture for Geometric Matching , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Torsten Sattler,et al.  D2-Net: A Trainable CNN for Joint Description and Detection of Local Features , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Seungryong Kim,et al.  Semantic Attribute Matching Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Hairong Qi,et al.  Image Super-Resolution by Neural Texture Transfer , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Tomás Pajdla,et al.  Neighbourhood Consensus Networks , 2018, NeurIPS.

[19]  Torsten Sattler,et al.  DGC-Net: Dense Geometric Correspondence Network , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Jan Kautz,et al.  Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Bohyung Han,et al.  Attentive Semantic Alignment with Offset-Aware Correlation Kernels , 2018, ECCV.

[22]  Lu Fang,et al.  CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping , 2018, ECCV.

[23]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[24]  Xiaoou Tang,et al.  LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Zhengqi Li,et al.  MegaDepth: Learning Single-View Depth Prediction from Internet Photos , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Lihi Zelnik-Manor,et al.  The Contextual Loss for Image Transformation with Non-Aligned Data , 2018, ECCV.

[27]  Josef Sivic,et al.  End-to-End Weakly-Supervised Semantic Alignment , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Deqing Sun,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[33]  C. Strickland,et al.  Correspondence , 1932, The Indian medical gazette.

[34]  Xiang Li,et al.  Arbicon-Net: Arbitrary Continuous Geometric Transformation Networks for Image Registration , 2019, NeurIPS.

[35]  Nanning Zheng,et al.  Pattern Recognition and Computer Vision , 2018, Lecture Notes in Computer Science.

[36]  Lu Fang,et al.  Learning Cross-scale Correspondence and Patch-based Synthesis for Reference-based Super-Resolution , 2017, BMVC.