Focal Network for Image Restoration

Image restoration aims to reconstruct a sharp image from its degraded counterpart, which plays an important role in many fields. Recently, Transformer models have achieved promising performance on various image restoration tasks. However, their quadratic complexity remains an intractable issue for practical applications. The aim of this study is to develop an efficient and effective framework for image restoration. Inspired by the fact that different regions in a corrupted image always undergo degradations in various degrees, we propose to focus more on the important areas for reconstruction. To this end, we introduce a dual-domain selection mechanism to emphasize crucial information for restoration, such as edge signals and hard regions. In addition, we split high-resolution features to insert multi-scale receptive fields into the network, which improves both efficiency and performance. Finally, the proposed network, dubbed FocalNet, is built by incorporating these designs into a U-shaped backbone. Extensive experiments demonstrate that our model achieves state-of-the-art performance on ten datasets for three tasks, including single-image defocus deblurring, image dehazing, and image desnowing. Our code is available at https://github.com/c-yn/FocalNet.

[1]  Y. Tao,et al.  Dual-Domain Attention for Image Deblurring , 2023, AAAI.

[2]  Lei Zhang,et al.  Spatial-Frequency Attention for Image Denoising , 2023, ArXiv.

[3]  Chen Change Loy,et al.  Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement , 2023, ICLR.

[4]  Tao Wang,et al.  Restoring Vision in Hazy Weather with Hierarchical Contrastive Learning , 2022, Pattern Recognit..

[5]  Xiaochun Cao,et al.  Optimizing Two-Way Partial AUC With an End-to-End Framework , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Wenqi Ren,et al.  Image Dehazing Transformer with Transmission-Aware 3D Position Embedding , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Mehmet Yamaç,et al.  Motion Aware Double Attention Network for Dynamic Scene Deblurring , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Bohan Zhuang,et al.  Fast Vision Transformers with HiLo Attention , 2022, NeurIPS.

[9]  Shuicheng Yan,et al.  Inception Transformer , 2022, NeurIPS.

[10]  Jian Sun,et al.  Simple Baselines for Image Restoration , 2022, ECCV.

[11]  Jizhou Li,et al.  Learning to Deblur using Light Field Generated and Real Defocus Images , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Hujun Yin,et al.  A survey of deep learning approaches to image restoration , 2022, Neurocomputing.

[13]  Ming-Hsuan Yang,et al.  Deep Image Deblurring: A Survey , 2022, International Journal of Computer Vision.

[14]  P. Milanfar,et al.  MAXIM: Multi-Axis MLP for Image Processing , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Shuicheng Yan,et al.  DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition , 2021, ECCV.

[16]  Xinchao Wang,et al.  Shunted Self-Attention via Multi-Scale Token Aggregation , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Vishal M. Patel,et al.  TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Syed Waqas Zamir,et al.  Restormer: Efficient Transformer for High-Resolution Image Restoration , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jianwei Ding,et al.  ALL Snow Removed: Single Image Desnowing Algorithm Using Hierarchical Dual-tree Complex Wavelet Representation and Contradict Channel Loss , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Wenbin Zou,et al.  SDWNet: A Straight Dilated Network with Wavelet Transformation for image Deblurring , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[21]  Luc Van Gool,et al.  SwinIR: Image Restoration Using Swin Transformer , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[22]  Seungyong Lee,et al.  Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutions , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[23]  Sung-Jea Ko,et al.  Rethinking Coarse-to-Fine Approach in Single Image Deblurring , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Yun Liu,et al.  P2T: Pyramid Pooling Transformer for Scene Understanding , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Seungyong Lee,et al.  Iterative Filter Adaptive Network for Single Image Defocus Deblurring , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  R. Weinberg Restoration , 2021, Canadian Medical Association Journal.

[27]  Ling Shao,et al.  Multi-Stage Progressive Image Restoration , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Chang Xu,et al.  Pre-Trained Image Processing Transformer , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Jiangxin Dong,et al.  Physics-Based Feature Dehazing Networks , 2020, ECCV.

[30]  Zheng-Jun Zha,et al.  Nighttime Dehazing with a Synthetic Benchmark , 2020, ACM Multimedia.

[31]  Jingang Zhang,et al.  VDFlow: Joint Learning for Optical Flow and Video Deblurring , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[32]  Wei Su,et al.  Efficient Dynamic Scene Deblurring Using Spatially Variant Deconvolution Network With Optical Flow Guided Training , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Robby T. Tan,et al.  All in One Bad Weather Removal Using Architectural Search , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Radu Timofte,et al.  NH-HAZE: An Image Dehazing Benchmark with Non-Homogeneous Hazy and Haze-Free Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  M. S. Brown,et al.  Defocus Deblurring Using Dual-Pixel Data , 2020, ECCV.

[36]  Lei Xiang,et al.  Multi-Scale Boosted Dehazing Network With Dense Feature Fusion , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Chia-Hung Yeh,et al.  A Motion Deblur Method Based on Multi-Scale High Frequency Residual Image Learning , 2020, IEEE Access.

[38]  Changxin Gao,et al.  Dynamic Scene Deblurring by Depth Guided Model , 2020, IEEE Transactions on Image Processing.

[39]  Xiaodong Xie,et al.  FFA-Net: Feature Fusion Attention Network for Single Image Dehazing , 2019, AAAI.

[40]  Jun Chen,et al.  GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Shuicheng Yan,et al.  Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[42]  Radu Timofte,et al.  Dense-Haze: A Benchmark for Image Dehazing with Dense-Haze and Haze-Free Images , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[43]  Xiaochun Cao,et al.  Deep Video Dehazing With Semantic Segmentation , 2019, IEEE Transactions on Image Processing.

[44]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[45]  Hazim Kemal Ekenel,et al.  Cycle-Dehaze: Enhanced CycleGAN for Single Image Dehazing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[46]  Hongdong Li,et al.  Semantic Single-Image Dehazing , 2018, ArXiv.

[47]  Radu Timofte,et al.  O-HAZE: A Dehazing Benchmark with Real Hazy and Haze-Free Outdoor Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[48]  Jan Kautz,et al.  Deep Semantic Face Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  Dan Feng,et al.  Benchmarking Single-Image Dehazing and Beyond , 2017, IEEE Transactions on Image Processing.

[51]  Dongwoo Lee,et al.  Joint Blind Motion Deblurring and Depth Estimation of Light Field , 2017, ECCV.

[52]  Jizheng Xu,et al.  AOD-Net: All-in-One Dehazing Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[53]  Jenq-Neng Hwang,et al.  DesnowNet: Context-Aware Deep Network for Snow Removal , 2017, IEEE Transactions on Image Processing.

[54]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[55]  Jing Zhang,et al.  Fast Haze Removal for Nighttime Image Using Maximum Reflectance Prior , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[57]  Dacheng Tao,et al.  DehazeNet: An End-to-End System for Single Image Haze Removal , 2016, IEEE Transactions on Image Processing.

[58]  Michael S. Brown,et al.  Nighttime Haze Removal with Glow and Multiple Light Colors , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[59]  Jing Zhang,et al.  Nighttime haze removal based on a new imaging model , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[60]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[61]  Wenqi Ren,et al.  Selective Frequency Network for Image Restoration , 2023, ICLR.

[62]  Wenqi Ren,et al.  IRNeXt: Rethinking Convolutional Network Design for Image Restoration , 2023, ICML.

[63]  Tian Ye,et al.  Perceiving and Modeling Density for Image Dehazing , 2022, ECCV.

[64]  Yan Wang,et al.  Deep Residual Fourier Transformation for Single Image Deblurring , 2021, ArXiv.

[65]  Jian-Jiun Ding,et al.  JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal , 2020, ECCV.