NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results

This paper reviews the NTIRE 2022 challenge on efficient single image super-resolution with focus on the proposed solutions and results. The task of the challenge was to super-resolve an input image with a magnification factor of ×4 based on pairs of low and corresponding high resolution images. The aim was to design a network for single image super-resolution that achieved improvement of efficiency measured according to several metrics including runtime, parameters, FLOPs, activations, and memory consumption while at least maintaining the PSNR of 29.00dB on DIV2K validation set. IMDN is set as the baseline for efficiency measurement. The challenge had 3 tracks including the main track (runtime), sub-track one (model complexity), and sub-track two (overall performance). In the main track, the practical runtime performance of the submissions was evaluated. The rank of the teams were determined directly by the absolute value of the average runtime on the validation set and test set. In sub-track one, the number of parameters and FLOPs were considered. And the individual rankings of the two metrics were summed up to determine a final ranking in this track. In sub-track two, all of the five metrics mentioned in the description of the challenge including runtime, parameter count, FLOPs, activations, and memory consumption were considered. Similar to sub-track one, the rankings of five metrics were summed up to determine a final ranking. The challenge had 303 registered participants, and 43 teams made valid submissions. They gauge the state-of-the-art in efficient single image super-resolution.

Syed Waqas Zamir | L. Gool | Hao Li | Chao Dong | Fei Wang | F. Khan | Xiansheng Hua | Haoqiang Fan | R. Timofte | Ding Liu | K. Zhang | Jinshan Pan | Yufei Wang | Shuaicheng Liu | Peiran Ren | Yingqi Liu | Y. Cai | Munawar Hayat | Jie Tang | J. Liu | Donghao Luo | Chengjie Wang | Jun Chen | Yan Wang | Jun Liu | Xuansong Xie | Aditya Arora | Youwei Li | Shen Cheng | Yuan Xie | Chengyi Xiong | Kun Zeng | Yi Zhu | Huan Huang | Jin-Tao Fang | Dan Ning | L. Zhang | Yuncheng Zhang | Keyan Wang | Wenxue Guan | Wei Xiong | Zhi-Hao Yang | Tao Yang | Xiangyu Chen | Wenbin Zou | Salman Khan | Long Sun | Ming-xing Li | Gang Wu | Rui Wen | Meiguang Jin | Zhizhong Zhang | Xiangzhen Kong | Jing Tang | Yawei Li | Haoming Cai | F. Kong | Songwei Liu | Zheyuan Wang | Deng-Guang Zhou | Yuan Zhang | Xin Li | Jie Liu | Xinyu Chen | Chao Chen | Yucong Wang | W. Zheng | Yanbo Wang | Huan Liu | Lei Sun | Shi-Cai Huang | Yuanjiao Qiao | Zheng Hui | Xiaozhong Ji | Zhi-hong Wen | Han-Yuan Lin | Zimo Huang | Zongcai Du | Chenhui Zhou | Jingyi Chen | Qingrui Han | Zheyuan Li | Zhikai Zong | Xiaoxiao Liu | Chuming Lin | Ying Tai | Ziwei Luo | Lei Yu | Qi Wu1 | Jian Sun | Yuanfei Huang | Jing Liu | Xinjian Zhang | L. Long | Gen Li | Zuo-yuan Cao | Panaetov Alexander | Mi Cai | Li Wang | Lu Tian | H. Ma | Weiran Wang | Honglei Lu | Shi Chen | Yu-Hsuan Miao | Mustafa Ayazouglu | Tian-Chun Ye | Han Huang | Z. Peng | Hao-Wen Li | Sheng-rong Gong | Wanjun Wang | Jin-shan Pan | Yucong Wang | Yuanfan Zhang | Salman H. Khan

[1]  M. S. Brown,et al.  NTIRE 2022 Challenge on Night Photography Rendering , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Xin Tao,et al.  NTIRE 2022 Image Inpainting Challenge: Report , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  R. Timofte,et al.  NTIRE 2022 Challenge on Learning the Super-Resolution Space , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Amir S. Bernat,et al.  NTIRE 2022 Spectral Demosaicing Challenge and Data Set , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  Jian Sun,et al.  NTIRE 2022 Burst Super-Resolution Challenge , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Amir S. Bernat,et al.  NTIRE 2022 Spectral Recovery Challenge and Data Set , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Jimmy S. J. Ren,et al.  NTIRE 2022 Challenge on Perceptual Image Quality Assessment , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  An Gia Vien,et al.  NTIRE 2022 Challenge on High Dynamic Range Imaging: Methods and Results , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  L. Gool,et al.  Revisiting Random Channel Pruning for Neural Network Compression , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Thuong Nguyen Canh,et al.  NTIRE 2022 Challenge on Super-Resolution and Quality Enhancement of Compressed Video: Dataset, Methods and Results , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  R. Timofte,et al.  NTIRE 2022 Challenge on Stereo Image Super-Resolution: Methods and Results , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Chengrou Lu,et al.  Visual attention network , 2022, Computational Visual Media.

[13]  L. Abrahamyan,et al.  Gradient Variance Loss for Structure-Enhanced Image Super-Resolution , 2022, ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Luc Van Gool,et al.  Revisiting RCAN: Improved Training for Image Super-Resolution , 2022, ArXiv.

[15]  Yali Wang,et al.  UniFormer: Unifying Convolution and Self-attention for Visual Recognition , 2022, ArXiv.

[16]  Syed Waqas Zamir,et al.  Restormer: Efficient Transformer for High-Resolution Image Restoration , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Ramon Matas Navarro,et al.  Collapsible Linear Blocks for Super-Efficient Super Resolution , 2021, MLSys.

[18]  Lei Zhang,et al.  Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices , 2021, ACM Multimedia.

[19]  Luc Van Gool,et al.  SwinIR: Image Restoration Using Swin Transformer , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[20]  Sung-Jea Ko,et al.  Rethinking Coarse-to-Fine Approach in Single Image Deblurring , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Mustafa Ayazoglu,et al.  Extremely Lightweight Quantization Robust Real-Time Single-Image Super Resolution for Mobile Devices , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Guangming Shi,et al.  Lightweight Image Super-Resolution with Hierarchical and Differentiable Neural Architecture Search , 2021, ArXiv.

[23]  Luc Van Gool,et al.  LocalViT: Bringing Locality to Vision Transformers , 2021, ArXiv.

[24]  Pei Wang,et al.  Rethinking and Improving the Robustness of Image Style Transfer , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Guiguang Ding,et al.  Diverse Branch Block: Building a Convolution as an Inception-like Unit , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Chunhua Shen,et al.  Generic Perceptual Loss for Modeling Structured Output Dependencies , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Chengyi Xiong,et al.  Attention augmented multi-scale network for single image super-resolution , 2021, Appl. Intell..

[28]  Ningning Ma,et al.  RepVGG: Making VGG-style ConvNets Great Again , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[30]  Luc Van Gool,et al.  Neural Architecture Search as Sparse Supernet , 2020, AAAI.

[31]  L. Gool,et al.  The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Y. Fu,et al.  Aligned Structured Sparsity Learning for Efficient Image Super-Resolution , 2021, NeurIPS.

[33]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Yu Qiao,et al.  Efficient Image Super-Resolution Using Pixel Attention , 2020, ECCV Workshops.

[35]  Jie Liu,et al.  Residual Feature Distillation Network for Lightweight Image Super-Resolution , 2020, ECCV Workshops.

[36]  Wei Wei,et al.  AIM 2020 Challenge on Efficient Super-Resolution: Methods and Results , 2020, ECCV Workshops.

[37]  Yongwoo Kim,et al.  Multi-attention Based Ultra Lightweight Image Super-Resolution , 2020, ECCV Workshops.

[38]  Xiaochun Cao,et al.  Correction to: Single Image Super-Resolution via a Holistic Attention Network , 2020, ECCV.

[39]  Amos Storkey,et al.  Optimizing Grouped Convolutions on Edge Devices , 2020, 2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP).

[40]  Jie Tang,et al.  Residual Feature Aggregation Network for Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Luc Van Gool,et al.  DHP: Differentiable Meta Pruning via HyperNetworks , 2020, ECCV.

[42]  Luc Van Gool,et al.  Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Sparsh Mittal,et al.  ULSAM: Ultra-Lightweight Subspace Attention Module for Compact Convolutional Neural Networks , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[44]  Chang Xu,et al.  Efficient Residual Dense Block Search for Image Super-Resolution , 2019, AAAI.

[45]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Yanyun Qu,et al.  LatticeNet: Towards Lightweight Image Super-Resolution with Lattice Block , 2020, ECCV.

[47]  Radu Timofte,et al.  AIM 2019 Challenge on Constrained Super-Resolution: Methods and Results , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[48]  Luc Van Gool,et al.  Fast Image Restoration With Multi-Bin Trainable Linear Units , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  Xinbo Gao,et al.  Lightweight Image Super-Resolution with Information Multi-distillation Network , 2019, ACM Multimedia.

[50]  Luc Van Gool,et al.  Learning Filter Basis for Convolutional Neural Network Compression , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[51]  Jungong Han,et al.  ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[52]  Greg Mori,et al.  Similarity-Preserving Knowledge Distillation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[53]  Lei Zhang,et al.  Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Xiangyu Zhang,et al.  MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[55]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[56]  Luc Van Gool,et al.  CARN: Convolutional Anchored Regression Network for Fast and Accurate Single Image Super-Resolution , 2018, ECCV Workshops.

[57]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[58]  Ning Xu,et al.  Wide Activation for Efficient and Accurate Image Super-Resolution , 2018, ArXiv.

[59]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[60]  Yi Yang,et al.  Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks , 2018, IJCAI.

[61]  Xinbo Gao,et al.  Fast and Accurate Single Image Super-Resolution via Information Distillation Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[62]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[63]  Song Han,et al.  AMC: AutoML for Model Compression and Acceleration on Mobile Devices , 2018, ECCV.

[64]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[65]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[66]  Xiangyu Zhang,et al.  ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[67]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[68]  Timo Aila,et al.  Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder , 2017, ACM Trans. Graph..

[69]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[70]  Thomas A. Funkhouser,et al.  Dilated Residual Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[72]  Peyman Milanfar,et al.  RAISR: Rapid and Accurate Image Super Resolution , 2016, IEEE Transactions on Computational Imaging.

[73]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[74]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[77]  Jian Sun,et al.  Accelerating Very Deep Convolutional Networks for Classification and Detection , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[79]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[80]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[81]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[82]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[83]  Andrew Zisserman,et al.  Speeding up Convolutional Neural Networks with Low Rank Expansions , 2014, BMVC.

[84]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[85]  Michael Elad,et al.  Generalizing the Nonlocal-Means to Super-Resolution Reconstruction , 2009, IEEE Transactions on Image Processing.

[86]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[87]  Andrew Zisserman,et al.  Computer vision applied to super resolution , 2003, IEEE Signal Process. Mag..

[88]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.