NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results

This paper reviews the first NTIRE challenge on quality enhancement of compressed video, with focus on proposed solutions and results. In this challenge, the new Large-scale Diverse Video (LDV) dataset is employed. The challenge has three tracks. Tracks 1 and 2 aim at enhancing the videos compressed by HEVC at a fixed QP, while Track 3 is de-signed for enhancing the videos compressed by x265 at a fixed bit-rate. Besides, the quality enhancement of Tracks 1 and 3 targets at improving the fidelity (PSNR), and Track 2 targets at enhancing the perceptual quality. The three tracks totally attract 482 registrations. In the test phase, 12 teams, 8 teams and 11 teams submitted the final results of Tracks 1, 2 and 3, respectively. The proposed methods and solutions gauge the state-of-the-art of video quality enhancement. The homepage of the challenge: https://github.com/RenYang-home/NTIRE21_VEnh

Radu Timofte | Mengxi Guo | Zhan Ma | Linjie Zhou | Wenming Yang | Yibin Huang | Xiaopeng Sun | Shangchen Zhou | Sijung Kim | Pablo Navarrete Michelini | Chen Change Loy | Matteo Maggioni | Kelvin C. K. Chan | Iaroslav Koshelev | Yiting Liao | Dongliang He | Ren Yang | Shuigeng Zhou | Thomas Tanay | Kelvin C.K. Chan | Xiangyu Xu | Andrey Somov | Pavel Ostyakov | Chong Mou | Kangdi Shi | Jun Chen | Xiaochao Qu | Gen Zhan | Yan Liu | Jing Liu | Xin Li | Wei Gao | Xueyi Zou | Fu Li | Fenglong Song | Minyi Zhao | Yi Xu | Xinjian Zhang | Fanglong Liu | He Zheng | Lielin Jiang | Qi Zhang | Qingqing Dang | Zhognqian Fu | Shuai Xiao | Cheng li | Wentao Chao | Qiang Guo | Jiang Li | Dewang Hou | Jiayu Yang | Lyn Jiang | Di You | Zhenyu Zhang | Jia Hao | Shijie Zhao | Yuanzhi Zhang | Qing Wang | Junlin Li | Ming Lu | Hai Wang | Yiyun Chen | Jingyu Guo | Liliang Zhang | Syehoon Oh | Yucong Wang | Minjie Cai | Wei Hao | Liangyan Li | Wang Liu | Xiaoyu Zhang | Sixin Lin | Ru Wang | Qingqing Dang | Linjie Zhou | R. Timofte | Xiangyu Xu | Yan Liu | Cheng Li | Zhan Ma | Ren Yang | Dongliang He | Fu Li | Yibin Huang | T. Tanay | Wenming Yang | Ming-Tse Lu | Qi Zhang | Qing Wang | Pavel Ostyakov | Xiaopeng Sun | X. Zou | Junlin Li | Xiaochao Qu | Matteo Maggioni | Qiang Guo | Shuigeng Zhou | Chong Mou | Yiting Liao | Lili Zhang | Wei-Nan Gao | Junfeng Chen | Mengxi Guo | Minyi Zhao | Gen Zhan | Min Cai | Shuai Xiao | Fenglong Song | Jiayu Yang | Shangchen Zhou | Yuanzhi Zhang | W. Hao | Jingyu Guo | Fanglong Liu | He Zheng | Shijie Zhao | Iaroslav Koshelev | Liang Li | Jing Liu | Xinjian Zhang | Jiaqi Hao | Zhen-ying Zhang | Hai Wang | Ru Wang | Kangdi Shi | Sixin Lin | A. Somov | Xin Li | Sijung Kim | Lyn Jiang | Xiaoyu Zhang | Yucong Wang | Dewang Hou | Yi Xu | Lielin Jiang | Zhognqian Fu | Wentao Chao | Jiang Li | Di You | Yiyun Chen | Syehoon Oh | Wang-gen Liu

[1]  Irwin Edward Sobel,et al.  Camera Models and Machine Perception , 1970 .

[2]  Radu Timofte,et al.  NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Dataset and Study , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[4]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[5]  Radu Timofte,et al.  NTIRE 2020 Challenge on Image and Video Deblurring , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Michel Barlaud,et al.  Two deterministic half-quadratic regularization algorithms for computed imaging , 1994, Proceedings of 1st International Conference on Image Processing.

[7]  Xiang Bai,et al.  Scene Text Image Super-Resolution in the Wild , 2020, ECCV.

[8]  Longwen Gao,et al.  Boosting the Performance of Video Compression Artifact Reduction with Reference Frame Proposals and Frequency Domain Information , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  F. Bossen,et al.  Common test conditions and software reference configurations , 2010 .

[10]  Stephen Lin,et al.  Deformable ConvNets V2: More Deformable, Better Results , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Guang Deng,et al.  A Generalized Unsharp Masking Algorithm , 2011, IEEE Transactions on Image Processing.

[12]  Tingting Wang,et al.  A Novel Deep Learning-Based Method of Improving Coding Efficiency from the Decoder-End for HEVC , 2017, 2017 Data Compression Conference (DCC).

[13]  Wei Zhang,et al.  The SJTU 4K video sequence dataset , 2013, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).

[14]  Radu Timofte,et al.  NTIRE 2021 Depth Guided Image Relighting Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Arthur Gretton,et al.  Demystifying MMD GANs , 2018, ICLR.

[16]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[17]  Kun Gao,et al.  NTIRE 2021 Learning the Super-Resolution Space Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Dmytro Mishkin,et al.  Kornia: an Open Source Differentiable Computer Vision Library for PyTorch , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Fahad Shahbaz Khan,et al.  NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Xiaoyan Sun,et al.  Quality-Gated Convolutional Lstm for Enhancing Compressed Video , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[22]  Radu Timofte,et al.  DIV8K: DIVerse 8K Resolution Image Dataset , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[23]  Martin Danelljan,et al.  NTIRE 2020 Challenge on Video Quality Mapping: Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Mingkui Tan,et al.  Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Hua Wang,et al.  Deformable Non-Local Network for Video Super-Resolution , 2019, IEEE Access.

[27]  Qilong Wang,et al.  ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Luc Van Gool,et al.  Seven Ways to Improve Example-Based Single Image Super Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Jenq-Neng Hwang,et al.  NTIRE 2021 Multi-modal Aerial View Object Classification Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[30]  Zulin Wang,et al.  Enhancing Quality for HEVC Compressed Videos , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Li Wang,et al.  Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement , 2020, AAAI.

[32]  Chen Change Loy,et al.  BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Radu Timofte,et al.  NTIRE 2021 Challenge on Video Super-Resolution , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Zulin Wang,et al.  Decoder-side HEVC quality enhancement with scalable convolutional neural network , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[35]  Radu Timofte,et al.  NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[36]  Shaoshi Yang,et al.  A recurrent video quality enhancement framework with multi-granularity frame-fusion and frame difference based attention , 2020, Neurocomputing.

[37]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[38]  Dongliang He,et al.  Adaptive Spatial-Temporal Fusion of Multi-Objective Networks for Compressed Video Perceptual Enhancement , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Radu Timofte,et al.  NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[41]  Jonathan T. Barron,et al.  Burst photography for high dynamic range and low-light imaging on mobile cameras , 2016, ACM Trans. Graph..

[42]  Mai Xu,et al.  Early Exit Or Not: Resource-Efficient Blind Quality Enhancement for Compressed Images , 2020, ECCV.

[43]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[44]  Mai Xu,et al.  Multi-level Wavelet-based Generative Adversarial Network for Perceptual Quality Enhancement of Compressed Video , 2020, ECCV.

[45]  Tie Liu,et al.  MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Radu Timofte,et al.  NTIRE 2021 Challenge on Burst Super-Resolution: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[47]  Angel Domingo Sappa,et al.  MPRNet: Multi-Path Residual Network for Lightweight Image Super Resolution , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[48]  Zulin Wang,et al.  Multi-frame Quality Enhancement for Compressed Video , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Ling Shao,et al.  NTIRE 2021 NonHomogeneous Dehazing Challenge Report , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[50]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[51]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[52]  Dan Zhu,et al.  Multi–Grid Back–Projection Networks , 2021, IEEE Journal of Selected Topics in Signal Processing.

[53]  Radu Timofte,et al.  NTIRE 2021 Challenge on Perceptual Image Quality Assessment , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  Radu Timofte,et al.  NTIRE 2021 Challenge on Image Deblurring , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[55]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Chen Change Loy,et al.  EDVR: Video Restoration With Enhanced Deformable Convolutional Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[57]  Julie Delon,et al.  FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[59]  Gregory Shakhnarovich,et al.  Recurrent Back-Projection Network for Video Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Alexia Jolicoeur-Martineau,et al.  The relativistic discriminator: a key element missing from standard GAN , 2018, ICLR.

[61]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[62]  K. Simonyan,et al.  High-Performance Large-Scale Image Recognition Without Normalization , 2021, ICML.

[63]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[64]  Ning Xu,et al.  Wide Activation for Efficient and Accurate Image Super-Resolution , 2018, ArXiv.

[65]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[66]  Dong Xu,et al.  Deep Kalman Filtering Network for Video Compression Artifact Reduction , 2018, ECCV.

[67]  Chen Change Loy,et al.  Understanding Deformable Alignment in Video Super-Resolution , 2020, AAAI.

[68]  L. Gool,et al.  Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Yi Xu,et al.  Non-Local ConvLSTM for Video Compression Artifact Reduction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[70]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[71]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[72]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.