NTIRE 2023 Quality Assessment of Video Enhancement Challenge

This paper reports on the NTIRE 2023 Quality Assessment of Video Enhancement Challenge, which will be held in conjunction with the New Trends in Image Restoration and Enhancement Workshop (NTIRE) at CVPR 2023. This challenge is to address a major challenge in the field of video processing, namely, video quality assessment (VQA) for enhanced videos. The challenge uses the VQA Dataset for Perceptual Video Enhancement (VDPVE), which has a total of 1211 enhanced videos, including 600 videos with color, brightness, and contrast enhancements, 310 videos with deblurring, and 301 deshaked videos. The challenge has a total of 167 registered participants. 61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions. A total of 176 submissions were submitted by 37 participating teams during the final testing phase. Finally, 19 participating teams submitted their models and fact sheets, and detailed the methods they used. Some methods have achieved better results than baseline methods, and the winning methods have demonstrated superior prediction performance.

[1]  Guangtao Zhai,et al.  Image Quality Score Distribution Prediction via Alpha Stable Model , 2023, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  L. Gool,et al.  NTIRE 2023 Challenge on Efficient Super-Resolution: Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Marcos V. Conde,et al.  Lens-to-Lens Bokeh Effect Transformation. NTIRE 2023 Challenge Report , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Feng Zhang,et al.  Efficient Deep Models for Real-Time 4K Image Super-Resolution. NTIRE 2023 Benchmark and Report , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  Ying Chen,et al.  Video Quality Assessment Based on Swin Transformer with Spatio-Temporal Feature Fusion and Data Augmentation , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Sunder Ali Khowaja,et al.  NTIRE 2023 Challenge on Image Super-Resolution (×4): Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Junshi Huang,et al.  NTIRE 2023 Image Shadow Removal Challenge Report , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Syed Waqas Zamir,et al.  NTIRE 2023 Challenge on Image Denoising: Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  R. Timofte,et al.  NTIRE 2023 Video Colorization Challenge , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[10]  R. Schettini,et al.  Quality assessment of enhanced videos guided by aesthetics and technical quality attributes , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  Pierluigi Zama Ramirez,et al.  NTIRE 2023 Challenge on HR Depth from Images of Specular and Transparent Surfaces , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Thomas Bo Schön,et al.  NTIRE 2023 Challenge on Stereo Image Super-Resolution: Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  Marcos V. Conde,et al.  NTIRE 2023 Challenge on Night Photography Rendering , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Jun Steed Huang,et al.  NTIRE 2023 Challenge on 360° Omnidirectional Image and Video Super-Resolution: Datasets, Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Jing-Kai Lou,et al.  SB-VQA: A Stack-Based Video Quality Assessment Framework for Video Enhancement , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[16]  R. Timofte,et al.  NTIRE 2023 Challenge on Light Field Image Super-Resolution: Dataset, Methods and Results , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Kai Zhao,et al.  Zoom-VQA: Patches, Frames and Clips Integration for Video Quality Assessment , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  Wei Sun,et al.  VDPVE: VQA Dataset for Perceptual Video Enhancement , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Wei Sun,et al.  Attention-Guided Neural Networks for Full-Reference and No-Reference Audio-Visual Quality Assessment , 2023, IEEE Transactions on Image Processing.

[20]  Xinlei Chen,et al.  ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  D. Vatolin,et al.  Video compression dataset and benchmark of learning-based video-quality metrics , 2022, NeurIPS.

[22]  Devansh Jain,et al.  GlobalFlowNet: Video Stabilization using Deep Distilled Global Motion Estimates , 2022, 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[23]  Wei Sun,et al.  Surveillance Video Quality Assessment Based on Quality Related Retraining , 2022, 2022 IEEE International Conference on Image Processing (ICIP).

[24]  Qiong Yan,et al.  Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment , 2022, ArXiv.

[25]  Guangtao Zhai,et al.  Image Quality Assessment: From Mean Opinion Score to Opinion Score Distribution , 2022, ACM Multimedia.

[26]  Qiong Yan,et al.  FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling , 2022, ECCV.

[27]  Wei Sun,et al.  Deep Neural Network for Blind Visual Quality Assessment of 4K Content , 2022, IEEE Transactions on Broadcasting.

[28]  Guangming Shi,et al.  Spatiotemporal Representation Learning for Blind Video Quality Assessment , 2022, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Wei Sun,et al.  A Deep Learning based No-reference Quality Assessment Model for UGC Videos , 2022, ACM Multimedia.

[30]  Shu Shi,et al.  MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[31]  Trevor Darrell,et al.  A ConvNet for the 2020s , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  L. Gool,et al.  Flow-Guided Sparse Transformer for Video Deblurring , 2022, ICML.

[33]  Richang Hong,et al.  Deep Color Consistent Network for Low-Light Image Enhancement , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Ming-Hsuan Yang,et al.  Video Frame Interpolation Transformer , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Li Dong,et al.  Swin Transformer V2: Scaling Up Capacity and Resolution , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Shen Zheng,et al.  Semantic-Guided Zero-Shot Learning for Low-Light Image/Video Enhancement , 2021, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW).

[37]  Xiongkuo Min,et al.  Attention Based Network For No-Reference UGC Video Quality Assessment , 2021, 2021 IEEE International Conference on Image Processing (ICIP).

[38]  Guangming Shi,et al.  Video Quality Assessment With Serial Dependence Modeling , 2021, IEEE Transactions on Multimedia.

[39]  Stephen Lin,et al.  Video Swin Transformer , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Xiongkuo Min,et al.  Deep Learning Based Full-Reference and No-Reference Quality Assessment Models for Compressed UGC Videos , 2021, 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[41]  Shaodi You,et al.  Learning Temporal Consistency for Low Light Video Enhancement from Single Images , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Shangchen Zhou,et al.  BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Quoc V. Le,et al.  EfficientNetV2: Smaller Models and Faster Training , 2021, ICML.

[44]  Raimondo Schettini,et al.  An Efficient Method for No-Reference Video Quality Assessment , 2021, J. Imaging.

[45]  Jun Chen,et al.  Learning for Unconstrained Space-Time Video Super-Resolution , 2021, IEEE Transactions on Broadcasting.

[46]  Alan C. Bovik,et al.  RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content , 2021, IEEE Open Journal of Signal Processing.

[47]  Alexandros Stergiou,et al.  Refining activation downsampling with SoftPool , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Alan Bovik University of Texas at Austin,et al.  Patch-VQ: ‘Patching Up’ the Video Quality Problem , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Tingting Jiang,et al.  Unified Quality Assessment of in-the-Wild Videos with Mixed Datasets Training , 2020, Int. J. Comput. Vis..

[50]  Jun Chen,et al.  Exploit Camera Raw Data for Video Super- Resolution via Hidden Markov Model Inference , 2020, IEEE Transactions on Image Processing.

[51]  Haoyu Chen,et al.  PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration , 2020, ECCV.

[52]  Kede Ma,et al.  Perceptual Quality Assessment of Smartphone Photography , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Ravi Ramamoorthi,et al.  Learning Video Stabilization Using Optical Flow , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Alan C. Bovik,et al.  UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content , 2020, IEEE Transactions on Image Processing.

[55]  Jongmin Park,et al.  NTIRE 2020 Challenge on NonHomogeneous Dehazing , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[56]  Guangtao Zhai,et al.  Perceptual image quality assessment: a survey , 2020, Science China Information Sciences.

[57]  Norimichi Ukita,et al.  Space-Time-Aware Multi-Resolution Video Enhancement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Minda Zhao,et al.  PWStableNet: Learning Pixel-Wise Warping Maps for Video Stabilization , 2020, IEEE Transactions on Image Processing.

[59]  Yang Zhou,et al.  End-To-End Trainable Video Super-Resolution Based on a New Mechanism for Implicit Motion Estimation and Compensation , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[60]  Praful Gupta,et al.  From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Dietmar Saupe,et al.  KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment , 2019, IEEE Transactions on Image Processing.

[62]  In So Kweon,et al.  Deep Iterative Frame Interpolation for Full-frame Video Stabilization , 2019, ACM Trans. Graph..

[63]  Zhangyang Wang,et al.  DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[64]  Ming Jiang,et al.  Quality Assessment of In-the-Wild Videos , 2019, ACM Multimedia.

[65]  Jari Korhonen,et al.  Two-Level Approach for No-Reference Consumer Video Quality Assessment , 2019, IEEE Transactions on Image Processing.

[66]  Balu Adsumilli,et al.  YouTube UGC Dataset for Video Compression Research , 2019, 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP).

[67]  Jitendra Malik,et al.  SlowFast Networks for Video Recognition , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[68]  Zhengfang Duanmu,et al.  End-to-End Blind Quality Assessment of Compressed Videos Using Deep Neural Networks , 2018, ACM Multimedia.

[69]  Jiying Zhao,et al.  Robust Multi-Frame Super-Resolution Based on Spatially Weighted Half-Quadratic Estimation and Adaptive BTV Regularization , 2018, IEEE Transactions on Image Processing.

[70]  Yutao Liu,et al.  Blind Image Quality Estimation via Distortion Aggravation , 2018, IEEE Transactions on Broadcasting.

[71]  Alan Conrad Bovik,et al.  Large-Scale Study of Perceptual Video Quality , 2018, IEEE Transactions on Image Processing.

[72]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[73]  Dietmar Saupe,et al.  The Konstanz natural video database (KoNViD-1k) , 2017, 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX).

[74]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Fabio Viola,et al.  The Kinetics Human Action Video Dataset , 2017, ArXiv.

[76]  Ping Wang,et al.  MCL-JCV: A JND-based H.264/AVC video quality assessment dataset , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[77]  Kai Zeng,et al.  Objective quality assessment of tone-mapped videos , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[78]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[80]  Bhupendra Gupta,et al.  Minimum mean brightness error contrast enhancement of color images using adaptive gamma correction with color preserving framework , 2016 .

[81]  Alan C. Bovik,et al.  Massive Online Crowdsourced Study of Subjective and Objective Picture Quality , 2015, IEEE Transactions on Image Processing.

[82]  Manish Narwaria,et al.  Study of high dynamic range video quality assessment , 2015, SPIE Optical Engineering + Applications.

[83]  C.-C. Jay Kuo,et al.  MCL-V: A streaming video quality assessment database , 2015, J. Vis. Commun. Image Represent..

[84]  Christophe Charrier,et al.  Blind Prediction of Natural Video Quality , 2014, IEEE Transactions on Image Processing.

[85]  Pascal Getreuer,et al.  Automatic Color Enhancement (ACE) and its Fast Implementation , 2012, Image Process. Line.

[86]  Naila Murray,et al.  AVA: A large-scale database for aesthetic visual analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[87]  Rajiv Soundararajan,et al.  Study of Subjective and Objective Quality Assessment of Video , 2010, IEEE Transactions on Image Processing.

[88]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[89]  D. E. Irwin,et al.  Our Eyes do Not Always Go Where we Want Them to Go: Capture of the Eyes by New Objects , 1998 .

[90]  N. Cowan,et al.  The cocktail party phenomenon revisited: attention and memory in the classic selective listening procedure of Cherry (1953). , 1995, Journal of experimental psychology. General.

[91]  Xiaohong Liu,et al.  Video Frame Interpolation via Generalized Deformable Convolution , 2022, IEEE Transactions on Multimedia.

[92]  Qiong Yan,et al.  Disentangling Aesthetic and Technical Effects for Video Quality Assessment of User Generated Content , 2022, ArXiv.

[93]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[94]  Taohong Zhang,et al.  CDCNN: A Model Based on Class Center Vectors and Distance Comparison for Wear Particle Recognition , 2020, IEEE Access.

[95]  Yinqiang Zheng,et al.  Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring , 2020, ECCV.

[96]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[97]  Mikko Nuutinen,et al.  CVD2014—A Database for Evaluating No-Reference Video Quality Assessment Algorithms , 2016, IEEE Transactions on Image Processing.

[98]  Mikko Nuutinen,et al.  CID2013: A Database for Evaluating No-Reference Image Quality Assessment Algorithms , 2015, IEEE Transactions on Image Processing.

[99]  Chao Wang,et al.  Brightness preserving histogram equalization with maximum entropy: a variational perspective , 2005, IEEE Transactions on Consumer Electronics.