On the Generalization of BasicVSR++ to Video Deblurring and Denoising

The exploitation of long-term information has been a long-standing problem in video restoration. The recent BasicVSR and BasicVSR++ have shown remarkable performance in video super-resolution through long-term propagation and effective alignment. Their success has led to a question of whether they can be transferred to different video restoration tasks. In this work, we extend BasicVSR++ to a generic framework for video restoration tasks. In tasks where inputs and outputs possess identical spatial size, the input resolution is reduced by strided convolutions to maintain efficiency. With only minimal changes from BasicVSR++, the proposed framework achieves compelling performance with great efficiency in various video restoration tasks including video deblurring and denoising. Notably, BasicVSR++ achieves comparable performance to Transformer-based approaches with up to 79% of parameter reduction and 44 × speedup. The promising results demonstrate the importance of propagation and alignment in video restoration tasks beyond just video super-resolution. Code and models are available at https://github.com/ ckkelvinchan/BasicVSR_PlusPlus .

[1]  Dong Liu,et al.  Neural Compression-Based Feature Learning for Video Restoration , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  L. Gool,et al.  VRT: A Video Restoration Transformer , 2022, IEEE Transactions on Image Processing.

[3]  L. Gool,et al.  Flow-Guided Sparse Transformer for Video Deblurring , 2022, ICML.

[4]  Shangchen Zhou,et al.  BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  L. Gool,et al.  Video Super-Resolution Transformer , 2021, ArXiv.

[6]  A. Rajagopalan,et al.  Gated Spatio-Temporal Attention-Guided Video Deblurring , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Radu Timofte,et al.  NTIRE 2021 Challenge on Video Super-Resolution , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Radu Timofte,et al.  NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Michael Elad,et al.  Patch Craft: Video Denoising by Deep Modeling and Patch Matching , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  H. Suominen,et al.  ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Chen Change Loy,et al.  BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chen Change Loy,et al.  Understanding Deformable Alignment in Video Super-Resolution , 2020, AAAI.

[13]  Hao Wei,et al.  Deep Video Deblurring Using Sharpness Features From Exemplars , 2020, IEEE Transactions on Image Processing.

[14]  Jinhui Tang,et al.  Cascaded Deep Video Deblurring Using Temporal Sharpness Prior , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  J. Delon,et al.  FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Julie Delon,et al.  DVDNET: A Fast Network for Deep Video Denoising , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[17]  Kyoung Mu Lee,et al.  Recurrent Neural Networks With Intra-Frame Iterations for Video Deblurring , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Chen Change Loy,et al.  EDVR: Video Restoration With Enhanced Deformable Convolutional Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Wangmeng Zuo,et al.  Spatio-Temporal Filter Adaptive Network for Video Deblurring , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Gregory Shakhnarovich,et al.  Recurrent Back-Projection Network for Video Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Stephen Lin,et al.  Deformable ConvNets V2: More Deformable, Better Results , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Hongdong Li,et al.  Adversarial Spatio-Temporal Learning for Video Deblurring , 2018, IEEE Transactions on Image Processing.

[23]  Jean-Michel Morel,et al.  Non-Local Video Denoising by CNN , 2018, ArXiv.

[24]  Bernt Schiele,et al.  Video Object Segmentation with Language Referring Expressions , 2018, ACCV.

[25]  Yi Wang,et al.  Scale-Recurrent Network for Deep Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  W. Freeman,et al.  Video Enhancement with Task-Oriented Flow , 2017, International Journal of Computer Vision.

[27]  Bernhard Schölkopf,et al.  Learning Blind Motion Deblurring , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Guillermo Sapiro,et al.  Deep Video Deblurring for Hand-Held Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[33]  Jean-Michel Morel,et al.  Towards a Bayesian Video Denoising Method , 2015, ACIVS.

[34]  Tae Hyun Kim,et al.  Generalized video deblurring for dynamic scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Karen O. Egiazarian,et al.  Video denoising using separable 4D nonlocal spatiotemporal transforms , 2011, Electronic Imaging.

[37]  Michel Barlaud,et al.  Two deterministic half-quadratic regularization algorithms for computed imaging , 1994, Proceedings of 1st International Conference on Image Processing.