Deep Video Compression for P-frame in Sub-sampled Color Spaces

In this paper, we propose a deep video compression method for P-frame in sub-sampled color spaces regarding the YUV420, which has been widely adopted in many state-of-art hybrid video compression standards, in an effort to achieve high compression performance. We adopt motion estimation and motion compression to facilitate the inter prediction of the videos with YUV420 color format, shrinking the total data volume of motion information. Moreover, the motion compensation module on YUV420 is cooperated to enhance the quality of the compensated frame with the consideration of the resolution alignment in the sub-sampled color spaces. To explore the cross-component correlation, the residual encoder-decoder is accompanied with two head-branches and color information fusion. Additionally, a weighted loss emphasizing more on the Y component is utilized to enhance the compression efficiency. Experimental results show that the proposed method can realize 19.82% bit rate reductions on average compared to the deep video compression (DVC) method in terms of the combined PSNR and predominant gains on the Y component.

[1]  Li Zhang,et al.  Convolutional Neural Network Based In-Loop Filter For VVC Intra Coding , 2021, 2021 IEEE International Conference on Image Processing (ICIP).

[2]  Marta Karczewicz,et al.  Transform Network Architectures for Deep Learning Based End-to-End Image/Video Coding in Subsampled Color Spaces , 2021, IEEE Open Journal of Signal Processing.

[3]  Gary J. Sullivan,et al.  Developments in International Video Coding Standardization After AVC, With an Overview of Versatile Video Coding (VVC) , 2021, Proceedings of the IEEE.

[4]  Akshay Pushparaja,et al.  CompressAI: a PyTorch library and evaluation platform for end-to-end compression research , 2020, ArXiv.

[5]  L. Gool,et al.  OpenDVC: An Open Source Implementation of the DVC Video Compression Method , 2020, ArXiv.

[6]  Houqiang Li,et al.  M-LVC: Multiple Frames Prediction for Learned Video Compression , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Li Chen,et al.  Content Adaptive and Error Propagation Aware Deep Video Compression , 2020, ECCV.

[8]  Masaru Takeuchi,et al.  Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Wen Gao,et al.  Residual in Residual Based Convolutional Neural Network In-loop Filter for AVS3 , 2019, 2019 Picture Coding Symposium (PCS).

[10]  Wen Gao,et al.  Recent Development of AVS Video Coding Standard: AVS3 , 2019, 2019 Picture Coding Symposium (PCS).

[11]  Xinfeng Zhang,et al.  Enhanced Bi-Prediction With Convolutional Neural Network for High-Efficiency Video Coding , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Taco S. Cohen,et al.  Video Compression With Rate-Distortion Autoencoders , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Dong Liu,et al.  Neural Network-Based Arithmetic Coding for Inter Prediction Information in HEVC , 2019, 2019 IEEE International Symposium on Circuits and Systems (ISCAS).

[14]  Wen Gao,et al.  Enhanced Motion-Compensated Video Coding With Deep Virtual Reference Frame Generation , 2019, IEEE Transactions on Image Processing.

[15]  Xinfeng Zhang,et al.  Image and Video Compression With Neural Networks: A Review , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Xiaoyun Zhang,et al.  DVC: An End-To-End Deep Video Compression Framework , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Dong Liu,et al.  A Hybrid Neural Network for Chroma Intra Prediction , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[18]  David Minnen,et al.  Joint Autoregressive and Hierarchical Priors for Learned Image Compression , 2018, NeurIPS.

[19]  Wenhan Yang,et al.  Progressive Spatial Recurrent Neural Network for Intra Prediction , 2018, IEEE Transactions on Multimedia.

[20]  Bin Li,et al.  Fully Connected Network-Based Intra Prediction for Image Coding , 2018, IEEE Transactions on Image Processing.

[21]  David Minnen,et al.  Variational image compression with a scale hyperprior , 2018, ICLR.

[22]  W. Freeman,et al.  Video Enhancement with Task-Oriented Flow , 2017, International Journal of Computer Vision.

[23]  Lubomir D. Bourdev,et al.  Real-Time Adaptive Image Compression , 2017, ICML.

[24]  Michael J. Black,et al.  Optical Flow Estimation Using a Spatial Pyramid Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Qionghai Dai,et al.  Deep Convolutional Neural Network for Decompressed Video Enhancement , 2016, 2016 Data Compression Conference (DCC).

[26]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.