MV-GNN: Multi-View Graph Neural Network for Compression Artifacts Reduction

Inevitable compression artifacts in multi-view video (MVV) can clearly degrade the quality of experience in many interaction-oriented 3D visual applications. Under the framework of asymmetric coding, low-quality images can be enhanced with high-quality images from the neighboring viewpoints considering the similarity among different views. However, compression artifacts and warping error cause different cross-view quality gaps for various sequences, and thus the contribution of cross-view priors can hardly be located and considered in previous works. In this paper, we propose a multi-view graph neural network (MV-GNN) to reduce compression artifacts in multi-view compressed images. We dedicate to design a fusion mechanism which can exploit contributions from neighboring viewpoints and meanwhile suppress the misleading information. In our method, a GNN-based fusion mechanism is designed to fuse the cross-view information under the aggregation and update mechanism of GNN. Experiments show that 1.672 dB and 0.0242 average gains on PSNR and SSIM metrics can be obtained, respectively. For the subjective evaluations, blocking effect in the compressed images are clearly suppressed and the damaged object boundary are better recovered. The experimental results demonstrate that our MV-GNN outperforms the state-of-the-art methods.

[1]  Thomas Wiegand,et al.  3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[2]  Ying Chen,et al.  Overview of the Multiview and 3D Extensions of High Efficiency Video Coding , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Klaus Hopf,et al.  Key technologies for an advanced 3D TV system , 2004, SPIE Optics East.

[4]  Xinfeng Zhang,et al.  Enhanced Image Decoding via Edge-Preserving Generative Adversarial Networks , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[5]  Zulin Wang,et al.  Decoder-side HEVC quality enhancement with scalable convolutional neural network , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[6]  Alan C. Bovik,et al.  Image information and visual quality , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[7]  Abhinav Gupta,et al.  Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Zulin Wang,et al.  Multi-frame Quality Enhancement for Compressed Video , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Sebastian Nowozin,et al.  Loss-Specific Training of Non-Parametric Image Restoration Models: A New State of the Art , 2012, ECCV.

[10]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[11]  Weisi Lin,et al.  Efficient Image Deblocking Based on Postfiltering in Shifted Windows , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Karen O. Egiazarian,et al.  Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images , 2007, IEEE Transactions on Image Processing.

[13]  Wenhan Yang,et al.  Dmcnn: Dual-Domain Multi-Scale Convolutional Neural Network for Compression Artifacts Removal , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[14]  Zulin Wang,et al.  Enhancing Quality for HEVC Compressed Videos , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Ming Lu,et al.  Learned Quality Enhancement via Multi-Frame Priors for HEVC Compliant Low-Delay Applications , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[16]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[17]  Yao Zhao,et al.  3D video super-resolution using fully convolutional neural networks , 2016, 2016 IEEE International Conference on Multimedia and Expo (ICME).

[18]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[20]  Tingting Wang,et al.  A Novel Deep Learning-Based Method of Improving Coding Efficiency from the Decoder-End for HEVC , 2017, 2017 Data Compression Conference (DCC).

[21]  Shuicheng Yan,et al.  Semantic Object Parsing with Graph LSTM , 2016, ECCV.

[22]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[23]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[24]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Michael K. Ng,et al.  Reducing Artifacts in JPEG Decompression Via a Learned Dictionary , 2014, IEEE Transactions on Signal Processing.

[26]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[27]  Dong Liu,et al.  A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding , 2016, MMM.

[28]  Xin He,et al.  Cross-View Multi-Lateral Filter for Compressed Multi-View Depth Video , 2019, IEEE Transactions on Image Processing.

[29]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[30]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Hongyang Chao,et al.  One-To-Many Network for Visually Pleasing Compression Artifacts Reduction , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[33]  Hong Yan,et al.  Blocking artifacts suppression in block-coded images using overcomplete wavelet representation , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[34]  Yi Xu,et al.  Non-Local ConvLSTM for Video Compression Artifact Reduction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Wen Gao,et al.  Low-Rank-Based Nonlocal Adaptive Loop Filter for High-Efficiency Video Compression , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  Licheng Jiao,et al.  Image deblocking via sparse representation , 2012, Signal Process. Image Commun..

[37]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[38]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[39]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[40]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[41]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[42]  Xiaoyan Sun,et al.  Quality-Gated Convolutional Lstm for Enhancing Compressed Video , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[43]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[45]  Xiaoyun Zhang,et al.  Enhancing HEVC Compressed Videos with a Partition-Masked Convolutional Neural Network , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[46]  Alberto Del Bimbo,et al.  Deep Universal Generative Adversarial Compression Artifact Removal , 2019, IEEE Transactions on Multimedia.

[47]  Sanja Fidler,et al.  3D Graph Neural Networks for RGBD Semantic Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Shiqi Wang,et al.  Convolutional Neural Network-Based Synthesized View Quality Enhancement for 3D Video Coding , 2018, IEEE Transactions on Image Processing.

[49]  Ying Chen,et al.  Standardized Extensions of High Efficiency Video Coding (HEVC) , 2013, IEEE Journal of Selected Topics in Signal Processing.

[50]  Zoe Liu,et al.  Learning-Based Multi-Frame Video Quality Enhancement , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[51]  Xianming Liu,et al.  Data-driven sparsity-based restoration of JPEG-compressed images in dual transform-pixel domain , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Heiko Schwarz,et al.  3D High-Efficiency Video Coding for Multi-View Video and Depth Data , 2013, IEEE Transactions on Image Processing.

[53]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.