Pixel-Level View Synthesis Distortion Estimation for 3D Video Coding

Recently, region-based 3D video coding has been proposed. However, existing view synthesis distortion estimation (VSDE) methods are performed at the frame level. To guide the rate-distortion optimization process of region-based 3D video coding schemes, this paper proposes the first pixel-level VSDE (PL-VSDE) method. We first give the definition of the pixel-level view synthesis distortion. To estimate it, a backward prediction method is then developed, which starts from the pixels of interest (POIs) in the virtual view and finds their corresponding pixels in the reference view via a coarse-to-fine approach, denoted as coarse-to-fine backward prediction (CFBP) method. Additionally, the CFBP fully considers the details of 3D warping, the rounding operation and the warping competition in view synthesis, leading to improve accuracy of the prediction. Besides, a table-lookup method and a warping property are introduced to speed up the CFBP. After integrating the CFBP into the PL-VSDE, we can estimate the view synthesis distortion at the pixel level. Our method is carried out pixel-by-pixel independently, which is friendly for parallel processing. The experimental results demonstrate that our proposed method has significant advantages in both accuracy and efficiency compared with the state-of-the-art frame-level VSDE methods.

[1]  Jian Jin,et al.  Synthesis-Aware Region-Based 3D Video Coding , 2015, PCM.

[2]  Dong Tian,et al.  Boundary Artifact Reduction in View Synthesis of 3D Video: From Perspective of Texture-Depth Alignment , 2011, IEEE Transactions on Broadcasting.

[3]  Liang-Gee Chen,et al.  Hybrid motion/depth-oriented inpainting for virtual view synthesis in multiview applications , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[4]  Tal Hassner,et al.  What Does the Scene Look Like from a Scene Point? , 2002, ECCV.

[5]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[6]  Aljoscha Smolic,et al.  Multi-View Video Plus Depth Representation and Coding , 2007, 2007 IEEE International Conference on Image Processing.

[7]  Yun Zhang,et al.  A Virtual View PSNR Estimation Method for 3-D Videos , 2016, IEEE Transactions on Broadcasting.

[8]  Gary J. Sullivan,et al.  Rate-distortion optimization for video compression , 1998, IEEE Signal Process. Mag..

[9]  Dong Zhang,et al.  View Synthesis Distortion Estimation With a Graphical Model and Recursive Calculation of Probability Distribution , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[11]  J. Li,et al.  An Epipolar Geometry-Based Fast Disparity Estimation Algorithm for Multiview Image and Video Coding , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Wei Xiang,et al.  Analysis of Packet-Loss-Induced Distortion in View Synthesis Prediction-Based 3D Video Coding , 2017, IEEE Transactions on Image Processing.

[13]  Lu Fang,et al.  An Analytical Model for Synthesis Distortion Estimation in 3D Video , 2014, IEEE Transactions on Image Processing.

[14]  Byung Tae Oh,et al.  View Synthesis Distortion Estimation for AVC- and HEVC-Compatible 3-D Video Coding , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Marco Grangetto,et al.  Panorama View With Spatiotemporal Occlusion Compensation for 3D Video Coding , 2015, IEEE Transactions on Image Processing.

[16]  Krzysztof Wegner,et al.  High Efficiency 3D Video Coding Using New Tools Based on View Synthesis , 2013, IEEE Transactions on Image Processing.

[17]  Tammam Tillo,et al.  Region-Based Multiple Description Coding for Multiview Video Plus Depth Video , 2018, IEEE Transactions on Multimedia.

[18]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[19]  Hsueh-Ming Hang,et al.  Virtual view synthesis using backward depth warping algorithm , 2013, 2013 Picture Coding Symposium (PCS).

[20]  Yao Zhao,et al.  A Depth-Bin-Based Graphical Model for Fast View Synthesis Distortion Estimation , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Krzysztof Wegner,et al.  3D video compression by coding of disoccluded regions , 2012, 2012 19th IEEE International Conference on Image Processing.

[22]  Dong Tian,et al.  View synthesis techniques for 3D video , 2009, Optical Engineering + Applications.

[23]  Yao Zhao,et al.  Depth Map Driven Hole Filling Algorithm Exploiting Temporal Correlation Information , 2014, IEEE Transactions on Broadcasting.

[24]  Marco Grangetto,et al.  Depth image based rendering with inverse mapping , 2013, 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP).