3D High-Efficiency Video Coding for Multi-View Video and Depth Data

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

[1]  Qingming Huang,et al.  Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model , 2009, Signal Process. Image Commun..

[2]  Antonio Ortega,et al.  Transform domain sparsification of depth maps using iterative quadratic programming , 2011, 2011 18th IEEE International Conference on Image Processing.

[3]  Christophe Tillier,et al.  Motion Vector Sharing and Bitrate Allocation for 3D Video-Plus-Depth Coding , 2009, EURASIP J. Adv. Signal Process..

[4]  Heiko Schwarz,et al.  Analysis of Hierarchical B Pictures and MCTF , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[5]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[6]  Heiko Schwarz,et al.  3D video coding using the synthesized view distortion change , 2012, 2012 Picture Coding Symposium.

[7]  Gary J. Sullivan,et al.  Rate-constrained coder control and comparison of video coding standards , 2003, IEEE Trans. Circuits Syst. Video Technol..

[8]  Nikolce Stefanoski,et al.  Automatic content creation for multiview autostereoscopic displays using image domain warping , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[9]  Gene Cheung,et al.  Arithmetic edge coding for arbitrarily shaped sub-block motion prediction in depth video compression , 2012, 2012 19th IEEE International Conference on Image Processing.

[10]  Jaejoon Lee,et al.  Depth Map Coding Based on Synthesized View Distortion Function , 2011, IEEE Journal of Selected Topics in Signal Processing.

[11]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[12]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[13]  Gary J. Sullivan,et al.  Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding (HEVC) , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Detlev Marpe,et al.  Block Merging for Quadtree-Based Partitioning in HEVC , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Ismo Rakkolainen,et al.  A Survey of 3DTV Displays: Techniques and Technologies , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Antonio Ortega,et al.  Depth map coding with distortion estimation of rendered view , 2010, Electronic Imaging.

[17]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[18]  Itu-T and Iso Iec Jtc.  Advanced video coding for generic audiovisual services , 2010 .

[19]  Emanuele Trucco,et al.  A compact algorithm for rectification of stereo pairs , 2000, Machine Vision and Applications.

[20]  Edson M. Hung,et al.  Efficiency improvements for a geometric-partition-based video coder , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[21]  Hideaki Kimata,et al.  View Scalable Multiview Video Coding Using 3-D Warping With Depth Map , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  DariboIsmaël,et al.  Motion vector sharing and bitrate allocation for 3D video-plus-depth coding , 2008 .

[23]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[24]  Thomas Wiegand,et al.  3-D Video Representation Using Depth Maps , 2011, Proceedings of the IEEE.

[25]  Yo-Sung Ho,et al.  H.264-Based Depth Map Sequence Coding Using Motion Information of Corresponding Texture Video , 2006, PSIVT.

[26]  Oliver Schreer,et al.  Stereo analysis by hybrid recursive matching for real-time immersive video conferencing , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  HOMAS,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H . 264 / MPEG-4 AVC Standard , 2022 .

[28]  Peter H. N. de With,et al.  Platelet-based coding of depth maps for the transmission of multiview images , 2006, Electronic Imaging.

[29]  Aljoscha Smolic,et al.  Efficient Prediction Structures for Multiview Video Coding , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[31]  Minh N. Do,et al.  Shape-adaptivewavelet encoding of depth maps , 2009, 2009 Picture Coding Symposium.

[32]  Yo-Sung Ho,et al.  Mesh-Based Depth Coding for 3D Video using Hierarchical Decomposition of Depth Maps , 2007, 2007 IEEE International Conference on Image Processing.

[33]  Marek Domanski,et al.  Depth-based inter-view prediction of motion vectors for improved multiview video coding , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[34]  Heiko Schwarz,et al.  3D video coding using advanced prediction, depth modeling, and encoder control methods , 2012, 2012 Picture Coding Symposium.

[35]  T. Cover,et al.  Rate Distortion Theory , 2001 .

[36]  G. Bjontegaard,et al.  Calculation of Average PSNR Differences between RD-curves , 2001 .

[37]  Detlev Marpe,et al.  3D video: Depth coding based on inter-component prediction of block partitions , 2012, 2012 Picture Coding Symposium.

[38]  Christophe Tillier,et al.  Adaptive wavelet coding of the depth map for stereoscopic view synthesis , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[39]  Yo-Sung Ho,et al.  Three-dimensional video generation using foreground separation and disocclusion detection , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[40]  Yo-Sung Ho,et al.  Depth Reconstruction Filter and Down/Up Sampling for Depth Coding in 3-D Video , 2009, IEEE Signal Processing Letters.

[41]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[42]  Ying Chen,et al.  The Emerging MVC Standard for 3D Video Services , 2008, EURASIP J. Adv. Signal Process..

[43]  Heiko Schwarz,et al.  Motion vector inheritance for high efficiency 3D video plus depth coding , 2012, 2012 Picture Coding Symposium.

[44]  M. Halle,et al.  3-D Displays and Signal Processing , 2007, IEEE Signal Processing Magazine.

[45]  M. Gross,et al.  Nonlinear disparity mapping for stereoscopic 3D , 2010, ACM Trans. Graph..

[46]  Luc Van Gool,et al.  ATTEST: Advanced Three-dimensional Television System Technologies , 2002 .

[47]  Heiko Schwarz,et al.  Inter-view prediction of motion data in multiview video coding , 2012, 2012 Picture Coding Symposium.

[48]  Luc Van Gool,et al.  Advanced three-dimensional television system technologies , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[49]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[50]  W. Gao,et al.  Inter-View Direct Mode for Multiview Video Coding , 2006, IEEE Transactions on Circuits and Systems for Video Technology.