Overview of the Multiview and 3D Extensions of High Efficiency Video Coding

The High Efficiency Video Coding (HEVC) standard has recently been extended to support efficient representation of multiview video and depth-based 3D video formats. The multiview extension, MV-HEVC, allows efficient coding of multiple camera views and associated auxiliary pictures, and can be implemented by reusing single-layer decoders without changing the block-level processing modules since block-level syntax and decoding processes remain unchanged. Bit rate savings compared with HEVC simulcast are achieved by enabling the use of inter-view references in motion-compensated prediction. The more advanced 3D video extension, 3D-HEVC, targets a coded representation consisting of multiple views and associated depth maps, as required for generating additional intermediate views in advanced 3D displays. Additional bit rate reduction compared with MV-HEVC is achieved by specifying new block-level video coding tools, which explicitly exploit statistical dependencies between video texture and depth and specifically adapt to the properties of depth maps. The technical concepts and features of both extensions are presented in this paper.

[1]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[2]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[3]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[4]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[5]  Neil A. Dodgson,et al.  Autostereoscopic 3D displays , 2005, Computer.

[6]  Han-Suh Koo,et al.  Motion Information Inferring Scheme for Multi-View Video Coding , 2008, IEICE Trans. Commun..

[7]  Ying Chen,et al.  The Emerging MVC Standard for 3D Video Services , 2008, EURASIP J. Adv. Signal Process..

[8]  David Bull,et al.  Impact of Camera Separation on the Performance of an H.264/AVC based Stereoscopic Video Codec , 2010 .

[9]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[10]  Hakan Urey,et al.  State of the Art in Stereoscopic and Autostereoscopic Displays , 2011, Proceedings of the IEEE.

[11]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[12]  Heiko Schwarz,et al.  3D video coding using the synthesized view distortion change , 2012, 2012 Picture Coding Symposium.

[13]  F. Jager.  Simplified depth map intra coding with an optional depth lookup table , 2012, 2012 International Conference on 3D Imaging (IC3D).

[14]  Mathias Wien,et al.  Model-based intra coding for depth maps in 3D video using a depth lookup table , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[15]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Thiow Keng Tan,et al.  Overview of HEVC High-Level Syntax and Reference Picture Management , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Heiko Schwarz,et al.  3D High-Efficiency Video Coding for Multi-View Video and Depth Data , 2013, IEEE Transactions on Image Processing.

[18]  Fabian Jager.  Depth-based block partitioning for 3D video coding , 2013, 2013 Picture Coding Symposium (PCS).

[19]  Ying Chen,et al.  Advanced residual prediction enhancement for 3D-HEVC , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[20]  Ying Chen,et al.  Generic segment-wise DC for 3D-HEVC depth intra coding , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[21]  Ying Chen,et al.  Low complexity Neighboring Block based Disparity Vector Derivation in 3D-HEVC , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[22]  Oscar C. Au,et al.  View Synthesis Prediction in the 3-D Video Coding Extensions of AVC and HEVC , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Ying Chen,et al.  Low-complexity advanced residual prediction design in 3D-HEVC , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[24]  Ying Chen,et al.  Motion Hooks for the Multiview Extension of HEVC , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[25]  Detlev Marpe,et al.  Depth Intra Coding for 3D Video Based on Geometric Primitives , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[26]  Jianle Chen,et al.  Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard , 2016, IEEE Transactions on Circuits and Systems for Video Technology.