Fast 3D-HEVC Depth Map Encoding Using Machine Learning

This paper presents a fast depth map encoding for 3D-High Efficiency Video Coding (3D-HEVC) based on static decision trees. We used data mining and machine learning to correlate the encoder context attributes, building the static decision trees. Each decision tree defines that a depth map Coding Unit (CU) must be or not be split into smaller blocks, considering the encoding context through the evaluation of the encoder attributes. Specialized decision trees for I-frames, P-frames and B-frames define the partitioning of <inline-formula> <tex-math notation="LaTeX">$64\times 64$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$32\times 32$ </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">$16\times 16$ </tex-math></inline-formula> CUs. We trained the decision trees using data extracted from the 3D-HEVC Test Model considering all-intra and random-access configurations, and we evaluated the proposed approach considering the common test conditions. The experimental results demonstrated that this approach can halve the 3D-HEVC encoder computational effort with less than 0.24% of BD-rate increase on the average for all-intra configuration. When running on random-access configuration, our solution is able to reduce up to 58% the complete 3D-HEVC encoder computational effort with a BD-rate drop of only 0.13%. These results surpass all related works regarding computational effort reduction and BD-rate.

[1]  Yui-Lam Chan,et al.  Early determination of intra mode and segment-wise DC coding for depth map based on hierarchical coding structure in 3D-HEVC , 2015, 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[2]  Yui-Lam Chan,et al.  Fast wedgelet pattern decision for DMM in 3D-HEVC , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[3]  Eduardo Peixoto,et al.  Fast H.264/AVC to HEVC transcoding based on machine learning , 2014, 2014 International Telecommunications Symposium (ITS).

[4]  Itu-T and Iso Iec Jtc Advanced video coding for generic audiovisual services , 2010 .

[5]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[6]  Gustavo Sanchez,et al.  Performance Analysis of Depth Intra-Coding in 3D-HEVC , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[7]  Detlev Marpe,et al.  Depth Intra Coding for 3D Video Based on Geometric Primitives , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[8]  Zhan Ma,et al.  Fast Mode and Partition Decision Using Machine Learning for Intra-Frame Coding in HEVC Screen Content Coding Extension , 2016, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[9]  Yui-Lam Chan,et al.  Quadtree decision for depth intra coding in 3D-HEVC by good feature , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Jianjun Lei,et al.  Depth Sensation Enhancement for Multiple Virtual View Rendering , 2015, IEEE Transactions on Multimedia.

[11]  Mohamed-Chaker Larabi,et al.  Heuristic inspired search method for fast wedgelet pattern decision in 3D-HEVC , 2016, 2016 6th European Workshop on Visual Information Processing (EUVIP).

[12]  Biao Min,et al.  A Fast CU Size Decision Algorithm for the HEVC Intra Encoder , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Guilherme Corrêa,et al.  Fast HEVC Encoding Decisions Using Data Mining , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Fabian Jager Depth-based block partitioning for 3D video coding , 2013, 2013 Picture Coding Symposium (PCS).

[15]  José Luis Martínez,et al.  Fast partitioning algorithm for HEVC Intra frame coding using machine learning , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[16]  Nam Ling,et al.  Fast single depth intra mode decision for depth map coding in 3D-HEVC , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[17]  Guorui Feng,et al.  Efficient Intra Mode Selection for Depth-Map Coding Utilizing Spatiotemporal, Inter-Component and Inter-View Correlations in 3D-HEVC , 2018, IEEE Transactions on Image Processing.

[18]  Wen Gao,et al.  Low Complexity Adaptive View Synthesis Optimization in HEVC Based 3D Video Coding , 2014, IEEE Transactions on Multimedia.

[19]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[20]  Wen-Nung Lie,et al.  Low complexity depth intra coding combining fast intra mode and fast CU size decision in 3D-HEVC , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[21]  Michael J. Pazzani,et al.  An Investigation of Noise-Tolerant Relational Concept Learning Algorithms , 1991, ML.

[22]  Yong Gan,et al.  Fast depth map mode decision based on depth-texture correlation and edge classification for 3D-HEVC , 2017, J. Vis. Commun. Image Represent..

[23]  Béatrice Pesquet-Popescu,et al.  Initialization, Limitation, and Predictive Coding of the Depth and Texture Quadtree in 3D-HEVC , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Wen Gao,et al.  New bi-prediction techniques for B pictures coding [video coding] , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[25]  Heiko Schwarz,et al.  3D High-Efficiency Video Coding for Multi-View Video and Depth Data , 2013, IEEE Transactions on Image Processing.

[26]  Detlev Marpe,et al.  Block Merging for Quadtree-Based Partitioning in HEVC , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Li Zhang,et al.  Multiview and 3D Video Compression Using Neighboring Block Based Disparity Vectors , 2016, IEEE Transactions on Multimedia.

[28]  Yong Wang,et al.  Reducing Wedgelet lookup table size with down-sampling for depth map coding in 3D-HEVC , 2015, 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP).

[29]  Bruno Zatt,et al.  Complexity reduction for 3D-HEVC depth maps intra-frame prediction using simplified edge detector algorithm , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[30]  Ying Chen,et al.  Overview of the Multiview and 3D Extensions of High Efficiency Video Coding , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[32]  Zhi Liu,et al.  Adaptive Inter-Mode Decision for HEVC Jointly Utilizing Inter-Level and Spatiotemporal Correlations , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Olgierd Stankiewicz,et al.  A Free-Viewpoint Television System for Horizontal Virtual Navigation , 2018, IEEE Transactions on Multimedia.

[34]  Ying Chen,et al.  Generic segment-wise DC for 3D-HEVC depth intra coding , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[35]  Bruno Zatt,et al.  Complexity reduction for the 3D-HEVC depth maps coding , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[36]  Zhenyu Liu,et al.  CU Partition Mode Decision for HEVC Hardwired Intra Encoder Using Convolution Neural Network , 2016, IEEE Transactions on Image Processing.

[37]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[38]  Qionghai Dai,et al.  A quad-tree and statistics based fast CU depth decision algorithm for 3D-HEVC , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[39]  Yong Gan,et al.  Fast intra mode decision for depth coding in 3D-HEVC , 2017, Multidimens. Syst. Signal Process..

[40]  Munchurl Kim,et al.  A Novel Fast CU Encoding Scheme Based on Spatiotemporal Encoding Parameters for HEVC Inter Coding , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[42]  Zhan Ma,et al.  Fast Intra Mode Decision for High Efficiency Video Coding (HEVC) , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[43]  Jianjun Lei,et al.  Fast Mode Decision Based on Grayscale Similarity and Inter-View Correlation for Depth Map Coding in 3D-HEVC , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[44]  C. Apte,et al.  Data mining with decision trees and decision rules , 1997, Future Gener. Comput. Syst..

[45]  G. Bjontegaard,et al.  Calculation of Average PSNR Differences between RD-curves , 2001 .

[46]  César A. M. Marcon,et al.  Fast 3D-Hevc Depth Maps Intra-Frame Prediction Using Data Mining , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[47]  Yo-Sung Ho,et al.  Depth Video Coding Using Adaptive Geometry Based Intra Prediction for 3-D Video Systems , 2012, IEEE Transactions on Multimedia.

[48]  Kai-Kuang Ma,et al.  Sum-of-gradient based fast intra coding in 3D-HEVC for depth map sequence (SOG-FDIC) , 2017, J. Vis. Commun. Image Represent..

[49]  Ying Chen,et al.  Standardized Extensions of High Efficiency Video Coding (HEVC) , 2013, IEEE Journal of Selected Topics in Signal Processing.

[50]  Yui-Lam Chan,et al.  Probability-Based Depth Intra-Mode Skipping Strategy and Novel VSO Metric for DMM Decision in 3D-HEVC , 2018, IEEE Transactions on Circuits and Systems for Video Technology.