Low-complexity multiview video coding

We consider the problem of complexity reduction in Multiview Video Coding (MVC). We provide a unique comprehensive study that integrates and compares the different low complexity encoding techniques that have been proposed at different levels of the MVC system. In addition, we propose a novel complexity reduction method that takes advantage of the relationship between disparity vectors along time. The relationship is exploited with respect to the motion activity in the frame, as well as with the position of the frame in the Group of Pictures. We integrate this technique into our unique comprehensive framework and evaluate the performance of the resulting system in different setups. We show that the effective combination of complexity reduction techniques results in saving up to 93% in encoding time at the cost of only 0.08 dB in peak signal-to-noise ratio (PSNR) and 1.64% increase in bitrate compared to the standard MVC implementation (JMVM 6.0).

[1]  Itu-T and Iso Iec Jtc Advanced video coding for generic audiovisual services , 2010 .

[2]  Zhi Liu,et al.  Selective Disparity Estimation and Variable Size Motion Estimation Based on Motion Homogeneity for Multi-View Coding , 2009, IEEE Transactions on Broadcasting.

[3]  Thomas Maugey,et al.  Bayesian Early Mode Decision Technique for View Synthesis Prediction-Enhanced Multiview Video Coding , 2013, IEEE Signal Processing Letters.

[4]  Emanuele Trucco,et al.  Fundamentals of Multiple‐View Geometry , 2006 .

[5]  Jiang Li,et al.  An Error Concealment Algorithm for Entire Frame Loss in Video Transmission , 2004 .

[6]  Ying Chen,et al.  Error Resilient Coding and Error Concealment in Scalable Video Coding , 2009 .

[7]  Tao Chen,et al.  3D-TV Content Storage and Transmission , 2011, IEEE Transactions on Broadcasting.

[8]  Ahmet M. Kondoz,et al.  Frame concealment algorithm for stereoscopic video using motion vector sharing , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[9]  Ying Chen,et al.  Frame loss error concealment for multiview video coding , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[10]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[11]  Georgios Paliouras,et al.  An evaluation of Naive Bayesian anti-spam filtering , 2000, ArXiv.

[12]  Thomas Wiegand,et al.  Mobile TV using scalable video coding and layer-aware forward error correction , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[13]  Yo-Sung Ho,et al.  Overview of Multi-view Video Coding , 2007, 2007 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services.

[14]  Reuben A. Farrugia,et al.  Fast inter-mode decision in multi-view video plus depth coding , 2012, 2012 Picture Coding Symposium.

[15]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[16]  B. Girod,et al.  Multiview Video Compression , 2007, IEEE Signal Processing Magazine.

[17]  Peng Chen,et al.  Fast macroblock encoding algorithm based on rate-distortion activity for multiview video coding , 2014, Signal Process. Image Commun..

[18]  Houqiang Li,et al.  Multiview-Video-Plus-Depth Coding Based on the Advanced Video Coding Standard , 2013, IEEE Transactions on Image Processing.

[19]  Heiko Schwarz,et al.  3D High-Efficiency Video Coding for Multi-View Video and Depth Data , 2013, IEEE Transactions on Image Processing.

[20]  Dong Tian,et al.  View synthesis techniques for 3D video , 2009, Optical Engineering + Applications.

[21]  Heiko Schwarz,et al.  Overview of the Scalable Video Coding Extension of the H.264/AVC Standard , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  Avishek Saha,et al.  SKIP Prediction for Fast Rate Distortion Optimization in H.264 , 2007, IEEE Transactions on Consumer Electronics.

[23]  Bo Yan,et al.  Efficient Frame Concealment for Depth Image-Based 3-D Video Transmission , 2012, IEEE Transactions on Multimedia.

[24]  Jonathan Loo,et al.  Measurement Campaign on Transmit Delay Diversity for Mobile DVB-T/H Systems , 2010, IEEE Transactions on Broadcasting.

[25]  Aljoscha Smolic,et al.  Coding Algorithms for 3DTV—A Survey , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[26]  Mtm Marc Lambooij,et al.  Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review , 2009 .

[27]  Georgios Tziritas,et al.  Joint disparity and motion field estimation in stereoscopic image sequences , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[28]  C.-C. Jay Kuo,et al.  Spatial and Temporal Error Concealment Techniques for Video Transmission Over Noisy Channels , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Lei Yang,et al.  Fast mode decision for multiview video coding , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[30]  Zhipin Deng,et al.  Iterative search strategy with selective bi-directional prediction for low complexity multiview video coding , 2012, J. Vis. Commun. Image Represent..

[31]  Kai-Kuang Ma,et al.  Fast Mode Decision for Multiview Video Coding Using Mode Correlation , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  Pascal Frossard,et al.  Fast encoding techniques for Multiview Video Coding , 2013, Signal Process. Image Commun..

[33]  Ying Chen,et al.  Overview of the MVC + D 3D video coding standard , 2014, J. Vis. Commun. Image Represent..

[34]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[35]  Chia-Hung Yeh,et al.  Fast Mode Decision Algorithm for Scalable Video Coding Using Bayesian Theorem Detection and Markov Process , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  Angela Chih-Wei Tang,et al.  Coding statistics based fast mode decision for multi-view video coding , 2013, J. Vis. Commun. Image Represent..

[37]  Bede Liu,et al.  Recovery of lost or erroneously received motion vectors , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[38]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[39]  Aljoscha Smolic,et al.  Efficient Prediction Structures for Multiview Video Coding , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[40]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[41]  Andreas Willig,et al.  A Gilbert-Elliot Bit Error Model and the Efficient Use in Packet Level Simulation , 1999 .

[42]  Yehezkel Yeshurun,et al.  Scene-consistent detection of feature points in video sequences , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[43]  Arnaud Bourge,et al.  MPEG-C PART 3 : ENABLING THE INTRODUCTION OF VIDEO PLUS DEPTH CONTENTS , 2006 .

[44]  Gerhard Stoll,et al.  ISO-MPEG-1 Audio: A Generic Standard for Coding of High-: Quality Digital Audio , 1994 .

[45]  Maja Bystrom,et al.  Fast H . 264 Skip Mode Selection Using an Estimation Framework , 2006 .

[46]  Tao Yan,et al.  View-Adaptive Motion Estimation and Disparity Estimation for Low Complexity Multiview Video Coding , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[47]  Bo Yan,et al.  A Hybrid Frame Concealment Algorithm for H.264/AVC , 2010, IEEE Transactions on Image Processing.

[48]  Anthony Vetro,et al.  View Synthesis for Multiview Video Compression , 2006 .

[49]  Iain E. G. Richardson,et al.  H.264 and MPEG-4 Video Compression: Video Coding for Next-Generation Multimedia , 2003 .

[50]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[51]  Ghassan Al-Regib,et al.  Hierarchical Hole-Filling For Depth-Based View Synthesis in FTV and 3D Video , 2012, IEEE Journal of Selected Topics in Signal Processing.

[52]  Jin Wang,et al.  Depth Image-Based Temporal Error Concealment for 3-D Video Transmission , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[53]  Jin Young Lee,et al.  A Fast and Efficient Multi-View Depth Image Coding Method Based on Temporal and Inter-View Correlations of Texture Images , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[54]  Gangyi Jiang,et al.  DIRECT Mode Early Decision Optimization Based on Rate Distortion Cost Property and Inter-view Correlation , 2013, IEEE Transactions on Broadcasting.

[55]  Anastasis A. Sofokleous,et al.  Review: H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia , 2005, Comput. J..

[56]  Wen Gao,et al.  Concealment of Whole-Picture Loss in Hierarchical B-Picture Scalable Video Coding , 2009, IEEE Transactions on Multimedia.

[57]  Zhipin Deng,et al.  Fast Motion and Disparity Estimation With Adaptive Search Range Adjustment in Stereoscopic Video Coding , 2012, IEEE Transactions on Broadcasting.

[58]  Walter Fischer,et al.  Digital Video and Audio Broadcasting Technology: A Practical Engineering Guide , 2008 .

[59]  Shai Avidan,et al.  Geometrically consistent stereo seam carving , 2011, 2011 International Conference on Computer Vision.

[60]  Yo-Sung Ho,et al.  Virtual View Synthesis Using Temporal Hole Filling with Bilateral Coefficients , 2012, 2012 IEEE RIVF International Conference on Computing & Communication Technologies, Research, Innovation, and Vision for the Future.

[61]  Hakan Urey,et al.  State of the Art in Stereoscopic and Autostereoscopic Displays , 2011, Proceedings of the IEEE.

[62]  Tao Yan,et al.  Early SKIP mode decision for MVC using inter-view correlation , 2010, Signal Process. Image Commun..

[63]  Sehoon Yea,et al.  View synthesis prediction for multiview video coding , 2009, Signal Process. Image Commun..

[64]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[65]  Chang-Su Kim,et al.  Frame loss concealment for stereoscopic video plus depth sequences , 2011, IEEE Transactions on Consumer Electronics.

[66]  B. S. Manjunath,et al.  Improving the quality of depth image based rendering for 3D Video systems , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[67]  Gangyi Jiang,et al.  Efficient Multi-Reference Frame Selection Algorithm for Hierarchical B Pictures in Multiview Video Coding , 2011, IEEE Transactions on Broadcasting.

[68]  Gerardo Rubino,et al.  Quality of Experience estimation using frame loss pattern and video encoding characteristics in DVB-H networks , 2010, 2010 18th International Packet Video Workshop.

[69]  Aljoscha Smolic,et al.  An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution , 2009, 2009 Picture Coding Symposium.

[70]  Fan Zhou,et al.  Fast disparity estimation using spatio-temporal correlation of disparity field for multiview video coding , 2010, IEEE Transactions on Consumer Electronics.

[71]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[72]  David Flynn,et al.  HEVC Complexity and Implementation Analysis , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[73]  Thomas Maugey,et al.  Consistent view synthesis in interactive multiview imaging , 2012, 2012 19th IEEE International Conference on Image Processing.

[74]  Miska M. Hannuksela,et al.  Nonlinear Depth Map Resampling for Depth-Enhanced 3-D Video Coding , 2013, IEEE Signal Processing Letters.

[75]  Marc Pollefeys,et al.  An evolutionary and optimised approach on 3D-TV , 2002 .

[76]  Yo-Sung Ho,et al.  Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video , 2009, 2009 Picture Coding Symposium.

[77]  Neil A. Dodgson,et al.  Autostereoscopic 3D displays , 2005, Computer.

[78]  Bastian Leibe,et al.  Joint 2D-3D temporally consistent semantic segmentation of street scenes , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[79]  Ken Chen,et al.  Asymmetric Coding of Multi-View Video Plus Depth Based 3-D Video for View Rendering , 2012, IEEE Transactions on Multimedia.

[80]  Muhammad Shafique,et al.  A complexity reduction scheme with adaptive search direction and mode elimination for multiview video coding , 2012, 2012 Picture Coding Symposium.

[81]  Faouzi Kossentini,et al.  H.263+: video coding at low bit rates , 1998, IEEE Trans. Circuits Syst. Video Technol..

[82]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[83]  Ming Lei Liou,et al.  Overview of the p×64 kbit/s video coding standard , 1991, CACM.

[84]  Daniel Scharstein,et al.  View Synthesis Using Stereo Vision , 2001, Lecture Notes in Computer Science.

[85]  Mei Yu,et al.  Statistical Early Termination Model for Fast Mode Decision and Reference Frame Selection in Multiview Video Coding , 2012, IEEE Transactions on Broadcasting.

[86]  Yo-Sung Ho,et al.  A framework of 3D video coding using view synthesis prediction , 2012, 2012 Picture Coding Symposium.

[87]  Chang-Su Kim,et al.  Error concealment of multi-view video sequences using inter-view and intra-view correlations , 2009, J. Vis. Commun. Image Represent..

[88]  Thomas Wiegand,et al.  Mixed resolution coding of stereoscopic video for Mobile devices , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[89]  Changick Kim,et al.  Depth-Based Disocclusion Filling for Virtual View Synthesis , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[90]  Sergio Bampi,et al.  A multi-level dynamic complexity reduction scheme for multiview video coding , 2011, 2011 18th IEEE International Conference on Image Processing.

[91]  Thomas Wiegand,et al.  3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[92]  K. Rijkse,et al.  H.263: video coding for low-bit-rate communication , 1996, IEEE Commun. Mag..

[93]  Aljoscha Smolic,et al.  Multi-View Video Plus Depth Representation and Coding , 2007, 2007 IEEE International Conference on Image Processing.