Emerging Imaging Technologies: Trends and Challenges

This chapter addresses image and video technologies related to 3D immersive multimedia delivery systems with special emphasis on the most promising digital formats. Besides recent research results and technical challenges associated with multiview image and image, video and lightfield acquisition and processing, the chapter also presents relevant results from international standardization activities in the scope of ISO, IEC, and ITU. Standard solutions to encode multiview image and video content and ongoing research are addressed, along with novel solutions to enable further developments in the emerging technologies dealing with capture and coding for lightfield content and free viewpoint television.

[1]  Ahmet M. Kondoz,et al.  Motion and Disparity Estimation with Self Adapted Evolutionary Strategy in 3D Video Coding , 2007, IEEE Transactions on Consumer Electronics.

[2]  Ying Chen,et al.  Joint texture and depth map video coding based on the scalable extension of H.264/AVC , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[3]  Toshiaki Fujii,et al.  Free-Viewpoint TV , 2011, IEEE Signal Processing Magazine.

[4]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[5]  Yo-Sung Ho,et al.  View-consistent multi-view depth estimation for three-dimensional video generation , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[6]  Jan Plogsties,et al.  MPEG-H 3D Audio—The New Standard for Coding of Immersive Spatial Audio , 2015, IEEE Journal of Selected Topics in Signal Processing.

[7]  Qifei Wang.  Computational Models for Multiview Dense Depth Maps of Dynamic Scene , 2015, ArXiv.

[8]  Péter Tamás Kovács,et al.  Real-time 3D light field transmission , 2010, Photonics Europe.

[9]  Luís Ducla Soares,et al.  Inter-Layer Prediction Scheme for Scalable 3-D Holoscopic Video Coding , 2013, IEEE Signal Processing Letters.

[10]  Marc Pollefeys,et al.  A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  Gauthier Lafruit,et al.  Multi-camera epipolar plane image feature detection for robust view synthesis , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[12]  Joachim Keinert,et al.  Light-Field Acquisition System That Facilitates Camera and Depth-of-Field Compositing in Post-Production , 2015 .

[13]  Luís Ducla Soares,et al.  Spatial prediction based on self-similarity compensation for 3D holoscopic image and video coding , 2011, 2011 18th IEEE International Conference on Image Processing.

[14]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[15]  Krzysztof Wegner,et al.  3D-HEVC extension for circular camera arrangements , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[16]  Bahram Javidi,et al.  Advances in three-dimensional integral imaging: sensing, display, and applications [Invited]. , 2013, Applied optics.

[17]  Krzysztof Wegner,et al.  High Efficiency 3D Video Coding Using New Tools Based on View Synthesis , 2013, IEEE Transactions on Image Processing.

[18]  Jens Ogniewski,et al.  Model-Based Video Coding Using Colour and Depth Cameras , 2011, 2011 International Conference on Digital Image Computing: Techniques and Applications.

[19]  Yo-Sung Ho,et al.  Virtual view synthesis method and self‐evaluation metrics for free viewpoint television and 3D video , 2010, Int. J. Imaging Syst. Technol..

[20]  Feng Yan,et al.  Stereoacuity-guided depth image based rendering , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[21]  Luís Ducla Soares,et al.  3D Holoscopic video coding using MVC , 2011, 2011 IEEE EUROCON - International Conference on Computer as a Tool.

[22]  Joachim Keinert,et al.  Dense lightfield reconstruction from multi aperture cameras , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[23]  Jianjun Lei,et al.  Depth Sensation Enhancement for Multiple Virtual View Rendering , 2015, IEEE Transactions on Multimedia.

[24]  L. Hong,et al.  Segment-based stereo matching using graph cuts , 2004, CVPR 2004.

[25]  Krzysztof Wegner,et al.  Multiview synthesis — Improved view synthesis for virtual navigation , 2016, 2016 Picture Coding Symposium (PCS).

[26]  Thomas Maugey,et al.  Encoder-Driven Inpainting Strategy in Multiview Video Compression , 2016, IEEE Transactions on Image Processing.

[27]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[28]  Ulf Jennehag,et al.  Scalable Coding of Plenoptic Images by Using a Sparse Set and Disparities , 2016, IEEE Transactions on Image Processing.

[29]  Yun Li,et al.  Coding of Focused Plenoptic Contents by Displacement Intra Prediction , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  Krzysztof Wegner,et al.  Immersive visual media — MPEG-I: 360 video, virtual navigation and beyond , 2017, 2017 International Conference on Systems, Signals and Image Processing (IWSSIP).

[31]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[33]  G. Lippmann.  Epreuves reversibles donnant la sensation du relief , 1908 .

[34]  Marek Domanski,et al.  Graph-based multiview depth estimation using segmentation , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[35]  M. Domanski,et al.  New coding technology for 3D video with depth maps as proposed for standardization within MPEG , 2012, 2012 19th International Conference on Systems, Signals and Image Processing (IWSSIP).

[36]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[37]  Anthony Vetro,et al.  Temporally consistent stereo matching using coherence function , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[38]  Luís Ducla Soares,et al.  HEVC-based 3D holoscopic video coding using self-similarity compensated prediction , 2016, Signal Process. Image Commun..

[39]  Jacob Benesty,et al.  An Acoustic MIMO Framework for Analyzing Microphone-Array Beamforming , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[40]  Andrew Lumsdaine,et al.  Focused plenoptic camera and rendering , 2010, J. Electronic Imaging.

[41]  Peter H. N. de With,et al.  Quality improving techniques in DIBR for free-viewpoint video , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[42]  Masayuki Tanimoto.  Overview of free viewpoint television , 2006, Signal Process. Image Commun..

[43]  Ju Liu,et al.  DIBR based view synthesis for free-viewpoint television , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[44]  Marek Domanski,et al.  Optimization of camera positions for free-navigation applications , 2016, 2016 International Conference on Signals and Electronic Systems (ICSES).

[45]  Houqiang Li,et al.  Joint multiview video plus depth coding , 2010, 2010 IEEE International Conference on Image Processing.

[47]  Toshiaki Fujii,et al.  FTV for 3-D Spatial Communication , 2012, Proceedings of the IEEE.

[48]  Oliver Schreer,et al.  Stereo analysis by hybrid recursive matching for real-time immersive video conferencing , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[49]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[50]  Ioannis Stamos,et al.  Integration of range and image sensing for photo-realistic 3D modeling , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[51]  Krzysztof Wegner,et al.  Coding of multiple video+depth using HEVC technology and reduced representations of side views and depth maps , 2012, 2012 Picture Coding Symposium.

[52]  Petros Daras,et al.  Immersive 3D Holoscopic Video System , 2013, IEEE MultiMedia.

[53]  Ying Chen,et al.  Overview of the Multiview and 3D Extensions of High Efficiency Video Coding , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[54]  Tom E. Bishop,et al.  Plenoptic depth estimation from multiple aliased views , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[55]  Carl J. Debono,et al.  Depth-based image processing for 3d video rendering applications , 2014, IWSSIP 2014 Proceedings.

[56]  Subhasis Chaudhuri,et al.  Disparity based compression technique for focused plenoptic images , 2014, ICVGIP '14.

[57]  Luís Ducla Soares,et al.  Locally linear embedding-based prediction for 3D holoscopic image coding using HEVC , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[58]  Marek Domanski,et al.  Adaptation of the 3D-HEVC coding tools to arbitrary locations of cameras , 2016, 2016 International Conference on Signals and Electronic Systems (ICSES).

[59]  Thomas Wiegand,et al.  3-D Video Representation Using Depth Maps , 2011, Proceedings of the IEEE.

[60]  Aljoscha Smolic,et al.  3D video objects for interactive applications , 2005, 2005 13th European Signal Processing Conference.

[61]  Ren Ng.  Fourier slice photography , 2005, ACM Trans. Graph..

[62]  Ying Chen,et al.  Overview of the MVC + D 3D video coding standard , 2014, J. Vis. Commun. Image Represent..

[63]  Oliver Schreer,et al.  Three-dimensional image processing in the future of immersive media , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[64]  Hideshi Yamada,et al.  Rendering for an Interactive 360 ◦ Light Field Display , 2007 .

[65]  Krzysztof Wegner,et al.  New results in free-viewpoint television systems for horizontal virtual navigation , 2016, 2016 IEEE International Conference on Multimedia and Expo (ICME).

[66]  Weisi Lin,et al.  Low-Complexity Depth Coding by Depth Sensitivity Aware Rate-Distortion Optimization , 2016, IEEE Transactions on Broadcasting.

[67]  Patrick Gioia,et al.  Efficient compression method for integral images using multi-view video coding , 2011, 2011 18th IEEE International Conference on Image Processing.

[68]  Heiko Schwarz,et al.  3D High-Efficiency Video Coding for Multi-View Video and Depth Data , 2013, IEEE Transactions on Image Processing.

[69]  Krzysztof Wegner,et al.  Estimation of temporally-consistent depth maps from video with reduced noise , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[70]  Arun N. Netravali,et al.  Digital Video: An introduction to MPEG-2 , 1996 .

[71]  Marek Domanski,et al.  Depth-based inter-view prediction of motion vectors for improved multiview video coding , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[72]  Detlev Marpe,et al.  3D video: Depth coding based on inter-component prediction of block partitions , 2012, 2012 Picture Coding Symposium.

[73]  Houqiang Li,et al.  Multiview-Video-Plus-Depth Coding Based on the Advanced Video Coding Standard , 2013, IEEE Transactions on Image Processing.

[74]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  S. Burak Gokturk,et al.  A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[76]  Li Zhang,et al.  Multiview and 3D Video Compression Using Neighboring Block Based Disparity Vectors , 2016, IEEE Transactions on Multimedia.

[77]  Zixiang Xiong,et al.  A gradient-based approach for interference cancelation in systems with multiple Kinect cameras , 2013, 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013).

[78]  Miska M. Hannuksela,et al.  Joint depth and texture filtering targeting MVD compression , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[79]  Peter Eisert,et al.  Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline , 2014, J. Vis. Commun. Image Represent..

[80]  Li Yu,et al.  A Joint Texture/Depth Edge-Directed Up-sampling Algorithm for Depth Map Coding , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[81]  Luís Ducla Soares,et al.  New HEVC prediction modes for 3D holoscopic video coding , 2012, 2012 19th IEEE International Conference on Image Processing.

[82]  Krzysztof Wegner,et al.  Intra Predictive Depth Map Coding Using Flexible Block Partitioning , 2015, IEEE Transactions on Image Processing.

[83]  Chin-Tser Huang,et al.  Signal Processing Applications in Network Intrusion Detection Systems , 2009, EURASIP Journal on Advances in Signal Processing.

[84]  Tomoyuki Ishida,et al.  Proposal of Tele-immersion System by the Fusion of Virtual Space and Real Space , 2010, 2010 13th International Conference on Network-Based Information Systems.

[85]  Toshiaki Fujii,et al.  View Generation with 3D Warping Using Depth Information for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[86]  Koichi Harada,et al.  View synthesis with depth information based on graph cuts for FTV , 2013, The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision.

[87]  Òscar Divorra Escoda,et al.  Depth estimation based on multiview matching with depth/color segmentation and memory efficient Belief Propagation , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[88]  Margrit Gelautz,et al.  Graph-based surface reconstruction from stereo pairs using image segmentation , 2005 .

[89]  Yo-Sung Ho,et al.  High-quality multi-view depth generation using multiple color and depth cameras , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[90]  Ying Chen,et al.  The Emerging MVC Standard for 3D Video Services , 2008, EURASIP J. Adv. Signal Process..

[91]  Nuno M. M. Rodrigues,et al.  Compressing depth maps using multiscale recurrent pattern image coding , 2010 .

[92]  Krzysztof Wegner,et al.  A practical approach to acquisition and processing of free viewpoint video , 2015, 2015 Picture Coding Symposium (PCS).

[93]  Adrian Hilton,et al.  Proj ective Surface Refinement for Free-Viewpoint Video , 2006 .

[94]  Shao-Yi Chien,et al.  Point-based model construction for free-viewpoint TV , 2013, 2013 IEEE Third International Conference on Consumer Electronics ¿ Berlin (ICCE-Berlin).

[95]  Krzysztof Wegner,et al.  Fast view synthesis using platelet-based depth representation , 2014, IWSSIP 2014 Proceedings.

[96]  Danillo B. Graziosi,et al.  Depth assisted compression of full parallax light fields , 2015, Electronic Imaging.

[97]  Detlev Marpe,et al.  Depth Intra Coding for 3D Video Based on Geometric Primitives , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[98]  Krzysztof Wegner,et al.  Methods of high efficiency compression for transmission of spatial representation of motion scenes , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).