Display-Independent 3D-TV Production and Delivery Using the Layered Depth Video Format

This paper discusses an approach to 3D-Television that is based on the Layered Depth Video (LDV) format. The LDV format contains explicit depth and occlusion information, allowing for the generation of novel viewpoints for stereoscopic and auto-stereoscopic multi-view displays. Thus, the format is effectively invariant to the display type and also allows the depth impression to be easily changed to best meet viewers' preferences for visual comfort. The major aspects of a content delivery chain based on the LDV format are discussed in this paper. The requirements placed on data acquisition are introduced, and a multi-camera system, which is well suited for LDV compliant data capture, is presented. Also discussed is the conversion of different input data streams, like standard stereo videos, multi-view data supplemented by depth data, and videos from wide baseline setups, to the LDV format. Moreover, the advantages of the LDV format in editing and mixing are examined. The paper also presents a transmission system based on currently available coding and transmission standards. Optimization of the bandwidth via different approaches to the compression of the LDV signal is analyzed, and the results of conducted experiments in this field are discussed. Finally, the aspects of perceptual human factors for the proper evaluation of 3D-TV services and the implemented LDV system are examined. This contribution reflects the efforts of the EU-funded project 3D4YOU to unify all aspects of 3D-TV production.

[1]  Reinhard Koch,et al.  Dense Depth Maps from Low Resolution Time-of-Flight Depth and High Resolution Color Views , 2009, ISVC.

[2]  R. Klein Gunnewiek,et al.  Robust image, depth, and occlusion generation from uncalibrated stereo , 2008, Electronic Imaging.

[3]  R. Koch,et al.  CALIBRATION OF A PMD-CAMERA USING A PLANAR CALIBRATION PATTERN TOGETHER WITH A MULTI-CAMERA SETUP , 2008 .

[4]  Marc Pollefeys,et al.  An evolutionary and optimised approach on 3D-TV , 2002 .

[5]  Aljoscha Smolic,et al.  Nonlinear disparity mapping for stereoscopic 3D , 2010, ACM Trans. Graph..

[6]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[7]  Bernard Mendiburu,et al.  3D Movie Making: Stereoscopic Digital Cinema from Script to Screen , 2009 .

[8]  In-So Kweon,et al.  Adaptive Support-Weight Approach for Correspondence Search , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Yu Huang,et al.  A layered method of visibility resolving in depth image-based rendering , 2008, 2008 19th International Conference on Pattern Recognition.

[10]  R. Klein Gunnewiek,et al.  Coherent spatial and temporal occlusion generation , 2009, Electronic Imaging.

[11]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[12]  Pjh Pieter Seuntiëns,et al.  Visual experience of 3D TV , 2006 .

[13]  Oliver Schreer,et al.  Stereo analysis by hybrid recursive matching for real-time immersive video conferencing , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[15]  Frederik Zilly,et al.  Adaptive cross-trilateral depth map filtering , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[16]  Alexander Toet,et al.  Visual comfort of binocular and 3D displays , 2004 .

[17]  Alice Biber,et al.  Time-of-flight range imaging with a custom solid state image sensor , 1999, Industrial Lasers and Inspection.

[18]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[19]  James E. Cutting,et al.  HIGH-PERFORMANCE COMPUTING AND HUMAN VISION I , 2002 .

[20]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[21]  A. Frick,et al.  3D-TV LDV content generation with a hybrid ToF-multicamera RIG , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[22]  Atsuo Murata,et al.  Evaluation of visual fatigue during VDT tasks , 2000, Smc 2000 conference proceedings. 2000 ieee international conference on systems, man and cybernetics. 'cybernetics evolving to systems, humans, organizations, and their complex interactions' (cat. no.0.

[23]  David M. Hoffman,et al.  Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. , 2008, Journal of vision.

[24]  Jens-Rainer Ohm,et al.  A realtime hardware system for stereoscopic videoconferencing with viewpoint adaptation , 1998, Signal Process. Image Commun..

[25]  Ruigang Yang,et al.  How Far Can We Go with Local Optimization in Real-Time Stereo Matching , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[26]  Oliver Grau,et al.  Stereoscopic 3D Sports Content without Stereo Rigs , 2010 .

[27]  Atsuo Murata,et al.  Proposal of an Index to Evaluate Visual Fatigue Induced During Visual Display Terminal Tasks , 2001, Int. J. Hum. Comput. Interact..

[28]  Wijnand A. IJsselsteijn,et al.  A survey of perceptual evaluations and requirements of three-dimensional TV , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[30]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[31]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[32]  Mtm Marc Lambooij,et al.  Visual Discomfort and Visual Fatigue of Stereoscopic Displays: A Review , 2009 .

[33]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[34]  Graham A. Thomas,et al.  Real-time camera tracking using sports pitch markings , 2007, Journal of Real-Time Image Processing.

[35]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[36]  Aljoscha Smolic,et al.  An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution , 2009, 2009 Picture Coding Symposium.

[37]  Marcus Barkowsky,et al.  NEW REQUIREMENTS OF SUBJECTIVE VIDEO QUALITY ASSESSMENT METHODOLOGIES FOR 3DTV , 2010 .

[38]  Rene Klein Gunnewiek,et al.  Options for a new efficient, compatible, flexible 3D standard , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[39]  Paul Kerbiriou,et al.  Looking for an adequate quality criterion for depth coding , 2010, Electronic Imaging.

[40]  A. Frick,et al.  Generation of 3D-TV LDV-content with Time-Of-Flight Camera , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[41]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[43]  Neil A. Dodgson,et al.  Autostereoscopic 3D displays , 2005, Computer.

[44]  Manuel Menezes de Oliveira Neto,et al.  Fast Digital Image Inpainting , 2001, VIIP.

[45]  Reinhard Koch,et al.  Real-time preview for layered depth video in 3D-TV , 2010, Photonics Europe.

[46]  Stephan Reichelt,et al.  Depth cues in human visual perception and their realization in 3D displays , 2010, Defense + Commercial Sensing.

[47]  Jörn Ostermann,et al.  AN ASSESSMENT OF 3DTV TECHNOLOGIES , 2006 .