A flexible architecture for multi-view 3DTV based on uncalibrated cameras

This paper presents a novel flexible architecture for 3DTV based on multiple uncalibrated cameras. The proposed signal representation improves the interactivity of dense point-based methods, making them appropriate for modeling the scene semantics and free-viewpoint 3DTV applications. The main concern is to address the shortcomings of depth image-based 3D video systems for free-viewpoint visualization, and to provide an efficient implementation of the rendering part which is computationally intensive as well potentially determine the view quality. Novel rendering algorithms are added that specifically aim at solving the rendering artifacts, and sampling issues encountered in wide baseline extensions and arbitrary camera movements. To optimize the process, a ''selective'' warping technique is proposed that takes the advantage of temporal coherence to reduce the computational overhead. Performance is illustrated on challenging videos to prove the suitability and flexibility of the architecture for advanced 3DTV systems.

[1]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[2]  Shang-Hong Lai,et al.  Spatio-Temporally Consistent Novel View Synthesis Algorithm From Video-Plus-Depth Sequences for Autostereoscopic Displays , 2011, IEEE Transactions on Broadcasting.

[3]  Euee S. Jang,et al.  An introduction to the MPEG-4 animation framework eXtension , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Frederik Zilly,et al.  3D capturing using multi-camera rigs, real-time depth estimation and depth-based content creation for multi-view and light-field auto-stereoscopic displays , 2012, SIGGRAPH '12.

[5]  Jana Kosecka,et al.  Multi-view Superpixel Stereo in Urban Environments , 2010, International Journal of Computer Vision.

[6]  Richard Szeliski,et al.  Piecewise planar stereo for image-based rendering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Qionghai Dai,et al.  A Point-Cloud-Based Multiview Stereo Algorithm for Free-Viewpoint Video , 2010, IEEE Transactions on Visualization and Computer Graphics.

[8]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[9]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[10]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[11]  Toshiaki Fujii,et al.  View Generation with 3D Warping Using Depth Information for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[12]  Namho Hur,et al.  Depth-image-based rendering for 3DTV service over T-DMB , 2009, Signal Process. Image Commun..

[13]  A. N. Rajagopalan,et al.  Depth Estimation with a Practical Camera , 2009, BMVC.

[14]  Richard Szeliski,et al.  Boundary Matting for View Synthesis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[15]  Jae-Hean Kim.  Linear stratified approach for 3D modelling and calibration using full geometric constraints , 2009, CVPR.

[16]  Pascal Fua,et al.  On benchmarking camera calibration and multi-view stereo for high resolution imagery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Michael Schmeing,et al.  Depth Image Based Rendering: A faithful approach for the disocclusion problem , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[18]  Hideo Saito,et al.  A Novel Inpainting-Based Layered Depth Video for 3DTV , 2011, IEEE Transactions on Broadcasting.

[19]  Margaret H. Pinson,et al.  A new standardized method for objectively measuring video quality , 2004, IEEE Transactions on Broadcasting.

[20]  Ken Shoemake,et al.  Animating rotation with quaternion curves , 1985, SIGGRAPH.

[21]  C. Guillemot,et al.  Joint projection filling method for occlusion handling in Depth-Image-Based Rendering , 2011 .

[22]  Thomas Wiegand,et al.  Depth Image-Based Rendering With Advanced Texture Synthesis for 3-D Video , 2010, IEEE Transactions on Multimedia.

[23]  Qionghai Dai,et al.  A Flexible Client-Driven 3DTV System for Real-Time Acquisition, Transmission, and Display of Dynamic Scenes , 2009, EURASIP J. Adv. Signal Process..

[24]  Andrew Blake,et al.  Geodesic star convexity for interactive image segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  Jan-Michael Frahm,et al.  Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Anders Heyden,et al.  Scene point constraints in camera auto-calibration: an implementational perspective , 2005, Image Vis. Comput..

[27]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[28]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[29]  Xiaoyi Jiang,et al.  Time-consistency of disocclusion filling algorithms in Depth Image Based Rendering , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[30]  D.M. Mount,et al.  An Efficient k-Means Clustering Algorithm: Analysis and Implementation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Long Quan,et al.  A quasi-dense approach to surface reconstruction from uncalibrated images , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Kai Luo,et al.  Depth-aided inpainting for disocclusion restoration of multi-view images using depth-image-based rendering , 2009 .

[33]  Antonio Torralba,et al.  Depth Estimation from Image Structure , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[35]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[36]  Andrew Zisserman,et al.  Combining scene and auto-calibration constraints , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[37]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[38]  Masayuki Tanimoto,et al.  Multiview Imaging and 3DTV , 2007, IEEE Signal Processing Magazine.

[39]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[40]  Peter H. N. de With,et al.  Free-viewpoint depth image based rendering , 2010, J. Vis. Commun. Image Represent..

[41]  Aljoscha Smolic,et al.  Scene Representation Technologies for 3DTV—A Survey , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[42]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[43]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[44]  Toshiaki Fujii.  Ray-space acquisition system for 3DTV: 100-camera and ray-based acquisition systems , 2012, Defense, Security, and Sensing.

[45]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[46]  Alexei A. Efros,et al.  Geometric context from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[47]  Pei-Jun Lee,et al.  Nongeometric Distortion Smoothing Approach for Depth Map Preprocessing , 2011, IEEE Transactions on Multimedia.

[48]  A. N. Rajagopalan,et al.  Range map superresolution-inpainting, and reconstruction from sparse data , 2012, Comput. Vis. Image Underst..

[49]  Liang Zhang,et al.  Stereoscopic image generation based on depth images for 3D TV , 2005, IEEE Transactions on Broadcasting.

[50]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.