Generation of multi-view video using a fusion camera system for 3D displays

In this paper, we present a fusion camera system combining one time-of-flight depth camera and two video cameras to generate multi-view video sequences. In order to obtain the multi-view video using the fusion camera system for 3D displays, we capture a stereo video using a pair of video cameras and a single view depth video with the depth camera. After performing a 3D warping operation for the depth video to obtain an initial depth map at each viewpoint, we refine it using segment-based stereo matching. To reduce mismatched depth values along object boundaries, we detect moving objects using color difference between frames. Finally, we recompute the depth value of each pixel in every segment using stereo matching with a new cost function. Experimental results show that the proposed fusion system produces multi-view video sequences with accurate depth maps, especially along the boundaries of objects. Therefore, it is suitable for generating more natural 3D views for 3D displays than previous works.

[1]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  G. Iddan,et al.  3D IMAGING IN THE STUDIO (AND ELSEWHERE...) , 2001 .

[3]  M. Kawakita,et al.  HDTV Axi-Vision Camera , 2002 .

[4]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Yuko Yamanouchi,et al.  High-definition real-time depth-mapping TV camera: HDTV Axi-Vision Camera. , 2004, Optics express.

[7]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[8]  Aljoscha Smolic,et al.  3DAV exploration of video-based rendering technology in MPEG , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[9]  S. Burak Gokturk,et al.  A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[10]  M. Levoy,et al.  Automatic Color Calibration for Large Camera Arrays , 2005 .

[11]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[12]  Chung-Hyun Ahn,et al.  Three-dimensional scene reconstruction using multiview images and depth camera , 2005, IS&T/SPIE Electronic Imaging.

[13]  CHRISTOPH FEHN,et al.  Interactive 3-DTV-Concepts and Key Technologies , 2006, Proceedings of the IEEE.

[14]  Yo-Sung Ho,et al.  Three-dimensional natural video system based on layered representation of depth maps , 2006, IEEE Transactions on Consumer Electronics.

[15]  C. Fehn,et al.  Interactive 3-DTV-Concepts and Key Technologies , 2006 .

[16]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[17]  Seung-man Kim,et al.  Depth Image Processing Technique for Representing Human Actors in 3DTV using Single Depth Camera , 2007, 2007 3DTV Conference.

[18]  Young Min Kim,et al.  Design and calibration of a multi-view TOF sensor fusion system , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[19]  Sebastian Thrun,et al.  High-quality scanning using time-of-flight depth superresolution , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Jong-Il Park,et al.  Fast view synthesis using GPU for 3D display , 2008, IEEE Transactions on Consumer Electronics.

[22]  Yo-Sung Ho,et al.  Geometrical compensation for multi-view video in multiple camera array , 2008, 2008 50th International Symposium ELMAR.

[23]  Yo-Sung Ho,et al.  3-D video generation using hybrid camera system , 2009, IMMERSCOM.

[24]  Aiqi Wang,et al.  A Simple Method of Radial Distortion Correction with Centre of Distortion Estimation , 2009, Journal of Mathematical Imaging and Vision.