Spatial and temporal up-conversion technique for depth video

This paper proposes a novel framework for up-conversion of depth video resolution both in spatial and in time domain. Time-of-Flight (TOF) sensors are widely used in computer vision fields. Although TOF sensors provide depth video in real time, there are some problems in a sense that it provides a low resolution and a low frame-rate depth video. We propose a cheaper solution that enhances depth video obtained by TOF sensor by combining it with CCD camera. The proposed method provides high quality video as a cheaper solution for low resolution, and low frame-rate depth video. It is useful when depth video is used in various applications such as 3DTV, free-view TV, teleconference system. High-quality depth video can be obtained by motion compensated frame interpolation (MCFI) and extended Joint Bilateral Upsampling (JBU). Experimental results show that depth video obtained by the proposed method has satisfactory quality.

[1]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Ruigang Yang,et al.  Fusion of time-of-flight depth and stereo for high accuracy depth maps , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Minglun Gong,et al.  Fast stereo matching using reliability-based dynamic programming and consistency constraints , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[4]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[5]  Chang-Su Kim,et al.  Motion-Compensated Frame Interpolation Using Bilateral Motion Estimation and Adaptive Overlapped Block Motion Compensation , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[6]  Truong Q. Nguyen,et al.  A Multistage Motion Vector Processing Method for Motion-Compensated Frame Interpolation , 2008, IEEE Transactions on Image Processing.

[7]  Frédo Durand,et al.  A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach , 2006, ECCV.

[8]  Robert L. Stevenson,et al.  Super-resolution from image sequences-a review , 1998, 1998 Midwest Symposium on Circuits and Systems (Cat. No. 98CB36268).

[9]  Sebastian Thrun,et al.  High-quality scanning using time-of-flight depth superresolution , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[10]  Michael T. Orchard,et al.  Overlapped block motion compensation: an estimation-theoretic approach , 1994, IEEE Trans. Image Process..

[11]  G. Iddan,et al.  3D IMAGING IN THE STUDIO (AND ELSEWHERE...) , 2001 .

[12]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Lucas J. van Vliet,et al.  Recursive implementation of the Gaussian filter , 1995, Signal Process..

[14]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[15]  Dani Lischinski,et al.  Joint bilateral upsampling , 2007, ACM Trans. Graph..

[16]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[17]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Miao Liao,et al.  High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).