Dynamic depth recovery from multiple synchronized video streams

This paper addresses the problem of extracting depth information of non-rigid dynamic 3D scenes from multiple synchronized video streams. Three main issues are discussed in this context. (i) temporally consistent depth estimation, (ii) sharp depth discontinuity estimation around object boundaries, and (iii) enforcement of the global visibility constraint. We present a framework in which the scene is modeled as a collection of 3D piecewise planar surface patches induced by color based image segmentation. This representation is continuously estimated using an incremental formulation in which the 3D geometric, motion, and global visibility constraints are enforced over space and time. The proposed algorithm optimizes a cost function that incorporates the spatial color consistency constraint and a smooth scene motion model.

[1]  Long Quan,et al.  Edge-constrained joint view triangulation for image interpolation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[2]  Ingemar J. Cox,et al.  A maximum-flow formulation of the N-camera stereo correspondence problem , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[3]  Jake K. Aggarwal,et al.  Structure from stereo-a review , 1989, IEEE Trans. Syst. Man Cybern..

[4]  Keith J. Hanna,et al.  Combining stereo and motion analysis for direct estimation of scene structure , 1993, 1993 (4th) International Conference on Computer Vision.

[5]  Hai Tao,et al.  Object Tracking with Bayesian Estimation of Dynamic Layer Representations , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Gilad Adiv,et al.  Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Rama Chellappa,et al.  3-D Motion Estimation Using a Sequence of Noisy Stereo Images: Models, Estimation, and Uniqueness Results , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Zhengyou Zhang,et al.  A Progressive Scheme for Stereo Matching , 2000, SMILE.

[10]  Hai Tao,et al.  Dynamic layer representation with applications to tracking , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[11]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[12]  Olivier Faugeras,et al.  3D Dynamic Scene Analysis , 1992 .

[13]  Richard Szeliski,et al.  A multi-view approach to motion and stereo , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[14]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[15]  Kiriakos N. Kutulakos,et al.  Multi-view scene capture by surfel sampling: from video streams to non-rigid 3D motion, shape and reflectance , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[16]  Hai Tao,et al.  Global matching criterion and color segmentation based stereo , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[17]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Olivier Faugeras,et al.  3D Dynamic Scene Analysis: A Stereo Based Approach , 1992 .

[19]  Ye Zhang,et al.  Integrated 3D scene flow and structure recovery from multiview image sequences , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).