3D reconstruction of stereo images for interaction between real and virtual worlds

Mixed reality is different from the virtual reality in that users can feel immersed in a space which is composed of not only virtual but also real objects. Thus, it is essential to realize seamless integration and interaction of the virtual and real worlds. We need depth information of the real scene to synthesize the real and virtual objects. We propose a two-stage algorithm to find smooth and precise disparity vector fields with sharp object boundaries in a stereo image pair for depth estimation. Hierarchical region-dividing disparity estimation increases the efficiency and the reliability of the estimation process, and a shape-adaptive window provides high reliability of the fields around the object boundary region. At the second stage, the vector fields are regularized with a energy model which produces smooth fields while preserving their discontinuities resulting from the object boundaries. The vector fields are used to reconstruct 3D surface of the real scene. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various kinds of images, and synthesized 3D models produce natural space where the virtual objects interact with the real world as if they are in the same world.

[1]  Joachim Weickert,et al.  A Review of Nonlinear Diffusion Filtering , 1997, Scale-Space.

[2]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[3]  Takeo Kanade,et al.  A stereo machine for video-rate dense depth mapping and its new applications , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  Marie-Odile Berger.  Resolving occlusion in augmented reality: a contour based approach without 3D reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  T. Poggio,et al.  A generalized ordering constraint for stereo correspondence , 1984 .

[6]  Emanuele Trucco,et al.  Introductory techniques for 3-D computer vision , 1998 .

[7]  Reinhard Männer,et al.  Calculating Dense Disparity Maps from Color Stereo Images, an Efficient Implementation , 2004, International Journal of Computer Vision.

[8]  David E. Breen,et al.  Interactive Occlusion and Collision of Real and Virtual Objects in Augmented Reality , 2000 .

[9]  Steve Benford,et al.  Understanding and constructing shared spaces with mixed-reality boundaries , 1998, TCHI.

[10]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[11]  David E. Breen,et al.  Confluence of Computer Vision and Interactive Graphies for Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[12]  Pierre Kornprobst,et al.  Mathematical problems in image processing - partial differential equations and the calculus of variations , 2010, Applied mathematical sciences.

[13]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[14]  G. Iddan,et al.  3D IMAGING IN THE STUDIO (AND ELSEWHERE...) , 2001 .

[15]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[16]  Naokazu Yokoya,et al.  A stereoscopic video see-through augmented reality system based on real-time vision-based registration , 2000, Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048).

[17]  Claes Johnson.  Numerical solution of partial differential equations by the finite element method , 1988 .

[18]  Takeo Kanade,et al.  A Cooperative Algorithm for Stereo Matching and Occlusion Detection , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Carlo Tomasi,et al.  Depth Discontinuities by Pixel-to-Pixel Stereo , 1999, International Journal of Computer Vision.

[20]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[21]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[22]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[23]  Kwanghoon Sohn,et al.  Hierarchical depth estimation for image synthesis in mixed reality , 2003, IS&T/SPIE Electronic Imaging.