3D reconstruction from stereo images for interactions between real and virtual objects

The 3D reconstruction algorithm in a stereo image pair for realizing mutual occlusion and interactions between the real and virtual world in an image synthesis is proposed. A two-stage algorithm, consisting of disparity estimation and regularization is used to locate a smooth and precise disparity vector. The hierarchical disparity estimation technique increases the efficiency and reliability of the estimation process, and edge-preserving disparity field regularization produces smooth disparity fields while preserving discontinuities that result from object boundaries. Depth information concerning the real scene is then recovered from the estimated disparity fields by stereo camera geometry. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various types of images, and the reconstructed 3D model produces a natural space in which the real world and virtual objects interact with each other as if they were in the same world.

[1]  Takeo Kanade,et al.  A stereo machine for video-rate dense depth mapping and its new applications , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Joachim Weickert,et al.  Reliable Estimation of Dense Optical Flow Fields with Large Displacements , 2000, International Journal of Computer Vision.

[3]  Marie-Odile Berger.  Resolving occlusion in augmented reality: a contour based approach without 3D reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  David E. Breen,et al.  Interactive Occlusion and Collision of Real and Virtual Objects in Augmented Reality , 2000 .

[5]  Woontack Woo,et al.  Image-based panoramic 3D virtual environment using rotating two multiview cameras , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[6]  Steve Benford,et al.  Understanding and constructing shared spaces with mixed-reality boundaries , 1998, TCHI.

[7]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[8]  Yoonsik Choe,et al.  Disparity estimation using a region-dividing technique and energy-based regularization , 2004 .

[9]  Francisco Abad,et al.  Integrating synthetic objects into real scenes , 2003, Comput. Graph..

[10]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Naokazu Yokoya,et al.  A stereoscopic video see-through augmented reality system based on real-time vision-based registration , 2000, Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048).

[12]  Claes Johnson.  Numerical solution of partial differential equations by the finite element method , 1988 .

[13]  Jens-Rainer Ohm,et al.  Image-based rendering and 3D modeling: A complete framework , 2000, Signal Process. Image Commun..

[14]  G. Iddan,et al.  3D IMAGING IN THE STUDIO (AND ELSEWHERE...) , 2001 .

[15]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion with a real-time stereovision system , 2001, Comput. Graph..

[16]  Emanuele Trucco,et al.  Introductory techniques for 3-D computer vision , 1998 .

[17]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[18]  T. Poggio,et al.  A generalized ordering constraint for stereo correspondence , 1984 .

[19]  Kwanghoon Sohn,et al.  Hierarchical disparity estimation with energy-based regularization , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[20]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[21]  Tim J. Dennis,et al.  Epipolar line estimation and rectification for stereo image pairs , 1996, IEEE Trans. Image Process..

[22]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  David E. Breen,et al.  Confluence of Computer Vision and Interactive Graphies for Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[24]  Takeo Kanade,et al.  A Cooperative Algorithm for Stereo Matching and Occlusion Detection , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Pierre Kornprobst,et al.  Mathematical problems in image processing - partial differential equations and the calculus of variations , 2010, Applied mathematical sciences.

[26]  Carlo Tomasi,et al.  Depth Discontinuities by Pixel-to-Pixel Stereo , 1999, International Journal of Computer Vision.

[27]  J. Hesser,et al.  Calculating Dense Disparity Maps from Color Stereo Images, an Efficient Implementation , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[28]  M.Ebroul Izquierdo.  Stereo image analysis for multi-viewpoint telepresence applications , 1998 .

[29]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[30]  Reinhard Männer,et al.  Calculating Dense Disparity Maps from Color Stereo Images, an Efficient Implementation , 2004, International Journal of Computer Vision.

[31]  Kwanghoon Sohn,et al.  Hierarchical depth estimation for image synthesis in mixed reality , 2003, IS&T/SPIE Electronic Imaging.

[32]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[33]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion of real and virtual environments , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[34]  Joachim Weickert,et al.  A Review of Nonlinear Diffusion Filtering , 1997, Scale-Space.

[35]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.