Hierarchical depth estimation for image synthesis in mixed reality

Mixed reality is different from the virtual reality in that users can feel immersed in a space which is composed of not only virtual but also real objects. Thus, it is essential to realize seamless integration and mutual occlusion of the virtual and real worlds. Therefore, we need depth information of the real scene to perform the synthesis. We propose the depth estimation algorithm with sharp object boundaries for mixed reality system based on hierarchical disparity estimation. Initial disparity vectors are obtained from downsampled stereo images using region-dividing disparity estimation technique. Then, background region is detected and flattened. With these initial vectors, dense disparities are estimated and regularized with shape-adaptive window in full resolution images. Finally, depth values are calculated by stereo geometry and camera parameters. As a result, virtual objects can be mixed into the image of real world by comparing the calculated depth values with the depth information of generated virtual objects. Experimental results show that occlusion between the virtual and real objects are correctly established with sharp boundaries in the synthesized images, so that user can observe the mixed scene with considerably natural sensation.

[1]  W. Eric L. Grimson,et al.  Computational Experiments with a Feature Based Stereo Algorithm , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Steve Benford,et al.  Understanding and constructing shared spaces with mixed-reality boundaries , 1998, TCHI.

[3]  Aggelos K. Katsaggelos,et al.  Dense Disparity Estimation with a Divide-and-Conquer Disparity Space Image Technique , 1999, IEEE Trans. Multim..

[4]  R.E.H. Franich.  Disparity estimation in stereoscopic digital images , 1996 .

[5]  Rachid Deriche,et al.  Dense Disparity Map Estimation Respecting Image Discontinuities: A PDE and Scale-Space BasedApproach , 2002, MVA.

[6]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[7]  Amnon Shashua,et al.  Projective depth: A geometric invariant for 3D reconstruction from two perspective/orthographic views and for visual recognition , 1993, 1993 (4th) International Conference on Computer Vision.

[8]  Panos Liatsis,et al.  Hybrid symbiotic genetic optimisation for robust edge-based stereo correspondence , 2001, Pattern Recognit..

[9]  Ebroul Izquierdo.  Disparity/segmentation analysis: matching with an adaptive window and depth-driven segmentation , 1999, IEEE Trans. Circuits Syst. Video Technol..

[10]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[11]  Rachid Deriche,et al.  Dense Depth Map Reconstruction: A Minimization and Regularization Approach which Preserves Discontinuities , 1996, ECCV.

[12]  G. Iddan,et al.  3D IMAGING IN THE STUDIO (AND ELSEWHERE...) , 2001 .

[13]  Takeo Kanade,et al.  A stereo machine for video-rate dense depth mapping and its new applications , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.