Real-time composition of stereo images for video see-through augmented reality

This paper describes a method of stereo image composition for video see-through augmented reality. In order to implement an augmented reality system, we must acquire the position and orientation of the user's viewpoint to display the composed image maintaining correct registration of real and virtual worlds. All the procedures must be done in real-time. We have built a prototype augmented reality system that adopts the combination of a vision-based tracking technique and a video see-through head mounted display (HMD). Display-timing is synchronized between the real and virtual environments, so that an alignment error is reduced. The system calculates camera parameters from three markers among which physical relationships are unknown in image sequences captured by a pair of stereo cameras mounted on the HMD. In addition, the user's hands are regarded as real-world objects that may occlude virtual objects; the system estimates the depth of hands in images and generates a composed image maintaining consistent occlusions between the hands and virtual objects.

[1]  Takashi Okuma.  Real-Time Camera Parameter Estimation for 3-D Annotation on a Wearable Vision System , 2001 .

[2]  Takeo Kanade,et al.  A stereo machine for video-rate dense depth mapping and its new applications , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[4]  Naokazu Yokoya,et al.  A stereoscopic video see-through augmented reality system based on real-time vision-based registration , 2000, Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048).

[5]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[6]  Katsuhiko Sakaue,et al.  3-D annotation of images captured from a wearer's camera based on object recognition , 2001 .

[7]  Takeo Kanade,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995, CVRMed.

[8]  Katsuhiko Sakaue,et al.  Real-time camera parameter estimation from images for a mixed reality system , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[9]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[10]  David E. Breen,et al.  Confluence of Computer Vision and Interactive Graphies for Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[11]  Katsuhiko Sakaue,et al.  Real-Time Camera Parameter Estimation from Images for a Wearable Vision System , 2000, MVA.

[12]  Jannick P. Rolland,et al.  Video see-through design for merging of real and virtual environments , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[13]  Bill Ross,et al.  A practical stereo vision system , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Mary C. Whitton,et al.  Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies , 1996, SIGGRAPH.

[15]  Naokazu Yokoya,et al.  An augmented reality system using a real-time vision based registration , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[16]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[17]  M. Kourogi,et al.  The Hand-mouse: A Human Interface Suitable for Augmented Reality Environments Enabled by Visual Wearables , 2001 .

[18]  David E. Breen,et al.  Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System , 1995, IEEE Trans. Vis. Comput. Graph..

[19]  Naokazu Yokoya,et al.  Stereo Vision Based Video See-through Mixed Reality , 1999 .

[20]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[21]  Katsuhiko Sakaue,et al.  VizWear: Toward Human-Centered Interaction through Wearable Vision and Visualization , 2001, IEEE Pacific Rim Conference on Multimedia.

[22]  Jun Rekimoto,et al.  Matrix: a realtime object identification and registration method for augmented reality , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[23]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.