Real-time vision-based camera tracking for augmented reality applications

Augmentedreality deals with the problem of dynamically augmenting or enhancing (images or live video of) the real world with computer generated data (e.g., graphics of virtual objects). This poses two major problems: (a) determining the precise alignment of real and virtual coordinate frames for overlay, and (b) capturingthe 3D environmentincluding camera and object motions. The latter is important for interactive augmented reality applications where users can interact with both real and virtual objects. Here we address the problem of accurately tracking the 3D motion of a monocular camera in a known 3D environment and dynamically estimating the 3D camera location. We utilize fully automated landmark-basedcamera calibration to initialize the motion estimation and employ extended Kalman filter techniques to track landmarksand to estimate the camera location. The implementation of our approach has been proven to be efficient and robust and our system successfully tracks in real-time at approximately 10 Hz.

[1]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[2]  Takeo Kanade,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995, CVRMed.

[3]  Alain Fournier.  Illumination Problems in Computer Augmented Reality , 1994 .

[4]  David E. Breen,et al.  Annotating Real-World Objects Using Augmented Reality , 1995, Computer Graphics.

[5]  .. McGlone,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995 .

[6]  Katsushi Ikeuchi,et al.  Numerical Shape from Shading and Occluding Boundaries , 1981, Artif. Intell..

[7]  J. W. Humberston.  Classical mechanics , 1980, Nature.

[8]  M. A. Naimark.  THE REPRESENTATIONS OF THE THREE-DIMENSIONAL ROTATION GROUP , 1964 .

[9]  J. Stuelpnagel.  On the Parametrization of the Three-Dimensional Rotation Group , 1964 .

[10]  Donald B. Gennery,et al.  Tracking Known Three-Dimensional Objects , 1982, AAAI.

[11]  Gilad Adiv,et al.  Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  David E. Breen,et al.  Distributed Augmented Reality for Collaborative Design Applications , 1995, Comput. Graph. Forum.

[13]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[14]  Zhengyou Zhang.  Estimating Motion and Structure from Correspondences of Line Segments between Two Perspective Images , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  W. Eric L. Grimson,et al.  An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[16]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[17]  Yaakov Bar-Shalom,et al.  Estimation and Tracking: Principles, Techniques, and Software , 1993 .

[18]  Mary C. Whitton,et al.  Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies , 1996, SIGGRAPH.

[19]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.

[20]  J. P. Mellor.  Realtime Camera Calibration for Enhanced Reality Visualization , 1995, CVRMed.

[21]  Junior Barrera,et al.  Mathematical morphology toolbox for the KHOROS system , 1994, Optics & Photonics.

[22]  R. Chellappa,et al.  Recursive 3-D motion estimation from a monocular image sequence , 1990 .

[23]  A.H. Haddad,et al.  Applied optimal estimation , 1976, Proceedings of the IEEE.

[24]  John Oliensis,et al.  A global algorithm for shape from shading , 1993, 1993 (4th) International Conference on Computer Vision.

[25]  Juyang Weng,et al.  Calibration of stereo cameras using a non-linear distortion model (CCD sensory) , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[26]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[27]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[28]  Olivier Faugeras,et al.  3D Dynamic Scene Analysis , 1992 .

[29]  David E. Breen,et al.  Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System , 1995, IEEE Trans. Vis. Comput. Graph..

[30]  William E. Lorensen,et al.  Enhancing reality in the operating room , 1993, Proceedings Visualization '93.

[31]  Hormoz Shariat,et al.  Motion Estimation with More than Two Frames , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  K. Kanatani.  Group-Theoretical Methods in Image Understanding , 1990 .

[33]  Christine Chevrier,et al.  Composing Computer-generated Images and Video Films: An Application for Visual Assessment in Urban Environments , 1995, Computer Graphics.

[34]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[35]  Kiriakos N. Kutulakos,et al.  Affine object representations for calibration-free augmented reality , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[36]  David E. Breen,et al.  Interactive Occlusion and Automatic Object Placement for Augmented Reality , 1996, Comput. Graph. Forum.

[37]  Shumin Zhai,et al.  Applications of augmented reality for human-robot communication , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).