Registration using natural features for augmented reality systems

Registration is one of the most difficult problems in augmented reality (AR) systems. In this paper, a simple registration method using natural features based on the projective reconstruction technique is proposed. This method consists of two steps: embedding and rendering. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In rendering, the Kanade-Lucas-Tomasi (KLT) feature tracker is used to track the natural feature correspondences in the live video. The natural features that have been tracked are used to estimate the corresponding projective matrix in the image sequence. Next, the projective reconstruction technique is used to transfer the four specified points to compute the registration matrix for augmentation. This paper also proposes a robust method for estimating the projective matrix, where the natural features that have been tracked are normalized (translation and scaling) and used as the input data. The estimated projective matrix will be used as an initial estimate for a nonlinear optimization method that minimizes the actual residual errors based on the Levenberg-Marquardt (LM) minimization method, thus making the results more robust and stable. The proposed registration method has three major advantages: 1) It is simple, as no predefined fiducials or markers are used for registration for either indoor and outdoor AR applications. 2) It is robust, because it remains effective as long as at least six natural features are tracked during the entire augmentation, and the existence of the corresponding projective matrices in the live video is guaranteed. Meanwhile, the robust method to estimate the projective matrix can obtain stable results even when there are some outliers during the tracking process. 3) Virtual objects can still be superimposed on the specified areas, even if some parts of the areas are occluded during the entire process. Some indoor and outdoor experiments have been conducted to validate the performance of this proposed method.

[1]  Stefano Soatto,et al.  Real-Time Feature Tracking and Outlier Rejection with Changes in Illumination , 2001, ICCV.

[2]  Mubarak Shah,et al.  A non-iterative greedy algorithm for multi-frame point correspondence , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[3]  Tyrone L. Vincent,et al.  Three-Dimensional Motion and Structure Estimation Using Inertial Sensors and Computer Vision for Augmented Reality , 2002, Presence: Teleoperators & Virtual Environments.

[4]  David E. Breen,et al.  Real-time vision-based camera tracking for augmented reality applications , 1997, VRST '97.

[5]  Emanuele Trucco,et al.  Making good features track better , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[6]  Reinhold Behringer,et al.  Registration for outdoor augmented reality applications using computer vision techniques and hybrid sensors , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[7]  Peter Meer,et al.  Point matching under large image deformations and illumination changes , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Yakup Genc,et al.  Marker-less tracking for AR: a learning-based approach , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[9]  Rajeev Sharma,et al.  Interactive evaluation of assembly sequences using augmented reality , 1999, IEEE Trans. Robotics Autom..

[10]  T. Hirabayashi,et al.  Development for teleoperation underwater grasping system in unclear environment , 2002, Proceedings of the 2002 Interntional Symposium on Underwater Technology (Cat. No.02EX556).

[11]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[12]  Patrick A. S. Sinclair,et al.  Integrating hypermedia techniques with augmented reality environments , 2004 .

[13]  Suya You,et al.  A robust hybrid tracking system for outdoor augmented reality , 2004, IEEE Virtual Reality 2004.

[14]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[15]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[16]  Éric Marchand,et al.  A real-time tracker for markerless augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[17]  Marie-Odile Berger,et al.  Pose Estimation for Planar Structures , 2002, IEEE Computer Graphics and Applications.

[18]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[19]  Kiriakos N. Kutulakos,et al.  Calibration-Free Augmented Reality , 1998, IEEE Trans. Vis. Comput. Graph..

[20]  Khoi Nguyen,et al.  Computer-vision-based registration techniques for augmented reality , 1996, Other Conferences.

[21]  Luc Van Gool,et al.  Markerless augmented reality with a real-time affine region tracker , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[22]  Steven K. Feiner,et al.  Information filtering for mobile augmented reality , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[23]  Mubarak Shah,et al.  A noniterative greedy algorithm for multiframe point correspondence , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[25]  Adrian David Cheok,et al.  Augmented Reality Camera Tracking with Homographies , 2002, IEEE Computer Graphics and Applications.

[26]  Yongduek Seo,et al.  Calibration-Free Augmented Reality in Perspective , 2000, IEEE Trans. Vis. Comput. Graph..

[27]  Hideyuki Tamura,et al.  A hybrid registration method for outdoor augmented reality , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[28]  Harald Ganster,et al.  Hybrid Tracking for Outdoor Augmented Reality Applications , 2002, IEEE Computer Graphics and Applications.

[29]  Henry Fuchs,et al.  Augmented reality guidance for needle biopsies: An initial randomized, controlled trial in phantoms , 2002, Medical Image Anal..

[30]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Soh-Khim Ong,et al.  Registration based on projective reconstruction technique for augmented reality systems , 2005, IEEE Transactions on Visualization and Computer Graphics.

[32]  Gene H. Golub,et al.  Matrix computations , 1983 .

[33]  Suya You,et al.  Natural Feature Tracking for Augmented Reality , 1999, IEEE Trans. Multim..

[34]  Gregory D. Hager,et al.  Real-time tracking of image regions with changes in geometry and illumination , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Blair MacIntyre,et al.  Integrating 2-D Video Actors into 3-D Augmented-Reality Systems , 2002, Presence: Teleoperators & Virtual Environments.