3-D Motion and Structure Estimation Using Inertial Sensors and Computer Vision for Augmented Reality

A new method for registration in augmented reality (AR) was developed that simultaneously tracks the position, orientation, and motion of the user’s head, as well as estimating the three-dimensional (3-D) structure of the scene. The method fuses data from headmounted cameras and head-mounted inertial sensors. Two Extended Kalman Filters (EKF) are used; one of which estimates the motion of the user’s head and the other that estimates the 3-D locations of points in the scene. A recursive loop is used between the two EKFs. The algorithm was tested using a combination of synthetic and real data, and in general was found to perform well. A further test showed that a system using two cameras performed much better than a system using a single camera, although improving the accuracy of the inertial sensors can partially compensate for the loss of one camera. The method is suitable for use in completely unstructured and unprepared environments. Unlike previous work in this area, this method requires no a priori knowledge about the scene, and can work in environments where the objects of interest are close to the user. Index terms : Augmented reality, pose estimation, registration, Kalman filter, structure from motion, computer vision, inertial sensors Draft of a paper submitted to Presence: Teleoperators and Virtual Environments, November 2000

[1]  Johan Forsberg,et al.  Mobile robot navigation using the range-weighted Hough transform , 1995, IEEE Robotics Autom. Mag..

[2]  James L. Crowley,et al.  Mathematical Foundations of Navigation and Perception for an Autonomous Mobile Robot , 1995, Reasoning with Uncertainty in Robotics.

[3]  Richard A. Brown,et al.  Introduction to random signals and applied kalman filtering (3rd ed , 2012 .

[4]  Takeo Kanade,et al.  A sequential factorization method for recovering shape and motion from image streams , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  R. Chellappa,et al.  Recursive 3-D motion estimation from a monocular image sequence , 1990 .

[6]  Suya You,et al.  Natural Feature Tracking for Augmented Reality , 1999, IEEE Trans. Multim..

[7]  Ronald Azuma,et al.  Tracking in unprepared environments for augmented reality systems , 1999, Comput. Graph..

[8]  Frank Biocca,et al.  A Survey of Position Trackers , 1992, Presence: Teleoperators & Virtual Environments.

[9]  Henry Sowizral,et al.  Note on dynamics of human head motions and on predictive filtering of head-set orientations , 1995, Other Conferences.

[10]  P Munger,et al.  Comparison of relative accuracy between a mechanical and an optical position tracker for image-guided neurosurgery. , 1995, Journal of image guided surgery.

[11]  Ronald Azuma,et al.  A motion-stabilized outdoor augmented reality system , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[12]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[13]  Ernst D. Dickmanns,et al.  Recursive 3-D Road and Relative Ego-State Recognition , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Alex Pentland,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Larry H. Matthies,et al.  Error modeling in stereo navigation , 1986, IEEE J. Robotics Autom..

[16]  B. Roberts,et al.  A system for obstacle detection during rotorcraft low altitude flight , 1996, IEEE Transactions on Aerospace and Electronic Systems.

[17]  John J. Craig Zhu,et al.  Introduction to robotics mechanics and control , 1991 .

[18]  Pietro Perona,et al.  Reducing "Structure From Motion": A General Framework for Dynamic Vision Part 2: Implementation and Experimental Assessment , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Narendra Ahuja,et al.  Segmentation and Factorization-Based Motion and Structure Estimation for Long Image Sequences , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Michael Harrington,et al.  Constellation: a wide-range wireless motion-tracking system for augmented reality and virtual set applications , 1998, SIGGRAPH.

[21]  Olivier D. Faugeras,et al.  Building, Registrating, and Fusing Noisy Visual Maps , 1988, Int. J. Robotics Res..

[22]  Liqiang Feng,et al.  Navigating Mobile Robots: Systems and Techniques , 1996 .

[23]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.

[24]  Takeo Kanade,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995, CVRMed.

[25]  J. P. Mellor.  Realtime Camera Calibration for Enhanced Reality Visualization , 1995, CVRMed.

[26]  Olivier Faugeras,et al.  Maintaining representations of the environment of a mobile robot , 1988, IEEE Trans. Robotics Autom..

[27]  Eric Foxlin,et al.  Inertial head-tracker sensor fusion by a complementary separate-bias Kalman filter , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[28]  Tyrone L. Vincent,et al.  Analysis of Head Pose Accuracy in Augmented Reality , 2000, IEEE Trans. Vis. Comput. Graph..

[29]  .. McGlone,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995 .

[30]  Rajeev Sharma,et al.  Computer Vision-Based Augmented Reality for Guiding Manual Assembly , 1997, Presence: Teleoperators & Virtual Environments.

[31]  Ewald von Puttkamer,et al.  Keeping track of position and orientation of moving indoor systems by correlation of range-finder scans , 1994, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94).

[32]  Hugh F. Durrant-Whyte,et al.  Inertial navigation systems for mobile robots , 1995, IEEE Trans. Robotics Autom..

[33]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[34]  Khoi Nguyen,et al.  Computer-vision-based registration techniques for augmented reality , 1996, Other Conferences.

[35]  William A. Hoff,et al.  Planetary terminal descent hazard avoidance using optical flow , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[36]  Bir Bhanu,et al.  A system for obstacle detection during rotorcraft low altitude flight , 1996 .

[37]  Pietro Perona,et al.  Reducing "Structure From Motion": A General Framework for Dynamic Vision Part 1: Modeling , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[39]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[40]  Nicholas Ayache,et al.  Computer Vision, Virtual Reality and Robotics in Medicine , 1995, Lecture Notes in Computer Science.

[41]  Tyrone L. Vincent,et al.  An adaptive estimator for registration in augmented reality , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[42]  D.J. Kriegman,et al.  Stereo vision and navigation in buildings for mobile robots , 1989, IEEE Trans. Robotics Autom..

[43]  Rama Chellappa,et al.  Passive ranging using a moving camera , 1992, J. Field Robotics.

[44]  Hans P. Moravec.  Sensor Fusion in Certainty Grids for Mobile Robots , 1988, AI Mag..

[45]  Thomas S. Huang,et al.  Some Experiments on Estimating the 3-D Motion Parameters of a Rigid Body from Two Consecutive Image Frames , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.