Registration errors in augmented reality systems

Augmented reality (AR) systems combine three-dimensional computer-generated imagery with the view of the real environment in order to make unseen objects visible or to present additional information. A critical problem is that the computer-generated objects do not currently remain correctly registered with the real environment--objects aligned from one viewpoint appear misaligned from another and appear to swim about as the viewer moves. This registration error is caused by a number of factors, such as system delay, optical distortion, and tracker measurement error, and is difficult to correct with existing technology. This dissertation presents a registration error model for AR systems and uses it to gain insight into the nature and severity of the registration error caused by the various error sources. My thesis is that a mathematical error model enables the system architect to determine (1) which error sources are the most significant, (2) the sensitivity of the net registration error to each error, (3) the nature of the distortions caused by each type of error, (4) the level of registration accuracy one can expect, and also provides insights on how best to calibrate the system. Analysis of a surgery planning application yielded the following main results: (1) Even for moderate head velocities, system delay causes more registration error than all other sources combined; (2) Using the eye's center of rotation as the eyepoint in the computer graphics model reduces the error due to eye rotation to zero for points along the line of gaze. This should obviate the need for eye tracking; (3) Tracker error is a significant problem both in head tracking and in system calibration; (4) The World coordinate system should be omitted when possible; (5) Optical distortion is a significant error source, but correcting it computationally in the graphics pipeline often induces delay error larger than the distortion error itself; (6) Knowledge of the nature of the various types of error facilitates identification and correction of errors in the calibration process. Although the model was developed for see-through head-mounted displays (STHMDs) for surgical planning, many of the results are applicable to other HMD systems as well.

[1]  R. S. Longhurst.  Geometrical and Physical Optics , 1967 .

[2]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[3]  Mark R. Mine,et al.  Just-In-Time Pixels , 1995 .

[4]  Lloyd Kaufman,et al.  Sight and mind , 1974 .

[5]  Michael W. Vannier,et al.  Introduction to 3D imaging , 1991 .

[6]  Henry Fuchs,et al.  Frameless rendering: double buffering considered harmful , 1994, SIGGRAPH.

[7]  Paul Milgram,et al.  Enhancement of 3-D Video Displays by Means of Superimposed Stereo-Graphics , 1991 .

[8]  Harry H. Poole,et al.  Fundamentals of display systems , 1966 .

[9]  Henry Fuchs,et al.  Pixel-planes 5: a heterogeneous multiprocessor graphics system using processor-enhanced memories , 1989, SIGGRAPH.

[10]  Jannick P. Rolland,et al.  Towards Quantifying Depth and Size Perception in Virtual Environments , 1993, Presence: Teleoperators & Virtual Environments.

[11]  Gregg Podnar,et al.  Geometry of binocular imaging , 1994, Electronic Imaging.

[12]  Thomas A. Furness.  The Super Cockpit and its Human Factors Challenges , 1986 .

[13]  M W Vannier,et al.  Three-dimensional surface imaging from CT scans for the study of craniofacial dysmorphology. , 1989, Journal of craniofacial genetics and developmental biology.

[14]  Sol Sherr,et al.  Fundamentals of display system design , 1970 .

[15]  Michael Deering,et al.  High resolution virtual reality , 1992, SIGGRAPH.

[16]  Pascal Roger Chesnais.  A graphic/photographic arthroscopy simulator , 1988 .

[17]  David E. Altobelli.  Intraoperative navigation in craniomaxillofacial surgery , 1991 .

[18]  Henry Fuchs,et al.  Research directions in virtual environments: report of an NSF Invitational Workshop, March 23-24, 1992, University of North Carolina at Chapel Hill , 1992, COMG.

[19]  Marc Levoy,et al.  Interactive visualization of 3D medical data , 1989, Computer.

[20]  Wesley E. Woodson,et al.  Human engineering guide for equipment designers , 1964 .

[21]  R. Maciunas,et al.  The accuracies of four stereotactic frame systems: an independent assessment. , 1991, Biomedical instrumentation & technology.

[22]  Patrick Min,et al.  Interactive stereoscopy optimization for head-mounted displays , 1994, Electronic Imaging.

[23]  Lawrence E. Tannas,et al.  Flat-Panel Displays and Crts , 1985 .

[24]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[25]  Warren Robinett,et al.  The Visual Display Transformation for Virtual Reality , 1995, Presence: Teleoperators & Virtual Environments.

[26]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[27]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Seungho Lee,et al.  Cranial computed tomography and MRI , 1987 .

[29]  Arnie Kwong,et al.  Fall Joint Computer Conference , 1987, Data Base.

[30]  Dean F. Kocian.  Design Considerations for Virtual Panoramic Display (VPD) Helmet Systems , 1990 .

[31]  Christopher Schmandt,et al.  Spatial input/display correspondence in a stereoscopic computer graphic work station , 1983, SIGGRAPH.

[32]  G. Herman,et al.  3D Imaging In Medicine , 1991 .

[33]  Roy Kalawsky,et al.  The science of virtual reality and virtual environments - a technical, scientific and engineering reference on virtual environments , 1993 .

[34]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[35]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[36]  David E. Altobelli.  Computer assisted three-dimensional planning for cranio-maxillofacial surgery: Osteotomy simulation , 1991 .

[37]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1991, Presence: Teleoperators & Virtual Environments.

[38]  Jannick P. Rolland,et al.  A Method of Computational Correction for Optical Distortioin in Head-Mounted Displays , 1993 .

[39]  David A. Southard,et al.  Viewing model for stereoscopic head-mounted displays , 1994, Electronic Imaging.

[40]  John H. Harris,et al.  Three-dimensional imaging of the musculoskeletal system , 1991 .

[41]  Gabor T. Herman.  Quantization using 3D images , 1991 .

[42]  R. B. House,et al.  At the University of North Carolina , 1945 .

[43]  Stanley M. Shinners.  Techniques of System Engineering , 1967 .

[44]  L A Whitaker,et al.  Evaluation of facial skeletal aesthetics and surgical planning. , 1991, Clinics in plastic surgery.

[45]  Ulrich Neumann,et al.  Dynamic Registration Correction in , 1995 .

[46]  Sol Sherr.  Video and digital electronic displays: A user's guide , 1982 .

[47]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[48]  John M. Hollerbach,et al.  Optical Design for a Head-Mounted Display , 1993, Presence: Teleoperators & Virtual Environments.

[49]  J. Taylor.  An Introduction to Error Analysis , 1982 .

[50]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[51]  Kenneth R. Boff,et al.  Engineering data compendium : human perception and performance , 1988 .

[52]  E.M. Friets,et al.  A frameless stereotaxic operating microscope for neurosurgery , 1989, IEEE Transactions on Biomedical Engineering.

[53]  Benjamin Watson,et al.  Using texture maps to correct for optical distortion in head-mounted displays , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[54]  Ronald Azuma,et al.  A demonstrated optical tracker with scalable work area for head-mounted display systems , 1992, I3D '92.

[55]  Thomas A. DeFanti,et al.  Ultrasonic calibration of a magnetic tracker in a virtual reality space , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[56]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[57]  Ronald Azuma,et al.  Predictive tracking for augmented reality , 1995 .

[58]  A. B. Nutt.  Binocular vision. , 1945, The British orthoptic journal.

[59]  F L Bookstein,et al.  Three‐Dimensional Computer‐Assisted Design of Craniofacial Surgical Procedures: Optimization and Interaction with Cephalometric and CT‐Based Models , 1986, Plastic and reconstructive surgery.

[60]  Ulrich Neumann,et al.  Dynamic registration correction in augmented-reality systems , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[61]  Bernard D. Adelstein,et al.  Dynamic Response of Electromagnetic Spatial Displacement Trackers , 1996, Presence: Teleoperators & Virtual Environments.

[62]  Scott S. Fisher,et al.  Defining, modeling, and measuring system lag in virtual environments , 1990, Other Conferences.

[63]  Werner Krybus,et al.  Computer-assisted surgery , 1990, IEEE Computer Graphics and Applications.

[64]  Marc Olano,et al.  Combatting rendering latency , 1995, I3D '95.

[65]  Hong Chen,et al.  Observing a volume rendered fetus within a pregnant patient , 1994, Proceedings Visualization '94.

[66]  Mark R. Mine.  Characterization of End-to-End Delays in Head-Mounted Display Systems , 1993 .

[67]  Thomas P. Caudell,et al.  Calibration of head-mounted displays for augmented reality applications , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[68]  Steve T. Bryson.  Measurement and calibration of static distortion of position data from 3D trackers , 1992, Electronic Imaging.

[69]  Larry F. Hodges,et al.  Geometric Considerations for Stereoscopic Virtual Environments , 1993, Presence: Teleoperators & Virtual Environments.

[70]  William F. Reinhart.  Gray-scale requirements for antialiasing of stereoscopic graphic imagery , 1992, Electronic Imaging.

[71]  Jayaram K. Udupa,et al.  Registration of 3D objects and surfaces , 1990, IEEE Computer Graphics and Applications.