Predictive tracking for augmented reality

In Augmented Reality systems, see-through Head-Mounted Displays (HMDs) superimpose virtual three-dimensional objects on the real world. This technology has the potential to enhance a user's perception of and interaction with the real world. However, many Augmented Reality applications will not be accepted unless virtual objects are accurately registered with their real counterparts. Good registration is difficult, because of the high resolution of the human visual system and its sensitivity to small differences. Registration errors fall into two categories: static errors, which occur even when the user remains still, and dynamic errors caused by system delays when the user moves. Dynamic errors are usually the largest errors. This dissertation demonstrates that predicting future head locations is an effective approach for significantly reducing dynamic errors. This demonstration is performed in real time with an operational Augmented Reality system. First, evaluating the effect of prediction requires robust static registration. Therefore, this system uses a custom optoelectronic head-tracking system and three calibration procedures developed to measure the viewing parameters. Second, the system predicts future head positions and orientations with the aid of inertial sensors. Effective use of these sensors requires accurate estimation of the varying prediction intervals, optimization techniques for determining parameters, and a system built to support real-time processes. On average, prediction with inertial sensors is 2 to 3 times more accurate than prediction without inertial sensors and 5 to 10 times more accurate than not doing any prediction at all. Prediction is most effective at short prediction intervals, empirically determined to be about 80 milliseconds or less. An analysis of the predictor in the frequency domain shows the predictor magnifies the signal by roughly the square of the angular frequency and the prediction interval. For specified head-motion sequences and prediction intervals, this analytical framework can also estimate the maximum possible time-domain error and the maximum tolerable system delay given a specified maximum time-domain error. Future steps that may further improve registration are discussed.

[1]  Michael Deering,et al.  High resolution virtual reality , 1992, SIGGRAPH.

[2]  Uwe H List,et al.  Nonlinear Prediction of Head Movements for Helmet-Mounted Displays , 1983 .

[3]  Henry Fuchs,et al.  Pixel-planes 5: a heterogeneous multiprocessor graphics system using processor-enhanced memories , 1989, SIGGRAPH.

[4]  H. P. Blom.  An efficient filter for abruptly changing systems , 1984, The 23rd IEEE Conference on Decision and Control.

[5]  John F. Hughes,et al.  Autocalibration for virtual environments tracking hardware , 1993, SIGGRAPH.

[6]  Michael J. Griffin,et al.  A Review and Investigation of Aiming and Tracking Performance with Head-Mounted Sights , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[7]  Douglas C. Montgomery,et al.  Forecasting and Time Series Analysis (2nd ed.). , 1992 .

[8]  Ronald Pose,et al.  Priority rendering with a virtual reality address recalculation pipeline , 1994, SIGGRAPH.

[9]  Jitendra K. Tugnait,et al.  Detection and estimation for abruptly changing systems , 1981, 1981 20th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes.

[10]  Douglas E. Holmgren.  Design and Construction of a 30-Degree See-Through Head-Mounted Display , 1992 .

[11]  Harry M. Assenheim,et al.  Large Screen Projection, Avionic, and Helmet-Mounted Displays , 1991 .

[12]  H. Sorenson.  Least-squares estimation: from Gauss to Kalman , 1970, IEEE Spectrum.

[13]  T. J. Lauterborn.  American Society Of Photogrammetry , 1984 .

[14]  Ronald Azuma,et al.  A demonstrated optical tracker with scalable work area for head-mounted display systems , 1992, I3D '92.

[15]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[16]  Michael J. Griffin,et al.  Compensating Lags in Head-Coupled Displays Using Head Position Prediction and Image Deflection , 1992 .

[17]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[18]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[19]  D. B. Preston.  Spectral Analysis and Time Series , 1983 .

[20]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[21]  P. Bogler.  Tracking a Maneuvering Target Using Input Estimation , 1987, IEEE Transactions on Aerospace and Electronic Systems.

[22]  Warren Robinson,et al.  Synthetic Experience: A Proposed Taxonomy , 1992, Presence Teleoperators Virtual Environ..

[23]  Matthias M. Wloka.  Lag in Multiprocessor Virtual Reality , 1995, Presence: Teleoperators & Virtual Environments.

[24]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[25]  R. Andres,et al.  Prediction of Head Movement Time Using Fitts’ Law , 1989 .

[26]  Carlo H. Séquin,et al.  Adaptive display algorithm for interactive frame rates during visualization of complex virtual environments , 1993, SIGGRAPH.

[27]  Mark R. Mine.  Characterization of End-to-End Delays in Head-Mounted Display Systems , 1993 .

[28]  Richard A. Brown,et al.  Introduction to random signals and applied kalman filtering (3rd ed , 2012 .

[29]  Dave Sims,et al.  New realities in aircraft design and manufacture , 1994, IEEE Computer Graphics and Applications.

[30]  M L Lin,et al.  A Method for Evaluating Head-Controlled Computer Input Devices Using Fitts' Law , 1990, Human factors.

[31]  D. F. Crane.  The effects of time delay in man-machine control systems: Implications for design of flight simulator Visual-Display-Delay compensation , 1984 .

[32]  R. E. Albrecht,et al.  An adaptive digital filter to predict pilot head look direction for helmet-mounted displays , 1989 .

[33]  Robert K. Rebo,et al.  A Helmet-Mounted Virtual Environment Display System , 1989, Defense, Security, and Sensing.

[34]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[35]  F. Lewis.  Optimal Estimation: With an Introduction to Stochastic Control Theory , 1986 .

[36]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1991, Presence: Teleoperators & Virtual Environments.

[37]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[38]  Thomas P. Caudell,et al.  Calibration of head-mounted displays for augmented reality applications , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[39]  J.C.K. Chou,et al.  Quaternion kinematic and dynamic differential equations , 1992, IEEE Trans. Robotics Autom..

[40]  Donald G. Childers,et al.  Modern Spectrum Analysis , 1978 .

[41]  Ted Morris,et al.  Using a Maximum Error Statistic to Evaluate Measurement Errors in 3D Position and Orientation Tracking Systems , 1993, Presence: Teleoperators & Virtual Environments.

[42]  Wolfgang Splettstösser.  On the prediction of band-limited signals from past samples , 1982, Inf. Sci..

[43]  F. P. Brooks,et al.  Grasping reality through illusion—interactive graphics serving science , 1988, CHI '88.

[44]  R. Welch.  Perceptual modification : adapting to altered sensory environments / Robert B. Welch , 1978 .

[45]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[46]  R A Abrams,et al.  Optimality in human motor performance: ideal control of rapid aimed movements. , 1988, Psychological review.

[47]  Won S. Kim,et al.  The phantom robot: predictive displays for teleoperation with time delay , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[48]  D. Magill.  Optimal adaptive estimation of sampled stochastic processes , 1965 .

[49]  Ronald Azuma,et al.  Space Resection by Collinearity: Mathematics Behind the Optical Ceiling Head-Tracker , 1991 .

[50]  R. E. Mcfarland,et al.  Transport delay compensation for computer-generated imagery systems , 1988 .

[51]  E. Chekaluk,et al.  Visual stimulus input, saccadic suppression, and detection of information from the postsaccade scene , 1990, Perception & psychophysics.

[52]  Warren Robinett,et al.  Implementation of flying, scaling and grabbing in virtual worlds , 1992, I3D '92.

[53]  Anthony Lawrence,et al.  Modern Inertial Technology: Navigation, Guidance, and Control , 1993 .

[54]  F. Raab,et al.  Magnetic Position and Orientation Tracking System , 1979, IEEE Transactions on Aerospace and Electronic Systems.

[55]  Jr. B. Smith.  Digital head tracking and position prediction for helmet mounted visual display systems , 1984 .

[56]  Dale H. Mugler.  Computationally efficient linear prediction from past samples of a band-limited signal and its derivative , 1990, IEEE Trans. Inf. Theory.

[57]  Roger Y. Tsai,et al.  Techniques for Calibration of the Scale Factor and Image Center for High Accuracy 3-D Machine Vision Metrology , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[59]  Frank J. Ferrin.  Survey of helmet tracking technologies , 1991, Medical Imaging.

[60]  Susumu Tachi,et al.  Experimental Study on Remote Manipulation Using Virtual Reality , 1993, Presence: Teleoperators & Virtual Environments.

[61]  Frank Cardullo,et al.  Predictive compensation of visual system time delays , 1987 .

[62]  W. Bradford Paley,et al.  Head-tracking stereo display: experiments and applications , 1992, Electronic Imaging.

[63]  Steven K. Feiner,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 1990 .

[64]  Shumin Zhai,et al.  ARGOS: a display system for augmenting reality , 1993, CHI '93.

[65]  Ronald Azuma,et al.  Tracking requirements for augmented reality , 1993, CACM.

[66]  Lawrence Koved,et al.  Design for interactive performance in a virtual laboratory , 1990, I3D '90.

[67]  Alex Pentland,et al.  Device synchronization using an optimal linear filter , 1992, I3D '92.

[68]  Richard E. Mcfarland.  CGI Delay Compensation , 1986 .

[69]  R. J. Leigh,et al.  Control of Head Movement , 1988, Neurology.

[70]  R J Jagacinski,et al.  Fitts' Law in two dimensions with hand and head movements. , 1983, Journal of motor behavior.

[71]  Chris Shaw,et al.  On temporal-spatial realism in the virtual reality environment , 1991, UIST '91.

[72]  Christopher D. Shaw,et al.  An Experiment to Characterize Head Motion in VR and RR Using MR , 1992 .

[73]  F. Harris.  On the use of windows for harmonic analysis with the discrete Fourier transform , 1978, Proceedings of the IEEE.

[74]  Chaw-Bing Chang,et al.  Application of state estimation to target tracking , 1984 .

[75]  D. G. Watts,et al.  Spectral analysis and its applications , 1968 .

[76]  R. Berg.  Estimation and prediction for maneuvering target trajectories , 1983 .

[77]  Anthony M. Cook.  The helmet-mounted visual system in flight simulation , 1988 .

[78]  Charles L. Phillips,et al.  Digital control system analysis and design (2nd ed.) , 1989 .

[79]  Randy F. Pausch,et al.  A Literature Survey for Virtual Environments: Military Flight Simulator Visual Systems and Simulator Sickness , 1992, Presence: Teleoperators & Virtual Environments.