The asynchronous time warp for virtual reality on consumer hardware

To help create a true sense of presence in a virtual reality experience, a so called "time warp" may be used. This time warp does not only correct for the optical aberration of the lenses used in a virtual reality headset, it also transforms the stereoscopic images based on the very latest head tracking information to significantly reduce the motion-to-photon delay (or end-to-end latency). The time warp operates as close as possible to the display refresh, retrieves updated head tracking information and transforms a stereoscopic pair of images from representing a view at the time it was rendered, to representing the correct view at the time it is displayed. When run asynchronously to the stereoscopic rendering, the time warp can be used to increase the perceived frame rate and to smooth out inconsistent frame rates. Asynchronous operation can also improve the overall graphics hardware utilization by not requiring the stereoscopic rendering to be synchronized with the display refresh cycle. However, on today's consumer hardware it is challenging to implement a high quality time warp that is fast, has predictable latency and throughput, and runs asynchronously. This paper discusses the various challenges and the different trade-offs that need to be considered when implementing an asynchronous time warp on consumer hardware.

[1]  Ronald Pose,et al.  An interactive graphics display architecture , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[2]  Ronald Pose,et al.  Virtual reality and telerobotics applications of an Address Recalculation Pipeline , 1994 .

[3]  Ozan Cakmakci,et al.  Computational optical distortion correction using a radial basis function-based mapping method. , 2012, Optics express.

[4]  Christof Lutteroth,et al.  More for less: Fast image warping for improving the appearance of head tracking on HMDs , 2013, 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013).

[5]  李横空.  Low-latency virtual reality display system , 2015 .

[6]  Bernd Fröhlich,et al.  The design and implementation of a VR-architecture for smooth motion , 2007, VRST '07.

[7]  William R. Mark,et al.  Efficient Reconstruction Techniques for Post-Rendering 3D Image Warping , 1998 .

[8]  Ronald Pose,et al.  techniques for Reducing Virtual Reality Latency with Architectural Support and Consideration on Human Factors , 1994, MHVR.

[9]  Christof Lutteroth,et al.  Image Warping for Enhancing Consumer Applications of Head-mounted Displays , 2014, AUIC.

[10]  Mark R. Mine,et al.  Just-In-Time Pixels , 1995 .

[11]  Michael J. Griffin,et al.  Compensating Lags in Head-Coupled Displays Using Head Position Prediction and Image Deflection , 1992 .

[12]  Ryugo Kijima,et al.  Reflex HMD to compensate lag and correction of derivative deformation , 2002, Proceedings IEEE Virtual Reality 2002.

[13]  Dandan Zhang,et al.  Single-trial ERPs elicited by visual stimuli at two contrast levels: Analysis of ongoing EEG and latency/amplitude jitters , 2012, 2012 IEEE Symposium on Robotics and Applications (ISRA).

[14]  Leonard McMillan,et al.  Post-rendering 3D warping , 1997, SI3D.

[15]  Bernd Fröhlich,et al.  A Programmable Display Layer for Virtual Reality System Architectures , 2010, IEEE Transactions on Visualization and Computer Graphics.

[16]  Leonard McMillan,et al.  Head-tracked stereoscopic display using image warping , 1995, Electronic Imaging.

[17]  Michael J. Griffin,et al.  Effects of time delays on head tracking performance and the benefits of lag compensation by image deflection , 1991 .

[18]  Ronald Pose.  Parallelism in a computer architecture to support orientation changes in virtual reality and other immersive interactive graphics systems , 2001 .

[19]  Bernd Fröhlich,et al.  An Image-Warping Architecture for VR: Low Latency versus Image Quality , 2009, 2009 IEEE Virtual Reality Conference.

[20]  Daniel Pohl,et al.  Improved pre-warping for wide angle, head mounted displays , 2013, VRST '13.

[21]  Bernd Fröhlich,et al.  A shared-scene-graph image-warping architecture for VR: Low latency versus image quality , 2010, Comput. Graph..

[22]  J. Meigs,et al.  WHO Technical Report , 1954, The Yale Journal of Biology and Medicine.

[23]  Jason Jerald,et al.  Latency compensation by horizontal scanline selection for head- mounted displays , 2007, Electronic Imaging.

[24]  Ronald Pose,et al.  Display Memory Access Issues and Anti-Aliasing with a Virtual Reality Graphics Controller , 1994, Workshop on Graphics Hardware.

[25]  Leonard McMillan,et al.  Post-Rendering Image Warping for Latency Compensation UNC-CH Computer Science Technical Report #96-020 , 1996 .

[26]  John P. Costella.  Motion Extrapolation at the Pixel Level , 2004 .

[27]  Randy Pausch,et al.  Proceedings of the Symposium on Interactive 3D Graphics: Preface , 2003, I3D 2003.

[28]  Tomasz Mazuryk,et al.  Two‐step Prediction and Image Deflection for Exact Head Tracking in Virtual Environments , 1995, Comput. Graph. Forum.

[29]  Uwe H List,et al.  Nonlinear Prediction of Head Movements for Helmet-Mounted Displays , 1983 .

[30]  Yuichi Motai,et al.  A Multiple Model Approach to Track Head Orientation With Delta Quaternions , 2013, IEEE Transactions on Cybernetics.

[31]  Benjamin Watson,et al.  Using texture maps to correct for optical distortion in head-mounted displays , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[32]  Ronald Pose,et al.  Priority rendering with a virtual reality address recalculation pipeline , 1994, SIGGRAPH.

[33]  Marc Olano,et al.  Combatting rendering latency , 1995, I3D '95.

[34]  Jason Jerald.  Latency Compensation for Head-Mounted Virtual Reality , 2004 .

[35]  Ronald D Pose.  ARP: A specialized computer architecture applicable to virtual reality, telerobotics and interactive television , 1997 .

[36]  Richard H. Y. So.  Lag Compensation by Image Deflection and Prediction: A Review on the Potential Benefits to Virtual Training Applications for Manufacturing Industry , 1997, HCI.

[37]  Leonard McMillan,et al.  Post-Rendering Image Warping for Latency Compensation , 1996 .

[38]  Ryugo Kijima,et al.  A Development of Reflex HMD - HMD with time delay compensation capability , 2001 .

[39]  Leonard McMillan,et al.  A List-Priority Rendering Algorithm for Redisplaying Projected Surfaces , 1995 .

[40]  Ronald Azuma,et al.  A frequency-domain analysis of head-motion prediction , 1995, SIGGRAPH.

[41]  William R. Mark,et al.  Post-Rendering 3D Image Warping: Visibility, Reconstruction, and Performance for Depth-Image Warping , 1999 .

[42]  Masahiko Inami,et al.  Improvement of Temporal Quality of HMD for Rotational Motion , 1998 .

[43]  Ferdi Smit,et al.  A programmable display-layer architecture for virtual-reality applications , 2009 .

[44]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1991, Presence: Teleoperators & Virtual Environments.

[45]  Bernd Fröhlich,et al.  An image-warping VR-architecture: design, implementation and applications , 2008, VRST '08.