Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering

Virtual and augmented reality (VR/AR) displays crucially rely on stereoscopic rendering to enable perceptually realistic user experiences. Yet, existing near-eye display systems ignore the gaze-dependent shift of the no-parallax point in the human eye. Here, we introduce a gaze-contingent stereo rendering technique that models this effect and conduct several user studies to validate its effectiveness. Our findings include experimental validation of the location of the no-parallax point, which we then use to demonstrate significant improvements of disparity and shape distortion in a VR setting, and consistent alignment of physical and digitally rendered objects across depths in optical see-through AR. Our work shows that gaze-contingent stereo rendering improves perceptual realism and depth perception of emerging wearable computing systems.

[1]  KonradRobert,et al.  Gaze-Contingent Ocular Parallax Rendering for Virtual Reality , 2020 .

[2]  A. GLENNERSTER,et al.  Stereoscopic Depth Constancy Depends on the Subject's Task , 1996, Vision Research.

[3]  Alexei A. Goon,et al.  Multifocal planes head-mounted displays. , 2000, Applied optics.

[4]  Byoungho Lee,et al.  Holographic near-eye display with expanded eye-box , 2018, ACM Trans. Graph..

[5]  Gordon Wetzstein,et al.  The light field stereoscope , 2015, ACM Trans. Graph..

[6]  Miguel A. Nacenta,et al.  Depth perception with gaze-contingent depth of field , 2014, CHI.

[7]  Yifan Peng,et al.  Neural Holography , 2020, SIGGRAPH Emerging Technologies.

[8]  Douglas Lanman,et al.  Fast gaze-contingent optimal decompositions for multifocal displays , 2017, ACM Trans. Graph..

[9]  Noboru Ohnishi,et al.  Effect of the sight line shift when a head-mounted display is used , 2000, Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No.00CH37143).

[10]  Desney S. Tan,et al.  Foveated 3D graphics , 2012, ACM Trans. Graph..

[11]  Radoslaw Mantiuk,et al.  Gaze-Dependent Tone Mapping , 2013, ICIAR.

[12]  E. Peli.  A binocular stereoscopic display system with coupled convergence and accommondation demands , 2001 .

[13]  B. V. K. Vijaya Kumar,et al.  Towards multifocal displays with dense focal stacks , 2018, ACM Trans. Graph..

[14]  Nassir Navab,et al.  Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[15]  LuebkeDavid,et al.  Near-eye light field holographic rendering with spherical waves for wide field of view interactive 3D computer graphics , 2017 .

[16]  Douglas Lanman,et al.  Near-eye light field displays , 2013, SIGGRAPH '13.

[17]  Sheng Liu,et al.  An optical see-through head mounted display with addressable focal planes , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[18]  David A. Atchison,et al.  Optics of the Human Eye , 2023 .

[19]  Hans-Peter Seidel,et al.  GazeStereo3D: seamless disparity manipulations , 2016, ACM Trans. Graph..

[20]  Gordon Wetzstein,et al.  Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays , 2017, Proceedings of the National Academy of Sciences.

[21]  Ersin Yumer,et al.  Learning Local Shape Descriptors from Part Correspondences with Multiview Convolutional Networks , 2017, ACM Trans. Graph..

[22]  David M. Hoffman,et al.  The zone of comfort: Predicting visual discomfort with stereo displays. , 2011, Journal of vision.

[23]  Thomas P. Caudell,et al.  Calibration of head-mounted displays for augmented reality applications , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[24]  Rachel A. Albert,et al.  Foveated AR: Dynamically-Foveated Augmented Reality Display , 2019 .

[25]  Jae-Hyeung Park,et al.  Optical see-through holographic near-eye-display with eyebox steering and depth of field control. , 2018, Optics express.

[26]  Felix A. Wichmann,et al.  Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data , 2016, Vision Research.

[27]  F A Wichmann,et al.  Ning for Helpful Comments and Suggestions. This Paper Benefited Con- Siderably from Conscientious Peer Review, and We Thank Our Reviewers the Psychometric Function: I. Fitting, Sampling, and Goodness of Fit , 2001 .

[28]  Joohwan Kim,et al.  Towards foveated rendering for gaze-tracked virtual reality , 2016, ACM Trans. Graph..

[29]  Felix Wichmann,et al.  The psychometric function: I , 2001 .

[30]  Daisuke Iwai,et al.  Gaussian Light Field: Estimation of Viewpoint-Dependent Blur for Optical See-Through Head-Mounted Displays , 2016, IEEE Transactions on Visualization and Computer Graphics.

[31]  Martin S. Banks,et al.  A stereo display prototype with multiple focal distances , 2004, ACM Trans. Graph..

[32]  Radoslaw Mantiuk,et al.  Gaze-Dependent Depth-of-Field Effect Rendering in Virtual Environments , 2011, SGDA.

[33]  Anatole Lécuyer,et al.  Using an Eye-Tracking System to Improve Camera Motions and Depth-of-Field Blur Effects in Virtual Environments , 2008, 2008 IEEE Virtual Reality Conference.

[34]  David R. Flatla,et al.  Gaze-Contingent Manipulation of Color Perception , 2016, CHI.

[35]  W. B. Thompson,et al.  Relative motion: Kinetic information for the order of depth at an edge , 1987, Perception & psychophysics.

[36]  Peter Shirley,et al.  Near-eye varifocal augmented reality display using see-through screens , 2017, ACM Trans. Graph..

[37]  Hans-Peter Seidel,et al.  Motion parallax in stereo 3D , 2016, ACM Trans. Graph..

[38]  Charless C. Fowlkes,et al.  Natural-Scene Statistics Predict How the Figure–Ground Cue of Convexity Affects Human Depth Perception , 2010, The Journal of Neuroscience.

[39]  Gordon Wetzstein,et al.  Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays , 2016, CHI.

[40]  David A. Atchison,et al.  Chapter 20 – The aging eye , 2000 .

[41]  Andreas Georgiou,et al.  Holographic near-eye displays for virtual and augmented reality , 2017, ACM Trans. Graph..

[42]  Touradj Ebrahimi,et al.  Subjective evaluation of two stereoscopic imaging systems exploiting visual attention to improve 3D quality of experience , 2014, Electronic Imaging.

[43]  Karol Myszkowski,et al.  Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors , 2017, IEEE Transactions on Visualization and Computer Graphics.

[44]  Jacek Turski,et al.  On binocular vision: The geometric horopter and Cyclopean eye , 2016, Vision Research.

[45]  Andrew T. Duchowski,et al.  Gaze-Contingent Displays: A Review , 2004, Cyberpsychology Behav. Soc. Netw..

[46]  E. Peli,et al.  53.2: A Binocular Stereoscopic Display System with Coupled Convergence and Accommodation Demands , 2001 .

[47]  Katerina Mania,et al.  Near‐Eye Display and Tracking Technologies for Virtual and Augmented Reality , 2019, Comput. Graph. Forum.

[48]  Geoffrey P. Bingham,et al.  Optical flow from eye movement with head immobilized: “Ocular occlusion” beyond the nose , 1993, Vision Research.

[49]  Marc Levoy,et al.  Simulating the Visual Experience of Very Bright and Very Dark Scenes , 2015, ACM Trans. Graph..

[50]  Ronald Azuma,et al.  ThinVR: Heterogeneous microlens arrays for compact, 180 degree FOV VR near-eye displays , 2020, IEEE Transactions on Visualization and Computer Graphics.

[51]  Hans-Peter Seidel,et al.  A perceptual model for disparity , 2011, ACM Trans. Graph..

[52]  Gudrun Klinker,et al.  Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[53]  Bozo Vojniković,et al.  Horopters--definition and construction. , 2013, Collegium antropologicum.

[54]  Hu Xinghong,et al.  Colorblind-shareable videos by synthesizing temporal-coherent polynomial coefficients , 2019, ACM Transactions on Graphics.

[55]  Joohwan Kim,et al.  Perceptually-guided foveation for light field displays , 2017, ACM Trans. Graph..

[56]  H. Kudo,et al.  Measurement of the ability in monocular depth perception during gazing at near visual target-effect of the ocular parallax cue , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[57]  Yifan Peng,et al.  Holographic near-eye displays based on overlap-add stereograms , 2019, ACM Trans. Graph..

[58]  Wilson S. Geisler,et al.  Real-time foveated multiresolution system for low-bandwidth video communication , 1998, Electronic Imaging.

[59]  Gordon Wetzstein,et al.  Gaze-contingent ocular parallax rendering for virtual reality , 2019, SIGGRAPH Talks.

[60]  Han Woong Lim,et al.  Normal range of eye movement and its relationship to age , 2016 .

[61]  Whitman Richards,et al.  Convergence as a cue to depth , 1969 .

[62]  Mikhail Okunev,et al.  DeepFovea , 2019, ACM Trans. Graph..

[63]  David P. Luebke,et al.  Perceptually-Driven Simplification for Interactive Rendering , 2001, Rendering Techniques.

[64]  Christopher D. Saunter,et al.  Dynamic lens and monovision 3D displays to improve viewer comfort , 2015, Optics express.

[65]  Bahram Javidi,et al.  A 3D integral imaging optical see-through head-mounted display. , 2014, Optics express.

[66]  Gudrun Klinker,et al.  Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays , 2015, IEEE Transactions on Visualization and Computer Graphics.

[67]  Donald H. House,et al.  Reducing visual discomfort of 3D stereoscopic displays with gaze-contingent depth-of-field , 2014, SAP.

[68]  Hideyuki Tamura,et al.  Gaze-directed adaptive rendering for interacting with virtual space , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[69]  Neil A. Dodgson,et al.  Variation and extrema of human interpupillary distance , 2004, IS&T/SPIE Electronic Imaging.

[70]  Robert S. Allison,et al.  Gaze-contingent depth of field in realistic scenes: the user experience , 2014, ETRA.

[71]  Andrew T. Duchowski,et al.  EUROGRAPHICS 2001 / Jonathan C. Roberts Short Presentations Gaze-Contingent Level Of Detail Rendering , 2022 .