Occlusion Leak Compensation for Optical See-Through Displays Using a Single-Layer Transmissive Spatial Light Modulator

We propose an occlusion compensation method for optical see-through head-mounted displays (OST-HMDs) equipped with a singlelayer transmissive spatial light modulator (SLM), in particular, a liquid crystal display (LCD). Occlusion is an important depth cue for 3D perception, yet realizing it on OST-HMDs is particularly difficult due to the displays' semitransparent nature. A key component for the occlusion support is the SLM—a device that can selectively interfere with light rays passing through it. For example, an LCD is a transmissive SLM that can block or pass incoming light rays by turning pixels black or transparent. A straightforward solution places an LCD in front of an OST-HMD and drives the LCD to block light rays that could pass through rendered virtual objects at the viewpoint. This simple approach is, however, defective due to the depth mismatch between the LCD panel and the virtual objects, leading to blurred occlusion. This led existing OST-HMDs to employ dedicated hardware such as focus optics and multi-stacked SLMs. Contrary to these viable, yet complex and/or computationally expensive solutions, we return to the single-layer LCD approach for the hardware simplicity while maintaining fine occlusion—we compensate for a degraded occlusion area by overlaying a compensation image. We compute the image based on the HMD parameters and the background scene captured by a scene camera. The evaluation demonstrates that the proposed method reduced the occlusion leak error by 61.4% and the occlusion error by 85.7%.

[1]  François Goudail,et al.  Minimization of diffraction peaks of spatial light modulators using Voronoi diagrams. , 2010, Optics express.

[2]  Tobias Höllerer,et al.  Resolving multiple occluded layers in augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[3]  Douglas Lanman,et al.  Near-eye light field displays , 2013, SIGGRAPH Emerging Technologies.

[4]  Anselmo Lastra,et al.  From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality , 2016, IEEE Transactions on Visualization and Computer Graphics.

[5]  Henry Fuchs,et al.  Computational augmented reality eyeglasses , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1991, Presence: Teleoperators & Virtual Environments.

[7]  Gordon Wetzstein,et al.  The light field stereoscope , 2015, ACM Trans. Graph..

[8]  Yasuhiro Takaki,et al.  See-through integral imaging display with background occlusion capability. , 2016, Applied optics.

[9]  Mark Billinghurst,et al.  An occlusion capable optical see-through head mount display for supporting co-located collaboration , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[10]  Eric Kolstad,et al.  Egocentric depth judgments in optical, see-through augmented reality , 2007, IEEE Transactions on Visualization and Computer Graphics.

[11]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion with a real-time stereovision system , 2001, Comput. Graph..

[12]  Gordon Wetzstein,et al.  Consistent illumination within optical see-through augmented environments , 2003, SIGGRAPH '03.

[13]  Gudrun Klinker,et al.  Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[14]  Narendra Ahuja,et al.  A testbed for precise registration, natural occlusion and interaction in an augmented environment using a head-mounted projective display (HMPD) , 2002, Proceedings IEEE Virtual Reality 2002.

[15]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[16]  Moshe Eizenman,et al.  General theory of remote gaze estimation using the pupil center and corneal reflections , 2006, IEEE Transactions on Biomedical Engineering.

[17]  FuchsHenry,et al.  Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors , 2017 .

[18]  André Stork,et al.  The daylight blocking optical stereo see-through HMD , 2008, IPT/EDT '08.

[19]  Gordon Wetzstein,et al.  Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays , 2017, Proceedings of the National Academy of Sciences.

[20]  S. Suyama,et al.  Three-Dimensional Display System with Dual-Frequency Liquid-Crystal Varifocal Lens , 2000 .

[21]  Steven K. Feiner,et al.  Perceptual issues in augmented reality revisited , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[22]  Gordon Wetzstein,et al.  A survey on computational displays: Pushing the boundaries of optics, computation, and perception , 2013, Comput. Graph..

[23]  Joonku Hahn,et al.  Holographic head-mounted display with RGB light emitting diode light source. , 2014, Optics express.

[24]  Andreas Georgiou,et al.  Holographic near-eye displays for virtual and augmented reality , 2017, ACM Trans. Graph..

[25]  Jurriaan D. Mulder.  Realistic occlusion effects in mirror-based co-located augmented reality systems , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[26]  Henry Fuchs,et al.  Comparison of optical and video see-through, head-mounted displays , 1995, Other Conferences.

[27]  Hong Hua,et al.  Enabling Focus Cues in Head-Mounted Displays , 2017, Proceedings of the IEEE.

[28]  Chang-Kun Lee,et al.  Holographic and light-field imaging for augmented reality , 2017, OPTO.

[29]  Hirokazu Kato,et al.  SharpView: Improved clarity of defocused content on optical see-through head-mounted displays , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[30]  Eric W. Tatham.  Technical opinion: getting the best of both real and virtual worlds , 1999, CACM.

[31]  Kiyoshi Kiyokawa.  Occlusion Displays , 2012, Handbook of Visual Display Technology.

[32]  Martin S. Banks,et al.  A stereo display prototype with multiple focal distances , 2004, ACM Trans. Graph..

[33]  Hong Hua,et al.  Optical see-through head-mounted display with occlusion capability , 2013, Defense, Security, and Sensing.

[34]  Yuta Itoh,et al.  AdaptiVisor: assisting eye adaptation via occlusive optical see-through head-mounted displays , 2017, AH.

[35]  Oliver Bimber,et al.  Occlusion shadows: using projected light to generate realistic occlusion effects for view-dependent optical see-through displays , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[36]  Gudrun Klinker,et al.  Semi-Parametric Color Reproduction Method for Optical See-Through Head-Mounted Displays , 2015, IEEE Transactions on Visualization and Computer Graphics.

[37]  Sheng Liu,et al.  An optical see-through head mounted display with addressable focal planes , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[38]  Karol Myszkowski,et al.  Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors , 2017, IEEE Transactions on Visualization and Computer Graphics.

[39]  Oliver Bimber,et al.  Displays: Fundamentals and Applications , 2011 .

[40]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[41]  Oliver Bimber,et al.  Displays: Fundamentals & Applications, Second Edition , 2016 .

[42]  Hirokazu Kato,et al.  EyeAR: Refocusable Augmented Reality Content through Eye Measurements , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[43]  Douglas Lanman,et al.  Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources , 2014, SIGGRAPH '14.

[44]  Holger Regenbrecht,et al.  Real-Time Radiometric Compensation for Optical See-Through Head-Mounted Displays , 2016, IEEE Transactions on Visualization and Computer Graphics.

[45]  Michael Gervautz,et al.  Occlusion in collaborative augmented environments , 1999, Comput. Graph..

[46]  Hideki Kakeya,et al.  Realization of an aerial 3D image that occludes the background scenery. , 2014, Optics express.

[47]  Jannick P. Rolland,et al.  A compact optical see-through head-worn display with occlusion support , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[48]  Hong Hua,et al.  Occlusion capable optical see-through head-mounted display using freeform optics , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).