Reproducing reality with a high-dynamic-range multi-focal stereo display

With well-established methods for producing photo-realistic results, the next big challenge of graphics and display technologies is to achieve perceptual realism --- producing imagery indistinguishable from real-world 3D scenes. To deliver all necessary visual cues for perceptual realism, we built a High-Dynamic-Range Multi-Focal Stereo Display that achieves high resolution, accurate color, a wide dynamic range, and most depth cues, including binocular presentation and a range of focal depth. The display and associated imaging system have been designed to capture and reproduce a small near-eye three-dimensional object and to allow for a direct comparison between virtual and real scenes. To assess our reproduction of realism and demonstrate the capability of the display and imaging system, we conducted an experiment in which the participants were asked to discriminate between a virtual object and its physical counterpart. Our results indicate that the participants can only detect the discrepancy with a probability of 0.44. With such a level of perceptual realism, our display apparatus can facilitate a range of visual experiments that require the highest fidelity of reproduction while allowing for the full control of the displayed stimuli.

[1]  Kevin J. MacKenzie,et al.  Vergence and accommodation to multiple-image-plane stereoscopic displays: 'Real world' responses with practical image-plane separations? , 2011, Electronic Imaging.

[2]  Giancarlo Calvagno,et al.  Demosaicing With Directional Filtering and a posteriori Decision , 2007, IEEE Transactions on Image Processing.

[3]  Hans-Peter Seidel,et al.  Depth from HDR: depth induction or increased realism? , 2014, SAP.

[4]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[5]  Wolfgang Heidrich,et al.  High dynamic range display systems , 2004, SIGGRAPH 2004.

[6]  Karol Myszkowski,et al.  Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors , 2017, IEEE Transactions on Visualization and Computer Graphics.

[7]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[8]  Gordon Wetzstein,et al.  3D displays , 1996, Annual review of vision science.

[9]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[10]  Yukihiro Nishida,et al.  Sensation of Realness From High-Resolution Images of Real Objects , 2013, IEEE Transactions on Broadcasting.

[11]  I. E. Sutherland,et al.  Three-dimensional data input by tablet , 1974, SIGGRAPH 1974.

[12]  James F. O'Brien,et al.  Optimal presentation of imagery with focus cues on multi-plane displays , 2015, ACM Trans. Graph..

[13]  Param Hanji,et al.  Noise-Aware Merging of High Dynamic Range Image Stacks without Camera Calibration , 2020, ECCV Workshops.

[14]  Kevin J. MacKenzie,et al.  Accommodation to multiple-focal-plane displays: Implications for improving stereoscopic displays and for accommodation control. , 2010, Journal of vision.

[15]  Martin Kraus,et al.  Practical Implementation of a Graphics Turing Test , 2012, ISVC.

[16]  Douglas Lanman,et al.  Fast gaze-contingent optimal decompositions for multifocal displays , 2017, ACM Trans. Graph..

[17]  Kevin J. MacKenzie,et al.  Vergence and accommodation to multiple-image-plane stereoscopic displays: "real world" responses with practical image-plane separations? , 2012, J. Electronic Imaging.

[18]  E. Reinhard Photographic Tone Reproduction for Digital Images , 2002 .

[19]  Hao Li,et al.  Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[21]  Rafal Mantiuk,et al.  A comparative review of tone‐mapping algorithms for high dynamic range video , 2017, Comput. Graph. Forum.

[22]  Donald P. Greenberg,et al.  Modeling the interaction of light between diffuse surfaces , 1984, SIGGRAPH.

[23]  Hans-Peter Seidel,et al.  Temporal Glare: Real‐Time Dynamic Simulation of the Scattering in the Human Eye , 2009, Comput. Graph. Forum.

[24]  Martin S. Banks,et al.  A stereo display prototype with multiple focal distances , 2004, SIGGRAPH 2004.

[25]  Hans-Peter Seidel,et al.  Analysis of Reproducing Real‐World Appearance on Displays of Varying Dynamic Range , 2006, Comput. Graph. Forum.

[26]  Wan-Yen Lo,et al.  Accelerating 3D deep learning with PyTorch3D , 2019, SIGGRAPH Asia 2020 Courses.

[27]  Ann McNamara Exploring perceptual equivalence between real and simulated imagery , 2005, APGV '05.

[28]  Supasorn Suwajanakorn,et al.  NeX: Real-time View Synthesis with Neural Basis Expansion , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Hans-Peter Seidel,et al.  A Perception-driven Hybrid Decomposition for Multi-layer Accommodative Displays , 2019, IEEE Transactions on Visualization and Computer Graphics.

[30]  David M. Hoffman,et al.  Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. , 2008, Journal of vision.

[31]  Ian Sexton,et al.  Emerging Autostereoscopic Displays , 2012, Handbook of Visual Display Technology.

[32]  Gordon Wetzstein,et al.  The light field stereoscope , 2015, ACM Trans. Graph..

[33]  Donald P. Greenberg,et al.  An experimental evaluation of computer graphics imagery , 1986, TOGS.

[34]  Mark E. Lucente Electronic Holographic Displays - 20 Years of Interactive Spatial Imaging , 2012, Handbook of Visual Display Technology.

[35]  B. V. K. Vijaya Kumar,et al.  Towards multifocal displays with dense focal stacks , 2018, ACM Trans. Graph..