Natural Environment Illumination: Coherent Interactive Augmented Reality for Mobile and Non-Mobile Devices

Augmented Reality offers many applications today, especially on mobile devices. Due to the lack of mobile hardware for illumination measurements, photorealistic rendering with consistent appearance of virtual objects is still an area of active research. In this paper, we present a full two-stage pipeline for environment acquisition and augmentation of live camera images using a mobile device with a depth sensor. We show how to directly work on a recorded 3D point cloud of the real environment containing high dynamic range color values. For unknown and automatically changing camera settings, a color compensation method is introduced. Based on this, we show photorealistic augmentations using variants of differential light simulation techniques. The presented methods are tailored for mobile devices and run at interactive frame rates. However, our methods are scalable to trade performance for quality and can produce quality renderings on desktop hardware.

[1]  Dieter Schmalstieg,et al.  Instant Mixed Reality Lighting from Casual Scanning , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[2]  Hannes Kaufmann,et al.  Differential Irradiance Caching for fast high-quality light transport between virtual and real worlds , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[3]  Irfan A. Essa,et al.  Post-processing approach for radiometric self-calibration of video , 2013, IEEE International Conference on Computational Photography (ICCP).

[4]  Martin Kraus,et al.  Efficient Point-Based Rendering Using Image Reconstruction , 2007, PBG@Eurographics.

[5]  Dieter Schmalstieg,et al.  Real-time photometric registration from arbitrary geometry , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  Stefan Leutenegger,et al.  ElasticFusion: Real-time dense SLAM and light source estimation , 2016, Int. J. Robotics Res..

[7]  Tobias Höllerer,et al.  Augmented reality: principles and practice , 2016, SIGGRAPH Courses.

[8]  Elmar Eisemann,et al.  Single-pass GPU solid voxelization for real-time applications , 2008, Graphics Interface.

[9]  Michael F. Cohen,et al.  Emptying, refurnishing, and relighting indoor spaces , 2016, ACM Trans. Graph..

[10]  Peter-Pike J. Sloan,et al.  Fast and Efficient Skinning of Animated Meshes , 2010, Comput. Graph. Forum.

[11]  Tobias Alexander Franke,et al.  Delta Voxel Cone Tracing , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[12]  Takafumi Saito,et al.  Comprehensible rendering of 3-D shapes , 1990, SIGGRAPH.

[13]  Matthias Nießner,et al.  BundleFusion , 2016, TOGS.

[14]  Andrew Gardner,et al.  Photorealistic rendering of mixed reality scenes , 2015, Comput. Graph. Forum.

[15]  Kenny Mitchell,et al.  Light factorization for mixed-frequency shadows in augmented reality , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[16]  Thorsten Grosch,et al.  Tiled Frustum Culling for Differential Rendering on Mobile Devices , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[17]  Peter Kán,et al.  Interactive HDR Environment Map Capturing on Mobile Devices , 2015, Eurographics.

[18]  Toby Howard,et al.  Per Christensen and Daniel Cohen-Or (Editors) Abstract Rapid Shadow Generation in Real-World Lighting Environments , 2022 .

[19]  Martin Knecht,et al.  Differential Instant Radiosity for mixed reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[20]  Tobias Alexander Franke,et al.  Delta Light Propagation Volumes for mixed reality , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[21]  Elmar Eisemann,et al.  Interactive Indirect Illumination Using Voxel Cone Tracing , 2011, Comput. Graph. Forum.

[22]  Matthias Nießner,et al.  Real-time 3D reconstruction at scale using voxel hashing , 2013, ACM Trans. Graph..

[23]  László Szirmay-Kalos,et al.  Approximate Ray‐Tracing on the GPU with Distance Impostors , 2005, Comput. Graph. Forum.

[24]  Toshiya Hachisuka,et al.  Parallel progressive photon mapping on GPUs , 2010, SIGGRAPH ASIA.

[25]  Dan B. Goldman,et al.  Vignette and exposure calibration and compensation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[26]  Thorsten Grosch,et al.  Consistent interactive augmentation of live camera images with correct near-field illumination , 2007, VRST '07.

[27]  Hannes Kaufmann,et al.  Differential Progressive Path Tracing for High-Quality Previsualization and Relighting in Augmented Reality , 2013, ISVC.

[28]  Thorsten Grosch,et al.  Interactive near-field illumination for photorealistic augmented reality on mobile devices , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[29]  Marc Pollefeys,et al.  Robust Radiometric Calibration and Vignetting Correction , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  John J. Leonard,et al.  Real-time large-scale dense RGB-D SLAM with volumetric fusion , 2014, Int. J. Robotics Res..

[31]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[32]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[33]  Martin Knecht,et al.  Adaptive camera-based color mapping for mixed-reality applications , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[34]  Carsten Dachsbacher,et al.  Reflective shadow maps , 2005, I3D '05.

[35]  Philipp Lensing,et al.  Instant indirect illumination for dynamic mixed reality scenes , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[36]  Gustavo Patow,et al.  A Survey of Inverse Rendering Problems , 2003, Comput. Graph. Forum.

[37]  Andrew I. Comport,et al.  3D High Dynamic Range dense visual SLAM and its application to real-time object re-lighting , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[38]  Dieter Schmalstieg,et al.  Image-space illumination for augmented reality in dynamic environments , 2015, 2015 IEEE Virtual Reality (VR).

[39]  Elmar Eisemann,et al.  Interactive Indirect Illumination Using Voxel Cone Tracing , 2011, Comput. Graph. Forum.

[40]  Thorsten Grosch.  PanoAR: INTERACTIVE AUGMENTATION OF OMNI-DIRECTIONAL IMAGES WITH CONSISTENT LIGHTING , 2005 .

[41]  Thorsten Grosch,et al.  Interactive Near-Field Illumination for Photorealistic Augmented Reality with Varying Materials on Mobile Devices , 2015, IEEE Transactions on Visualization and Computer Graphics.

[42]  James T. Kajiya,et al.  The rendering equation , 1998 .

[43]  A. Fournier,et al.  Common Illumination between Real and Computer Generated Scenes , 1992 .

[44]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[45]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.