Indoor Lighting Estimation using an Event Camera

Image-based methods for indoor lighting estimation suffer from the problem of intensity-distance ambiguity. This paper introduces a novel setup to help alleviate the ambiguity based on the event camera. We further demonstrate that estimating the distance of a light source becomes a well-posed problem under this setup, based on which an optimization-based method and a learning-based method are proposed. Our experimental results validate that our approaches not only achieve superior performance for indoor lighting estimation (especially for the close light) but also significantly alleviate the intensity-distance ambiguity.

[1]  Wan-Chun Ma,et al.  DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Stephen Lin,et al.  Faces as Lighting Probes via Unsupervised Deep Highlight Extraction , 2018, ECCV.

[3]  Tiejun Huang,et al.  Neuromorphic Camera Guided High Dynamic Range Imaging , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Shih-Chii Liu,et al.  Learning to Exploit Multiple Vision Modalities by Using Grafted Networks , 2020, ECCV.

[5]  Vladlen Koltun,et al.  Events-To-Video: Bringing Modern Computer Vision to Event Cameras , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Lin Wang,et al.  EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Oliver Cossairt,et al.  Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Kostas Daniilidis,et al.  Realtime Time Synchronized Event-based Stereo , 2018, ECCV.

[9]  Davide Scaramuzza,et al.  End-to-End Learning of Representations for Asynchronous Event-Based Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Hyun-Ho Kim,et al.  Thermal transient characteristics of die attach in high power LED PKG , 2008, Microelectron. Reliab..

[11]  Yannick Hold-Geoffroy,et al.  Deep Parametric Indoor Lighting Estimation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Zhe Wu,et al.  A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo , 2019, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ling-Yu Duan,et al.  What Does Plate Glass Reveal About Camera Calibration? , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Kenny Mitchell,et al.  From Faces to Outdoor Light Probes , 2018, Comput. Graph. Forum.

[15]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[16]  Hong Yang,et al.  DART: Distribution Aware Retinal Transform for Event-Based Cameras , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Kalyan Sunkavalli,et al.  Automatic Scene Inference for 3D Object Compositing , 2014, ACM Trans. Graph..

[18]  Narciso García,et al.  Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  M. Matteucci,et al.  A Differentiable Recurrent Surface for Asynchronous Event-Based Data , 2020, ECCV.

[20]  Noah Snavely,et al.  Neural Rerendering in the Wild , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[22]  Kalyan Sunkavalli,et al.  Fast Spatially-Varying Indoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Yannick Hold-Geoffroy,et al.  Deep Sky Modeling for Single Image Outdoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Vladlen Koltun,et al.  High Speed and High Dynamic Range Video with an Event Camera , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Jonghyun Choi,et al.  Learning to Super Resolve Intensity Images From Events , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Jean-François Lalonde,et al.  Learning to Estimate Indoor Lighting from 3D Objects , 2018, 2018 International Conference on 3D Vision (3DV).

[27]  Dongqing Zou,et al.  Learning Event-Based Motion Deblurring , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Davide Scaramuzza,et al.  A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Gang Pan,et al.  Unsupervised AER Object Recognition Based on Multiscale Spatio-Temporal Features and Spiking Neurons , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[32]  Kirk Y. W. Scheper,et al.  Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Yiannis Aloimonos,et al.  Learning Visual Motion Segmentation Using Event Surfaces , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Davide Scaramuzza,et al.  Event-Based, 6-DOF Camera Tracking from Photometric Depth Maps , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Davide Scaramuzza,et al.  Focus Is All You Need: Loss Functions for Event-Based Vision , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Davide Scaramuzza,et al.  Asynchronous, Photometric Feature Tracking using Events and Frames , 2018, ECCV.

[37]  Luc Van Gool,et al.  What is Around the Camera? , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  Ajay Kumar,et al.  Numerical Reflectance Compensation for Non-Lambertian Photometric Stereo , 2019, IEEE Transactions on Image Processing.

[39]  Miao Liao,et al.  Light Fall-off Stereo , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Ryad Benosman,et al.  HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Yi Zhou,et al.  Semi-Dense 3D Reconstruction with a Stereo Event Camera , 2018, ECCV.

[42]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[43]  Tom Drummond,et al.  Event-Based Motion Segmentation by Motion Compensation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[45]  Mario Fritz,et al.  Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  R. Mahony,et al.  Reducing the Sim-to-Real Gap for Event Cameras , 2020, ECCV.

[47]  Miaomiao Liu,et al.  Single Image Optical Flow Estimation With an Event Camera , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Wolfgang Heidrich,et al.  Stereo Event-Based Particle Tracking Velocimetry for 3D Fluid Flow Reconstruction , 2020, ECCV.

[49]  Alexander Wilkie,et al.  An analytic model for full spectral sky-dome radiance , 2012, ACM Trans. Graph..

[50]  Anders Ynnerman,et al.  Spatially varying image based lighting by light probe sequences , 2007, The Visual Computer.

[51]  Alexei A. Efros,et al.  Estimating natural illumination from a single outdoor image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[52]  Jonathan T. Barron,et al.  Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Jean-Yves Audibert.  Optimization for Machine Learning , 1995 .

[54]  Jeong Joon Park,et al.  Seeing the World in a Bag of Chips , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Hideo Saito,et al.  EventNet: Asynchronous Recursive Event Processing , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Gang Pan,et al.  Effective AER Object Classification Using Segmented Probability-Maximization Learning in Spiking Neural Networks , 2020, AAAI.

[57]  T. Delbruck,et al.  A 128 128 120 dB 15 s Latency Asynchronous Temporal Contrast Vision Sensor , 2006 .

[58]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Thomas Funkhouser,et al.  Neural Illumination: Lighting Prediction for Indoor Environments , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Vincent Lepetit,et al.  Speed Invariant Time Surface for Learning to Detect Corner Points With Event-Based Cameras , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Bowen Du,et al.  EV-Gait: Event-Based Robust Gait Recognition Using Dynamic Vision Sensors , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Kostas Daniilidis,et al.  Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Xin Yu,et al.  Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Kalyan Sunkavalli,et al.  Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Tat-Jun Chin,et al.  Globally Optimal Contrast Maximisation for Event-Based Motion Estimation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Katsushi Ikeuchi,et al.  Determining Surface Orientations of Specular Surfaces by Using the Photometric Stereo Method , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Alexander Andreopoulos,et al.  A Low Power, High Throughput, Fully Event-Based Stereo System , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[68]  Jing Chen,et al.  Learning Event-Driven Video Deblurring and Interpolation , 2020, ECCV.

[69]  Yiannis Andreopoulos,et al.  Graph-Based Object Classification for Neuromorphic Vision Sensing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[70]  Raquel Urtasun,et al.  Efficient Exact Inference for 3D Indoor Scene Understanding , 2012, ECCV.

[71]  Ryad Benosman,et al.  Real-Time High Speed Motion Prediction Using Fast Aperture-Robust Event-Driven Visual Flow , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Christian Theobalt,et al.  EventCap: Monocular 3D Capture of High-Speed Human Motions Using an Event Camera , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Yannick Hold-Geoffroy,et al.  All-Weather Deep Outdoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Kenny Mitchell,et al.  The shading probe: fast appearance acquisition for mobile AR , 2013, SA '13.

[75]  Paul Graham,et al.  A single-shot light probe , 2012, SIGGRAPH '12.

[76]  Peter V. Gehler,et al.  Learning an Event Sequence Embedding for Dense Event-Based Deep Stereo , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).