Seeing in Extra Darkness Using a Deep-Red Flash

We propose a new flash technique for low-light imaging, using deep-red light as an illuminating source. Our main observation is that in a dim environment, the human eye mainly uses rods for the perception of light, which are not sensitive to wavelengths longer than 620nm, yet the camera sensor still has a spectral response. We propose a novel modulation strategy when training a modern CNN model for guided image filtering, fusing a noisy RGB frame and a flash frame. This fusion network is further extended for video reconstruction. We have built a prototype with minor hardware adjustments and tested the new flash technique on a variety of static and dynamic scenes. The experimental results demonstrate that our method produces compelling reconstructions, even in extra dim conditions.

[1]  F. Durand,et al.  Flash photography enhancement via intrinsic relighting , 2004, ACM Trans. Graph..

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[4]  T. J. T. P. Van Den Berg,et al.  On the relation between glare and straylight , 2004, Documenta Ophthalmologica.

[5]  M. Alpern Effect of a Bright Light Flash on Dark Adaptation of Human Rods , 1971, Nature.

[6]  Tariq M Aslam,et al.  Principles of disability glare measurement: an ophthalmological perspective. , 2007, Acta ophthalmologica Scandinavica.

[7]  In-So Kweon,et al.  Simultaneous Estimation of Near IR BRDF and Fine-Scale Surface Geometry , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Lizhuang Ma,et al.  Multispectral Joint Image Restoration via Optimizing a Scale Map , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[10]  R. Fergus,et al.  Dark flash photography , 2009, ACM Trans. Graph..

[11]  Jan Kautz,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Orly Liba,et al.  Handheld mobile photography in very low light , 2019, ACM Trans. Graph..

[13]  Marjukka Eloholma,et al.  New Model for Mesopic Photometry and its Application to Road Lighting , 2006 .

[14]  Ersin Yumer,et al.  Learning Blind Video Temporal Consistency , 2018, ECCV.

[15]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[16]  Minh N. Do,et al.  Seeing Motion in the Dark , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Sabine Süsstrunk,et al.  Designing color filter arrays for the joint capture of visible and near-infrared images , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[18]  C. Landis Determinants of the critical flicker-fusion threshold. , 1954, Physiological reviews.

[19]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Feng Liu,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries in Wavelet Domain , 2009, 2009 Fifth International Conference on Image and Graphics.

[22]  Kiriakos N. Kutulakos,et al.  High Resolution Photography with an RGB-Infrared Camera , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[23]  D. J. Bradley Night Vision: Basic, Clinical and Applied Aspects , 1991 .

[24]  Jianhua Wu,et al.  MBLLEN: Low-Light Image/Video Enhancement Using CNNs , 2018, BMVC.

[25]  Randy J. Nelson,et al.  Effects of Light at Night on Laboratory Animals and Research Outcomes , 2018, Behavioral neuroscience.

[26]  Jonathan T. Barron,et al.  Burst photography for high dynamic range and low-light imaging on mobile cameras , 2016, ACM Trans. Graph..

[27]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[29]  Yinqiang Zheng,et al.  Learning to See Moving Objects in the Dark , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[31]  Wolfgang Heidrich,et al.  Display considerations for night and low-illumination viewing , 2009, APGV '09.

[32]  James A. Ferwerda,et al.  Fundamentals of Spatial Vision , 1998 .

[33]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[34]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[35]  Amit K. Agrawal,et al.  Removing photography artifacts using gradient projection and flash-exposure sampling , 2005, ACM Trans. Graph..

[36]  Wolfgang Heidrich,et al.  Robust Joint Image Reconstruction from Color and Monochrome Cameras , 2019, BMVC.

[37]  Jian Wang,et al.  Stereoscopic Dark Flash for Low-light Photography , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[38]  Jian Sun,et al.  Fast burst images denoising , 2014, ACM Trans. Graph..

[39]  T. Chan,et al.  Edge-preserving and scale-dependent properties of total variation regularization , 2003 .

[40]  Marc Levoy,et al.  Simulating the Visual Experience of Very Bright and Very Dark Scenes , 2015, ACM Trans. Graph..

[41]  Hiroshi Takahashi,et al.  A 1/4-inch 8Mpixel back-illuminated stacked CMOS image sensor , 2013, 2013 IEEE International Solid-State Circuits Conference Digest of Technical Papers.

[42]  A. Stockman,et al.  The spectral sensitivities of the middle- and long-wavelength-sensitive cones derived from measurements in observers of known genotype , 2000, Vision Research.

[43]  Narendra Ahuja,et al.  Joint Image Filtering with Deep Convolutional Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  A. Kelber Vision: Rods See in Bright Light , 2018, Current Biology.