Beyond Visual Attractiveness: Physically Plausible Single Image HDR Reconstruction for Spherical Panoramas

HDR reconstruction is an important task in computer vision with many industrial needs. The traditional approaches merge multiple exposure shots to generate HDRs that correspond to the physical quantity of illuminance of the scene. However, the tedious capturing process makes such multi-shot approaches inconvenient in practice. In contrast, recent single-shot methods predict a visually appealing HDR from a single LDR image through deep learning. But it is not clear whether the previously mentioned physical properties would still hold, without training the network to explicitly model them. In this paper, we introduce the physical illuminance constraints to our single-shot HDR reconstruction framework, with a focus on spherical panoramas. By the proposed physical regularization, our method can generate HDRs which are not only visually appealing but also physically plausible. For evaluation, we collect a large dataset of LDR and HDR images with ground truth illuminance measures. Extensive experiments show that our HDR images not only maintain high visual quality but also top all baseline methods in illuminance prediction accuracy.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Ramesh Raskar,et al.  Unbounded High Dynamic Range Photography Using a Modulo Camera , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[3]  Mehlika Inanici,et al.  Evaluation of high dynamic range photography as a luminance data acquisition system , 2006 .

[4]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[5]  Suk-Ju Kang,et al.  Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks , 2018, ECCV.

[6]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[7]  Ing Ren Tsang,et al.  Single image HDR reconstruction using a CNN with masked features and perceptual loss , 2020, ACM Trans. Graph..

[8]  Thomas Bashford-Rogers,et al.  ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content , 2018, Comput. Graph. Forum.

[9]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, ACM Trans. Graph..

[11]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[12]  Martin Moeck,et al.  Illuminance Analysis from High Dynamic Range Images , 2006 .

[13]  Yoshihiro Kanamori,et al.  Deep reverse tone mapping , 2017, ACM Trans. Graph..

[14]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[15]  Zicheng Liu,et al.  Face relighting with radiance environment maps , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Mehlika Inanici Evalution of High Dynamic Range Image-Based Sky Models in Lighting Simulation , 2010 .

[19]  David Zhang,et al.  A Hybrid l1-l0 Layer Decomposition Model for Tone Mapping , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Alan Chalmers,et al.  Evaluation of tone mapping operators using a High Dynamic Range display , 2005, ACM Trans. Graph..

[21]  Jan Wienold,et al.  Tutorial: Luminance Maps for Daylighting Studies from High Dynamic Range Photography , 2020 .

[22]  Gabriel Eilertsen,et al.  HDR image reconstruction from a single exposure using deep CNNs , 2017, ACM Trans. Graph..

[23]  Ersin Yumer,et al.  Learning to predict indoor illumination from a single image , 2017, ACM Trans. Graph..

[24]  Yannick Hold-Geoffroy,et al.  Deep Sky Modeling for Single Image Outdoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[26]  Arnaud Deneyer,et al.  Comparison of the Vignetting Effects of Two Identical Fisheye Lenses , 2012 .

[27]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[28]  Yu Zhu,et al.  Deep HDR Imaging via A Non-Local Network , 2020, IEEE Transactions on Image Processing.

[29]  Mike Wilson,et al.  DETERMINING LENS VIGNETTING WITH HDR TECHNIQUES , 2007 .

[30]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[31]  Chi-Keung Tang,et al.  Deep High Dynamic Range Imaging with Large Foreground Motions , 2017, ECCV.

[32]  Suk-Ju Kang,et al.  Deep Chain HDRI: Reconstructing a High Dynamic Range Image from a Single Low Dynamic Range Image , 2018, IEEE Access.

[33]  Yung-Yu Chuang,et al.  Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  G. W. Larson,et al.  Rendering with radiance - the art and science of lighting visualization , 2004, Morgan Kaufmann series in computer graphics and geometric modeling.

[36]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[37]  Wan-Chun Ma,et al.  DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Max Welling,et al.  Spherical CNNs , 2018, ICLR.

[39]  Jinsong Zhang,et al.  Learning High Dynamic Range from Outdoor Panoramas , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[40]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).