Hybrid Saturation Restoration for LDR Images of HDR Scenes

There are shadow and highlight regions in a low dynamic range (LDR) image which is captured from a high dynamic range (HDR) scene. It is an ill-posed problem to restore the saturated regions of the LDR image. In this paper, the saturated regions of the LDR image are restored by fusing model-based and data-driven approaches. With such a neural augmentation, two synthetic LDR images are first generated from the underlying LDR image via the model-based approach. One is brighter than the input image to restore the shadow regions and the other is darker than the input image to restore the high-light regions. Both synthetic images are then refined via a novel exposedness aware saturation restoration network (EASRN). Finally, the two synthetic images and the input image are combined together via an HDR synthesis algorithm or a multi-scale exposure fusion algorithm. The proposed algorithm can be embedded in any smart phones or digital cameras to produce an information-enriched LDR image.

[1]  Noah Snavely,et al.  Material recognition in the wild with the Materials in Context Database , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Yu Li,et al.  LIME: Low-Light Image Enhancement via Illumination Map Estimation , 2017, IEEE Transactions on Image Processing.

[3]  Masahiro Kobayashi,et al.  A 3.4 μm pixel pitch global shutter CMOS image sensor with dual in-pixel charge domain memory , 2019, Japanese Journal of Applied Physics.

[4]  Alexei A. Efros,et al.  Fast bilateral filtering for the display of high-dynamic-range images , 2002 .

[5]  Erik Reinhard,et al.  Photographic tone reproduction for digital images , 2002, ACM Trans. Graph..

[6]  Tomoo Mitsunaga,et al.  Coded rolling shutter photography: Flexible space-time sampling , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[7]  Yoshihiro Kanamori,et al.  Deep reverse tone mapping , 2017, ACM Trans. Graph..

[8]  Suk-Ju Kang,et al.  End-to-End Differentiable Learning to HDR Image Synthesis for Multi-exposure Images , 2020, AAAI.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Zhengguo Li,et al.  Exposure Interpolation for Two Large-Exposure-Ratio Images , 2020, IEEE Access.

[11]  Susanto Rahardja,et al.  Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects , 2013, IEEE Transactions on Image Processing.

[12]  Shiqian Wu,et al.  Single Image Brightening via Multi-Scale Exposure Fusion With Hybrid Learning , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Zeev Farbman,et al.  Edge-preserving decompositions for multi-scale tone and detail manipulation , 2008, ACM Trans. Graph..

[14]  Wei Cao,et al.  Multi-Scale Fusion of Two Large-Exposure-Ratio Images , 2018, IEEE Signal Processing Letters.

[15]  Zhengguo Li,et al.  Multi-scale exposure fusion via gradient domain guided image filtering , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[16]  Yung-Yu Chuang,et al.  Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Monson H. Hayes,et al.  Single image-based ghost-free high dynamic range imaging using local histogram stretching and spatially-adaptive denoising , 2011, IEEE Transactions on Consumer Electronics.

[18]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[19]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[20]  Chi-Keung Tang,et al.  Deep High Dynamic Range Imaging with Large Foreground Motions , 2017, ECCV.

[21]  Shiqian Wu,et al.  Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects , 2014, IEEE Transactions on Image Processing.

[22]  Zhengfang Duanmu,et al.  Deep Guided Learning for Fast Multi-Exposure Image Fusion , 2019, IEEE Transactions on Image Processing.

[23]  Fahad Shahbaz Khan,et al.  CycleISP: Real Image Restoration via Improved Data Synthesis , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Ge Li,et al.  LECARM: Low-Light Image Enhancement Using the Camera Response Model , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[25]  Francesco Banterle,et al.  Inverse tone mapping , 2006, GRAPHITE '06.

[26]  Chi-Wing Fu,et al.  Underexposed Photo Enhancement Using Deep Illumination Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  David Zhang,et al.  A Hybrid l1-l0 Layer Decomposition Model for Tone Mapping , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Kai Zeng,et al.  Perceptual Quality Assessment for Multi-Exposure Image Fusion , 2015, IEEE Transactions on Image Processing.

[29]  Weihai Chen,et al.  Restoration of HDR Images for SVE-Based HDRI via a Novel DCNN , 2021, 2021 IEEE International Conference on Multimedia and Expo (ICME).

[30]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[31]  Hui Li,et al.  Fast Multi-Scale Structural Patch Decomposition for Multi-Exposure Image Fusion , 2020, IEEE Transactions on Image Processing.

[32]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, ACM Trans. Graph..

[33]  Jan Kautz,et al.  Exposure Fusion , 2009, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[34]  Erik Reinhard,et al.  Do HDR displays support LDR content?: a psychophysical evaluation , 2007, ACM Trans. Graph..

[35]  Jun Hu,et al.  Exposure Stacks of Live Scenes with Hand-Held Cameras , 2012, ECCV.

[36]  Jun Fu,et al.  Dual Attention Network for Scene Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[38]  Christophe De Vleeschouwer,et al.  Single-Scale Fusion: An Effective Approach to Merging Images , 2017, IEEE Trans. Image Process..

[39]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[41]  Susanto Rahardja,et al.  Noise reduction for differently exposed images , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[42]  Thomas Bashford-Rogers,et al.  ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content , 2018, Comput. Graph. Forum.

[43]  Nam Ik Cho,et al.  Generation of high dynamic range illumination from a single image for the enhancement of undesirably illuminated images , 2017, Multimedia Tools and Applications.

[44]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Zhan Ma,et al.  Multi-Camera Color Correction via Hybrid Histogram Matching , 2020 .

[46]  Ing Ren Tsang,et al.  Single image HDR reconstruction using a CNN with masked features and perceptual loss , 2020, ACM Trans. Graph..

[47]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.