ADNet: Attention-guided Deformable Convolutional Network for High Dynamic Range Imaging

In this paper, we present an attention-guided deformable convolutional network for hand-held multi-frame high dynamic range (HDR) imaging, namely ADNet. This problem comprises two intractable challenges of how to handle saturation and noise properly and how to tackle misalignments caused by object motion or camera jittering. To address the former, we adopt a spatial attention module to adaptively select the most appropriate regions of various expo-sure low dynamic range (LDR) images for fusion. For the latter one, we propose to align the gamma-corrected images in the feature-level with a Pyramid, Cascading and Deformable (PCD) alignment module. The proposed AD-Net shows state-of-the-art performance compared with previous methods, achieving a PSNR-l of 39.4471 and a PSNR-μ of 37.6359 in NTIRE 2021 Multi-Frame HDR Challenge.

[1]  Jian Sun,et al.  MeshFlow: Minimum Latency Online Video Stabilization , 2016, ECCV.

[2]  Chen Change Loy,et al.  EDVR: Video Restoration With Enhanced Deformable Convolutional Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Yu Zhu,et al.  Deep HDR Imaging via A Non-Local Network , 2020, IEEE Transactions on Image Processing.

[4]  Shree K. Nayar,et al.  High dynamic range imaging: spatially varying pixel exposures , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[5]  Chi-Keung Tang,et al.  Deep High Dynamic Range Imaging with Large Foreground Motions , 2017, ECCV.

[6]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[7]  Erik Reinhard,et al.  Ghost Removal in High Dynamic Range Images , 2006, 2006 International Conference on Image Processing.

[8]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[9]  Masahiro Okuda,et al.  Motion blur free HDR image acquisition using multiple exposures , 2008, 2008 15th IEEE International Conference on Image Processing.

[10]  Ing Ren Tsang,et al.  Single image HDR reconstruction using a CNN with masked features and perceptual loss , 2020, ACM Trans. Graph..

[11]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[12]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Wai-kuen Cham,et al.  Gradient-Directed Multiexposure Composition , 2012, IEEE Transactions on Image Processing.

[14]  Joachim Weickert,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement , 2022 .

[15]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[16]  Suk-Ju Kang,et al.  Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks , 2018, ECCV.

[17]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[18]  Jan Kautz,et al.  Bitmap Movement Detection: HDR for Dynamic Scenes , 2010, 2010 Conference on Visual Media Production.

[19]  Jan Kautz,et al.  Exposure Fusion , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[20]  Chen Change Loy,et al.  Understanding Deformable Alignment in Video Super-Resolution , 2020, AAAI.

[21]  Thomas A. Funkhouser,et al.  Dilated Residual Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Thorsten Grosch,et al.  Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement , 2006 .

[23]  Radu Timofte,et al.  NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[25]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[26]  Jian Sun,et al.  UPFlow: Upsampling Pyramid for Unsupervised Optical Flow Learning , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[28]  Yoshihiro Kanamori,et al.  Deep reverse tone mapping , 2017, ACM Trans. Graph..

[29]  Tae-Hyun Oh,et al.  Robust High Dynamic Range Imaging by Rank Minimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Yung-Yu Chuang,et al.  Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Yanning Zhang,et al.  Multi-Scale Dense Networks for Deep High Dynamic Range Imaging , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[32]  Greg Ward,et al.  Automatic High-Dynamic Range Image Generation for Dynamic Scenes , 2008, IEEE Computer Graphics and Applications.

[33]  Ramesh Raskar,et al.  Why I Want a Gradient Camera , 2022 .

[34]  Luca Bogoni,et al.  Extending dynamic range of monochrome and color images through fusion , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[35]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Zhengfang Duanmu,et al.  Deep Guided Learning for Fast Multi-Exposure Image Fusion , 2019, IEEE Transactions on Image Processing.

[37]  Gabriel Eilertsen,et al.  HDR image reconstruction from a single exposure using deep CNNs , 2017, ACM Trans. Graph..