Towards Practical and Efficient High-Resolution HDR Deghosting with CNN

Generating High Dynamic Range (HDR) image in the presence of camera and object motion is a tedious task. If uncorrected, these motions will manifest as ghosting artifacts in the fused HDR image. On one end of the spectrum, there exist methods that generate highquality results that are computationally demanding and too slow. On the other end, there are few faster methods that produce unsatisfactory results. With ever increasing sensor/display resolution, currently we are very much in need of faster methods that produce high-quality images. In this paper, we present a deep neural network based approach to generate high-quality ghost-free HDR for high-resolution images. Our proposed method is fast and fuses a sequence of three high-resolution images (16-megapixel resolution) in about 10 seconds. Through experiments and ablations, on different publicly available datasets, we show that the proposed method achieves state-of-the-art performance in terms of accuracy and speed.

[1]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, ACM Trans. Graph..

[2]  R. Venkatesh Babu,et al.  A Fast, Scalable, and Reliable Deghosting Method for Extreme Exposure Fusion , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[3]  Thorsten Grosch,et al.  Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement , 2006 .

[4]  Subhasis Chaudhuri,et al.  Reconstruction of high contrast images for dynamic scenes , 2011, The Visual Computer.

[5]  Yanning Zhang,et al.  Multi-Scale Dense Networks for Deep High Dynamic Range Imaging , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[6]  Jun Hu,et al.  Locally non-rigid registration for mobile HDR photography , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  William Puech,et al.  Ghost detection and removal in High Dynamic Range Images , 2009, 2009 17th European Signal Processing Conference.

[8]  Ravi Ramamoorthi,et al.  Deep high dynamic range imaging of dynamic scenes , 2017, ACM Trans. Graph..

[9]  Erik Reinhard,et al.  Ghost Removal in High Dynamic Range Images , 2006, 2006 International Conference on Image Processing.

[10]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[11]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[12]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting , 2010 .

[13]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Wen-Chung Kao,et al.  Integrating image fusion and motion stabilization for capturing still images in high dynamic range scenes , 2006, IEEE Transactions on Consumer Electronics.

[15]  Joachim Weickert,et al.  Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement , 2011, Comput. Graph. Forum.

[16]  Richard Szeliski,et al.  Seamless Image Stitching of Scenes with Large Motions and Exposure Differences , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[17]  Pradeep Sen Overview of State-of-the-Art Algorithms for Stack-Based High-Dynamic Range (HDR) Imaging , 2018 .

[18]  R. Venkatesh Babu,et al.  Ghosting-free multi-exposure image fusion in gradient domain , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[19]  Chi-Keung Tang,et al.  Deep High Dynamic Range Imaging with Large Foreground Motions , 2017, ECCV.

[20]  Yanning Zhang,et al.  Attention-Guided Network for Ghost-Free High Dynamic Range Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yu Zhu,et al.  Deep HDR Imaging via A Non-Local Network , 2020, IEEE Transactions on Image Processing.

[22]  Miguel Granados,et al.  Automatic noise modeling for ghost-free HDR reconstruction , 2013, ACM Trans. Graph..

[23]  Sang Uk Lee,et al.  Ghost-Free High Dynamic Range Imaging , 2010, ACCV.

[24]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[25]  Aykut Erdem,et al.  An Objective Deghosting Quality Metric for HDR Images , 2016, Comput. Graph. Forum.

[26]  Didier Stricker,et al.  SDC – Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Anna Tomaszewska,et al.  Image Registration for Multi-exposure High Dynamic Range Image Acquisition , 2007 .

[28]  Jun Hu,et al.  Exposure Stacks of Live Scenes with Hand-Held Cameras , 2012, ECCV.

[29]  Aykut Erdem,et al.  The State of the Art in HDR Deghosting: A Survey and Evaluation , 2015, Comput. Graph. Forum.

[30]  Jan Kautz,et al.  Bitmap Movement Detection: HDR for Dynamic Scenes , 2010, 2010 Conference on Visual Media Production.

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Greg Ward,et al.  Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures , 2003, J. Graphics, GPU, & Game Tools.

[33]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Rae-Hong Park,et al.  Histogram based ghost removal in high dynamic range images , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[35]  Susanto Rahardja,et al.  A robust and fast anti-ghosting algorithm for high dynamic range imaging , 2010, 2010 IEEE International Conference on Image Processing.

[36]  R. Venkatesh Babu,et al.  DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[38]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[39]  Lei Zhang,et al.  Robust Multi-Exposure Image Fusion: A Structural Patch Decomposition Approach , 2017, IEEE Transactions on Image Processing.

[40]  Wai-kuen Cham,et al.  Reference-guided exposure fusion in dynamic scenes , 2012, J. Vis. Commun. Image Represent..

[41]  Luca Bogoni,et al.  Extending dynamic range of monochrome and color images through fusion , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[42]  Jonathan T. Barron,et al.  Deep bilateral learning for real-time image enhancement , 2017, ACM Trans. Graph..

[43]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[44]  Jiawen Chen,et al.  Bilateral guided upsampling , 2016, ACM Trans. Graph..

[45]  Pradeep Sen,et al.  Practical High Dynamic Range Imaging of Everyday Scenes: Photographing the world as we see it with our own eyes , 2016, IEEE Signal Processing Magazine.

[46]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..