Deep high dynamic range imaging of dynamic scenes

Producing a high dynamic range (HDR) image from a set of images with different exposures is a challenging process for dynamic scenes. A category of existing techniques first register the input images to a reference image and then merge the aligned images into an HDR image. However, the artifacts of the registration usually appear as ghosting and tearing in the final HDR images. In this paper, we propose a learning-based approach to address this problem for dynamic scenes. We use a convolutional neural network (CNN) as our learning model and present and compare three different system architectures to model the HDR merge process. Furthermore, we create a large dataset of input LDR images and their corresponding ground truth HDR images to train our system. We demonstrate the performance of our system by producing high-quality HDR images from a set of three LDR images. Experimental results show that our method consistently produces better results than several state-of-the-art approaches on challenging scenes.

[1]  Chul Lee,et al.  Ghost-Free High Dynamic Range Imaging via Rank Minimization , 2014, IEEE Signal Processing Letters.

[2]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, SIGGRAPH 2011.

[4]  Lilong Shi,et al.  The Rehabilitation of MaxRGB , 2010, CIC.

[5]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[6]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[7]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[8]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[9]  Shiqian Wu,et al.  Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects , 2014, IEEE Transactions on Image Processing.

[10]  Susanto Rahardja,et al.  Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects , 2013, IEEE Transactions on Image Processing.

[11]  Wolfgang Heidrich,et al.  HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions , 2011, ACM Trans. Graph..

[12]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[13]  Dani Lischinski,et al.  Non-rigid dense correspondence with applications for image enhancement , 2011, ACM Trans. Graph..

[14]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[15]  Subhasis Chaudhuri,et al.  Reconstruction of high contrast images for dynamic scenes , 2011, The Visual Computer.

[16]  Jun Hu,et al.  Locally non-rigid registration for mobile HDR photography , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Gordon Wetzstein,et al.  Convolutional Sparse Coding for High Dynamic Range Imaging , 2016, Comput. Graph. Forum.

[18]  Ying Wu,et al.  Large Displacement Optical Flow from Nearest Neighbor Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Greg Ward,et al.  Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures , 2003, J. Graphics, GPU, & Game Tools.

[20]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[21]  Bin Sheng,et al.  Deep Colorization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  John Flynn,et al.  Deep Stereo: Learning to Predict New Views from the World's Imagery , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[24]  Rafal Mantiuk,et al.  Subjective and Objective Evaluation of Multi-exposure High Dynamic Range Image Deghosting Methods , 2016, Eurographics.

[25]  Aykut Erdem,et al.  An Objective Deghosting Quality Metric for HDR Images , 2016, Comput. Graph. Forum.

[26]  Erik Reinhard,et al.  Ghost Removal in High Dynamic Range Images , 2006, 2006 International Conference on Image Processing.

[27]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[28]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[29]  Masahiro Okuda,et al.  Motion blur free HDR image acquisition using multiple exposures , 2008, 2008 15th IEEE International Conference on Image Processing.

[30]  Anna Tomaszewska,et al.  Image Registration for Multi-exposure High Dynamic Range Image Acquisition , 2007 .

[31]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Hans-Peter Seidel,et al.  Optimal HDR reconstruction with linear digital cameras , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[34]  Jonathan T. Barron,et al.  Burst photography for high dynamic range and low-light imaging on mobile cameras , 2016, ACM Trans. Graph..

[35]  Tae-Hyun Oh,et al.  Robust High Dynamic Range Imaging by Rank Minimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Harald Brendel,et al.  Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays , 2014, Electronic Imaging.

[37]  Thorsten Grosch,et al.  Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement , 2006 .

[38]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[39]  Luca Bogoni,et al.  Extending dynamic range of monochrome and color images through fusion , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[40]  Wai-kuen Cham,et al.  Gradient-Directed Multiexposure Composition , 2012, IEEE Transactions on Image Processing.

[41]  Jun Hu,et al.  Exposure Stacks of Live Scenes with Hand-Held Cameras , 2012, ECCV.

[42]  Aykut Erdem,et al.  The State of the Art in HDR Deghosting: A Survey and Evaluation , 2015, Comput. Graph. Forum.

[43]  Jan Kautz,et al.  Bitmap Movement Detection: HDR for Dynamic Scenes , 2010, 2010 Conference on Visual Media Production.

[44]  Joachim Weickert,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement , 2022 .

[45]  Pradeep Sen,et al.  Robust Radiometric Calibration for Dynamic Scenes in the Wild , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[46]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[47]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[48]  Kari Pulli,et al.  FlexISP , 2014, ACM Trans. Graph..

[49]  Miguel Granados,et al.  Automatic noise modeling for ghost-free HDR reconstruction , 2013, ACM Trans. Graph..

[50]  Jonas Unger,et al.  Adaptive dualISO HDR reconstruction , 2015, EURASIP J. Image Video Process..

[51]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[52]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Ramesh Raskar,et al.  Unbounded High Dynamic Range Photography Using a Modulo Camera , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[54]  Shree K. Nayar,et al.  Optical Splitting Trees for High-Precision Monocular Imaging , 2007, IEEE Computer Graphics and Applications.

[55]  Greg Ward,et al.  Automatic High-Dynamic Range Image Generation for Dynamic Scenes , 2008, IEEE Computer Graphics and Applications.

[56]  Li Zhang,et al.  Denoising vs. deblurring: HDR imaging techniques using moving cameras , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[57]  Shree K. Nayar,et al.  Determining the Camera Response from Images: What Is Knowable? , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Sang Uk Lee,et al.  Ghost-Free High Dynamic Range Imaging , 2010, ACCV.