Stack-Based Algorithms for HDR Capture and Reconstruction

Abstract High-dynamic-range (HDR) images can be created with standard camera hardware by the capturing and combining of multiple pictures, each sampling a different segment of the irradiance distribution of a scene. This seemingly straightforward process involves several important steps, which will be the focus of this chapter. We start by examining the problem of selecting the set of exposures that properly measures the full dynamic range of a particular scene, a process known as metering for HDR. We then describe how to perform radiometric calibration, which is needed to estimate the incoming irradiance from the low-dynamic-range images. After that, we offer an overview of the methods to merge multiple low-dynamic-range images into a single HDR image. Finally, we discuss methods to compensate for camera and scene motion, which would otherwise cause artifacts in the final HDR image.

[1]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Hans Jørgen Andersen,et al.  Noise Characterization of Weighting Schemes for Combination of Multiple Exposures , 2006, BMVC.

[3]  Irfan A. Essa,et al.  Post-processing approach for radiometric self-calibration of video , 2013, IEEE International Conference on Computational Photography (ICCP).

[4]  Natasha Gelfand,et al.  Multi-exposure imaging on mobile devices , 2010, ACM Multimedia.

[5]  Hany Farid,et al.  Blind inverse gamma correction , 2001, IEEE Trans. Image Process..

[6]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[7]  Masahiro Okuda,et al.  Motion blur free HDR image acquisition using multiple exposures , 2008, 2008 15th IEEE International Conference on Image Processing.

[8]  Abbas El Gamal,et al.  Optimal scheduling of capture times in a multiple-capture imaging system , 2002, IS&T/SPIE Electronic Imaging.

[9]  Wai-kuen Cham,et al.  Reference-guided exposure fusion in dynamic scenes , 2012, J. Vis. Commun. Image Represent..

[10]  Erik Reinhard,et al.  Ghost Removal in High Dynamic Range Images , 2006, 2006 International Conference on Image Processing.

[11]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[12]  Sang Uk Lee,et al.  Ghost-Free High Dynamic Range Imaging , 2010, ACCV.

[13]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[14]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Rafal Mantiuk,et al.  Comparison of Deghosting Algorithms for Multi-exposure High Dynamic Range Imaging , 2013, SCCG.

[16]  Radu Ciprian Bilcu,et al.  High dynamic range imaging on mobile devices , 2008, 2008 15th IEEE International Conference on Electronics, Circuits and Systems.

[17]  Anna Tomaszewska,et al.  Image Registration for Multi-exposure High Dynamic Range Image Acquisition , 2007 .

[18]  B. C. Madden,et al.  Extended Intensity Range Imaging , 1993 .

[19]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics) , 2005 .

[20]  Dani Lischinski,et al.  Non-rigid dense correspondence with applications for image enhancement , 2011, ACM Trans. Graph..

[21]  Nam Ik Cho,et al.  A multi-exposure image fusion algorithm without ghost effect , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[23]  Steve Mann,et al.  Comparametric equations with practical applications in quantigraphic image processing , 2000, IEEE Trans. Image Process..

[24]  Hans-Peter Seidel,et al.  Optimal HDR reconstruction with linear digital cameras , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  John Wright,et al.  RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Trevor Darrell,et al.  From pixels to physics: Probabilistic color de-rendering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Subhasis Chaudhuri,et al.  Reconstruction of high contrast images for dynamic scenes , 2011, The Visual Computer.

[28]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[29]  Jun Hu,et al.  Locally non-rigid registration for mobile HDR photography , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[30]  Susanto Rahardja,et al.  A robust and fast anti-ghosting algorithm for high dynamic range imaging , 2010, 2010 IEEE International Conference on Image Processing.

[31]  Shree K. Nayar,et al.  Determining the Camera Response from Images: What Is Knowable? , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Greg Ward,et al.  Automatic High-Dynamic Range Image Generation for Dynamic Scenes , 2008, IEEE Computer Graphics and Applications.

[33]  Takeo Kanade,et al.  Statistical calibration of CCD imaging process , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[34]  Desire Sidibé,et al.  Ghost detection and removal for high dynamic range images: Recent advances , 2012, Signal Process. Image Commun..

[35]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Jianbo Shi,et al.  Generalized Random Walks for Fusion of Multi-Exposure Images , 2011, IEEE Transactions on Image Processing.

[37]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Luca Bogoni,et al.  Extending dynamic range of monochrome and color images through fusion , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[39]  Eli Shechtman,et al.  Regenerative morphing , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[41]  Henry Enfield Roscoe,et al.  III. Photochemical researches.—Part V. On the measurement of the chemical action of direct and diffuse sunlight , 1863, Proceedings of the Royal Society of London.

[42]  Robert L. Stevenson,et al.  Estimation-theoretic approach to dynamic range enhancement using multiple exposures , 2003, J. Electronic Imaging.

[43]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[44]  Joachim Weickert,et al.  Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement , 2011, Comput. Graph. Forum.

[45]  Pradeep Sen,et al.  Robust Radiometric Calibration for Dynamic Scenes in the Wild , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[46]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[47]  Jan Kautz,et al.  Exposure Fusion , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[48]  Katsushi Ikeuchi,et al.  Radiometric Calibration by Rank Minimization , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Greg Ward,et al.  Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures , 2003, J. Graphics, GPU, & Game Tools.

[50]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[52]  Shree K. Nayar,et al.  What is the space of camera response functions? , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[53]  Jitendra Malik,et al.  Large displacement optical flow , 2009, CVPR.

[54]  Stephen Lin,et al.  Radiometric Calibration from Noise Distributions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  William Puech,et al.  Ghost detection and removal in High Dynamic Range Images , 2009, 2009 17th European Signal Processing Conference.

[56]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[57]  Tae-Hyun Oh,et al.  Robust High Dynamic Range Imaging by Rank Minimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Kristin J. Dana,et al.  Real-time scene stabilization and mosaic construction , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[59]  Wai-kuen Cham,et al.  Gradient-directed composition of multi-exposure images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[60]  Robert L. Stevenson,et al.  Dynamic range improvement through multiple exposures , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[61]  Stephen Lin,et al.  A New In-Camera Imaging Model for Color Computer Vision and Its Application , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Stephen Lin,et al.  Radiometric calibration from a single image , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[63]  Jun Hu,et al.  Exposure Stacks of Live Scenes with Hand-Held Cameras , 2012, ECCV.

[64]  Aykut Erdem,et al.  The State of the Art in HDR Deghosting: A Survey and Evaluation , 2015, Comput. Graph. Forum.

[65]  Roberto Manduchi,et al.  Metering for Exposure Stacks , 2012, Comput. Graph. Forum.

[66]  Jan Kautz,et al.  Bitmap Movement Detection: HDR for Dynamic Scenes , 2010, 2010 Conference on Visual Media Production.

[67]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[68]  Richard Szeliski,et al.  Seamless Image Stitching of Scenes with Large Motions and Exposure Differences , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[69]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[70]  Miguel Granados,et al.  Automatic noise modeling for ghost-free HDR reconstruction , 2013, ACM Trans. Graph..

[71]  Thorsten Grosch,et al.  Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement , 2006 .

[72]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.