Motion Aware Exposure Bracketing for HDR Video

Mobile phones and tablets are rapidly gaining significance as omnipresent image and video capture devices. In this context we present an algorithm that allows such devices to capture high dynamic range (HDR) video. The design of the algorithm was informed by a perceptual study that assesses the relative importance of motion and dynamic range. We found that ghosting artefacts are more visually disturbing than a reduction in dynamic range, even if a comparable number of pixels is affected by each. We incorporated these findings into a real‐time, adaptive metering algorithm that seamlessly adjusts its settings to take exposures that will lead to minimal visual artefacts after recombination into an HDR sequence. It is uniquely suitable for real‐time selection of exposure settings. Finally, we present an off‐line HDR reconstruction algorithm that is matched to the adaptive nature of our real‐time metering approach.

[1]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Scott Daly,et al.  A Psychophysical Study Exploring Judder Using Fundamental Signals and Complex Imagery , 2014 .

[3]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[4]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[5]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[6]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[7]  Marcus A. Magnor,et al.  Temporal Video Filtering and Exposure Control for Perceptual Motion Blur , 2015, IEEE Transactions on Visualization and Computer Graphics.

[8]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[9]  Anita Sellent,et al.  A ghosting artifact detector for interpolated image quality assessment , 2009, IEEE International Symposium on Consumer Electronics (ISCE 2010).

[10]  Aykut Erdem,et al.  The State of the Art in HDR Deghosting: A Survey and Evaluation , 2015, Comput. Graph. Forum.

[11]  Frédo Durand,et al.  Antialiasing for automultiscopic 3D displays , 2006, EGSR '06.

[12]  Roberto Manduchi,et al.  Metering for Exposure Stacks , 2012, Comput. Graph. Forum.

[13]  Seungyong Lee,et al.  Single‐shot High Dynamic Range Imaging Using Coded Electronic Shutter , 2014, Comput. Graph. Forum.

[14]  Wolfgang Effelsberg,et al.  Determining exposure values from HDR histograms for smartphone photography , 2013, MM '13.

[15]  Laurie M. Wilcox,et al.  Determinants of perceived image quality: ghosting vs. brightness , 2003, IS&T/SPIE Electronic Imaging.

[16]  Anita Sellent,et al.  A ghosting artifact detector for interpolated image quality assessment , 2010, ISCE 2010.

[17]  Mickaël Raulet,et al.  HDS, a real-time multi-DSP motion estimator for MPEG-4 H.264 AVC high definition video encoding , 2008, Journal of Real-Time Image Processing.

[18]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[19]  Patrick J. Wolfe,et al.  Optimal exposure control for high dynamic range imaging , 2010, 2010 IEEE International Conference on Image Processing.

[20]  Neil Barakat,et al.  Minimal-Bracketing Sets for High-Dynamic-Range Image Capture , 2008, IEEE Transactions on Image Processing.

[21]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[22]  Stephen Mangiat,et al.  High dynamic range video with ghost removal , 2010, Optical Engineering + Applications.

[23]  Jan Kautz,et al.  Exposure Fusion , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[24]  Harald Brendel,et al.  Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays , 2014, Electronic Imaging.

[25]  Shree K. Nayar,et al.  Fibonacci Exposure Bracketing for High Dynamic Range Imaging , 2013, 2013 IEEE International Conference on Computer Vision.

[26]  Tomoo Mitsunaga,et al.  Coded rolling shutter photography: Flexible space-time sampling , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[27]  Robert L. Stevenson,et al.  Estimation-theoretic approach to dynamic range enhancement using multiple exposures , 2003, J. Electronic Imaging.

[28]  Shree K. Nayar,et al.  Adaptive dynamic range imaging: optical control of pixel exposures over space and time , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[29]  Kari Pulli,et al.  FlexISP , 2014, ACM Trans. Graph..

[30]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[31]  Wolfgang Effelsberg,et al.  Algorithms for a real-time HDR video system , 2013, Pattern Recognit. Lett..

[32]  Diego Gutierrez,et al.  Evaluation of reverse tone mapping through varying exposure conditions , 2009, SIGGRAPH 2009.

[33]  George Drettakis,et al.  Perception of Visual Artifacts in Image‐Based Rendering of Façades , 2011, EGSR '11.

[34]  Stefan Gustavson,et al.  Unified HDR reconstruction from raw CFA data , 2013, IEEE International Conference on Computational Photography (ICCP).

[35]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[36]  Roberto Manduchi,et al.  One-Shot Optimal Exposure Control , 2010, ECCV.

[37]  Andrew J. Woods,et al.  Crosstalk in stereoscopic displays: a review , 2012, J. Electronic Imaging.

[38]  Marc Levoy,et al.  CMOS Image Sensors With Multi-Bucket Pixels for Computational Photography , 2012, IEEE Journal of Solid-State Circuits.

[39]  Hans-Peter Seidel,et al.  A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging , 2013, ACM Trans. Graph..

[40]  Shree K. Nayar,et al.  High Dynamic Range from Multiple Images: Which Exposures to Combine?∗ , 2003 .

[41]  Jun Hu,et al.  HDR Deghosting: How to Deal with Saturation? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Ming C. Lin EIC Farewell and New EIC Introduction , 2015, IEEE Trans. Vis. Comput. Graph..

[43]  Andrew B. Watson,et al.  High Frame Rates and Human Vision: A View through the Window of Visibility , 2013 .

[44]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[45]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[46]  Hans-Peter Seidel,et al.  Optimal HDR reconstruction with linear digital cameras , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[47]  Diego Gutierrez,et al.  Evaluation of reverse tone mapping through varying exposure conditions , 2009, ACM Trans. Graph..

[48]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, SIGGRAPH 2011.

[49]  Desire Sidibé,et al.  Ghost detection and removal for high dynamic range images: Recent advances , 2012, Signal Process. Image Commun..