Convolutional Sparse Coding for High Dynamic Range Imaging

Current HDR acquisition techniques are based on either (i) fusing multibracketed, low dynamic range (LDR) images, (ii) modifying existing hardware and capturing different exposures simultaneously with multiple sensors, or (iii) reconstructing a single image with spatially‐varying pixel exposures. In this paper, we propose a novel algorithm to recover high‐quality HDRI images from a single, coded exposure. The proposed reconstruction method builds on recently‐introduced ideas of convolutional sparse coding (CSC); this paper demonstrates how to make CSC practical for HDR imaging. We demonstrate that the proposed algorithm achieves higher‐quality reconstructions than alternative methods, we evaluate optical coding schemes, analyze algorithmic parameters, and build a prototype coded HDR camera that demonstrates the utility of convolutional sparse HDRI coding with a custom hardware platform.

[1]  Hans-Peter Seidel,et al.  A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging , 2013, ACM Trans. Graph..

[2]  André Kaup,et al.  Evaluation of a high dynamic range video camera with non-regular sensor , 2013, Electronic Imaging.

[3]  Gordon Wetzstein,et al.  Sensor saturation in Fourier multiplexed imaging , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  David B. Dunson,et al.  Deep Learning with Hierarchical Convolutional Factor Analysis , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Rafal Mantiuk,et al.  Display adaptive tone mapping , 2008, ACM Trans. Graph..

[6]  Yonina C. Eldar,et al.  Compressed Sensing with Coherent and Redundant Dictionaries , 2010, ArXiv.

[7]  Julie Delon,et al.  Single shot high dynamic range imaging using piecewise linear estimators , 2014, 2014 IEEE International Conference on Computational Photography (ICCP).

[8]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, ACM Trans. Graph..

[9]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Shree K. Nayar,et al.  Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Shree K. Nayar,et al.  High dynamic range imaging: spatially varying pixel exposures , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[12]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[13]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[14]  Kari Pulli,et al.  FlexISP , 2014, ACM Trans. Graph..

[15]  Miguel Granados,et al.  Automatic noise modeling for ghost-free HDR reconstruction , 2013, ACM Trans. Graph..

[16]  广野亮 Image processing apparatus, image processing method, and program , 2011 .

[17]  Jürgen Seiler,et al.  Building a high dynamic range video sensor with spatially nonregular optical filtering , 2012, Other Conferences.

[18]  Joachim Weickert,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement Freehand Hdr Imaging of Moving Scenes with Simultaneous Resolution Enhancement , 2022 .

[19]  Pieter Peers,et al.  Compressive light transport sensing , 2009, ACM Trans. Graph..

[20]  André Kaup,et al.  High dynamic range video reconstruction from a stereo camera setup , 2014, Signal Process. Image Commun..

[21]  Steve Marschner,et al.  Dual photography , 2005, ACM Trans. Graph..

[22]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[23]  Eli Shechtman,et al.  Robust patch-based hdr reconstruction of dynamic scenes , 2012, ACM Trans. Graph..

[24]  Anders P. Eriksson,et al.  Fast Convolutional Sparse Coding , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Clemens Birklbauer,et al.  Coded exposure HDR light-field video recording , 2013, SIGGRAPH Talks.

[26]  Marc Levoy,et al.  Dual photography , 2005, SIGGRAPH 2005.

[27]  Ramesh Raskar,et al.  Unbounded High Dynamic Range Photography Using a Modulo Camera , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[28]  Simon Lucey,et al.  Optimization Methods for Convolutional Sparse Coding , 2014, ArXiv.

[29]  Abbas El Gamal,et al.  High dynamic range image sensors , 2002 .

[30]  Pradeep Sen,et al.  A versatile HDR video production system , 2011, SIGGRAPH 2011.

[31]  Desire Sidibé,et al.  Ghost detection and removal for high dynamic range images: Recent advances , 2012, Signal Process. Image Commun..

[32]  Y-Lan Boureau,et al.  Learning Convolutional Feature Hierarchies for Visual Recognition , 2010, NIPS.

[33]  Jerry D. Gibson,et al.  Spatially adaptive filtering for registration artifact removal in HDR video , 2011, 2011 18th IEEE International Conference on Image Processing.

[34]  Graham W. Taylor,et al.  Adaptive deconvolutional networks for mid and high level feature learning , 2011, 2011 International Conference on Computer Vision.

[35]  Seungyong Lee,et al.  Single‐shot High Dynamic Range Imaging Using Coded Electronic Shutter , 2014, Comput. Graph. Forum.

[36]  Hans-Peter Seidel,et al.  Interactive by-example design of artistic packing layouts , 2013, ACM Trans. Graph..

[37]  Qionghai Dai,et al.  Supplementary Document : Spatial-spectral Encoded Compressive Hyperspectral Imaging , 2014 .

[38]  Niloy J. Mitra,et al.  Coupled structure-from-motion and 3D symmetry detection for urban facades , 2014, ACM Trans. Graph..

[39]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[40]  Stefan Gustavson,et al.  Unified HDR reconstruction from raw CFA data , 2013, IEEE International Conference on Computational Photography (ICCP).

[41]  Yann LeCun,et al.  Convolutional Matching Pursuit and Dictionary Training , 2010, ArXiv.

[42]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[43]  Soheil Darabi,et al.  A novel framework for imaging using compressed sensing , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[44]  Qionghai Dai,et al.  Robust and accurate transient light transport decomposition via convolutional sparse coding. , 2014, Optics letters.

[45]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[46]  Soheil Darabi,et al.  Compressive Rendering: A Rendering Application of Compressed Sensing , 2011, IEEE Transactions on Visualization and Computer Graphics.

[47]  Clemens Birklbauer,et al.  Coded exposure HDR light‐field video recording , 2014, Comput. Graph. Forum.

[48]  Robert Bridson,et al.  Synthesizing waves from animated height fields , 2013, ACM Trans. Graph..

[49]  Jonas Unger,et al.  HDR Reconstruction for Alternating Gain (ISO) Sensor Readout , 2014, Eurographics.

[50]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[51]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[52]  Hans-Peter Seidel,et al.  Perception-motivated high dynamic range video encoding , 2004, SIGGRAPH 2004.

[53]  Pat Hanrahan,et al.  A realistic camera model for computer graphics , 1995, SIGGRAPH.

[54]  Stéphane Mallat,et al.  Solving Inverse Problems With Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity , 2010, IEEE Transactions on Image Processing.

[55]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[56]  Jonas Unger,et al.  Compressive Image Reconstruction in Reduced Union of Subspaces , 2015, Comput. Graph. Forum.

[57]  Katsushi Ikeuchi,et al.  Radiometric Calibration by Rank Minimization , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Wolfgang Heidrich,et al.  Imaging in scattering media using correlation image sensors and sparse convolutional coding. , 2014, Optics express.

[59]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[60]  Tomoo Mitsunaga,et al.  Coded rolling shutter photography: Flexible space-time sampling , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[61]  Li Zhang,et al.  Random coded sampling for high-speed HDR video , 2013, IEEE International Conference on Computational Photography (ICCP).

[62]  Gordon Wetzstein,et al.  Fast and flexible convolutional sparse coding , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[64]  Min H. Kim,et al.  Multisampling Compressive Video Spectroscopy , 2016, Comput. Graph. Forum.

[65]  Marius Tico,et al.  Artifact-free High Dynamic Range imaging , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[66]  Diego Gutierrez,et al.  Compressive High Speed Video Acquisition , 2015, CEIG.

[67]  Gordon Wetzstein,et al.  A survey on computational displays: Pushing the boundaries of optics, computation, and perception , 2013, Comput. Graph..

[68]  Soheil Darabi,et al.  Compressive Dual Photography , 2009, Comput. Graph. Forum.

[69]  Shree K. Nayar,et al.  Assorted pixels: multi-sampled imaging with structural models , 2005, SIGGRAPH Courses.

[70]  Eli Shechtman,et al.  Patch-based high dynamic range video , 2013, ACM Trans. Graph..

[71]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.