High Dynamic Range Spectral Imaging Pipeline For Multispectral Filter Array Cameras

Spectral filter arrays imaging exhibits a strong similarity with color filter arrays. This permits us to embed this technology in practical vision systems with little adaptation of the existing solutions. In this communication, we define an imaging pipeline that permits high dynamic range (HDR)-spectral imaging, which is extended from color filter arrays. We propose an implementation of this pipeline on a prototype sensor and evaluate the quality of our implementation results on real data with objective metrics and visual examples. We demonstrate that we reduce noise, and, in particular we solve the problem of noise generated by the lack of energy balance. Data are provided to the community in an image database for further research.

[1]  Jean-Baptiste Thomas,et al.  Background subtraction with multispectral video sequences , 2014 .

[2]  Gaurav Sharma,et al.  Digital color imaging , 1997, IEEE Trans. Image Process..

[3]  Pierre Gouton,et al.  A Database of Spectral Filter Array Images that Combine Visible and NIR , 2017, CCIW.

[4]  Jason M. Eichenholz,et al.  Ultracompact fully integrated megapixel multispectral imager , 2009, OPTO.

[5]  Jie Jia,et al.  Fourier Spectral Filter Array for Optimal Multispectral Imaging , 2016, IEEE Transactions on Image Processing.

[6]  Young J. Yu,et al.  Vertical silicon nanowire photodetectors: Spectral sensitivity via nanowire radius , 2013, CLEO: 2013.

[7]  Yassine Ruichek,et al.  Energy balance in single exposure multispectral sensors , 2013, 2013 Colour and Visual Computing Symposium (CVCS).

[8]  Yassine Ruichek,et al.  Energy balance in Spectral Filter Array camera design , 2017, Journal of the European Optical Society-Rapid Publications.

[9]  Jean-Baptiste Thomas,et al.  Illuminant estimation from uncalibrated multispectral images , 2015, 2015 Colour and Visual Computing Symposium (CVCS).

[10]  Hans-Peter Seidel,et al.  High dynamic range imaging pipeline: perception-motivated representation of visual content , 2007, Electronic Imaging.

[11]  Dominique Ginhac,et al.  Hardware-based smart camera for recovering high dynamic range video from multiple exposures , 2014 .

[12]  Dani Lischinski,et al.  Gradient Domain High Dynamic Range Compression , 2023 .

[13]  Laurence Meylan,et al.  Digital camera workflow for high dynamic range images using a model of retinal processing , 2008, Electronic Imaging.

[14]  B. Dang,et al.  3D silicon integration , 2008, 2008 58th Electronic Components and Technology Conference.

[15]  Johannes Brauers,et al.  Vision Color Accuracy and Noise Analysis in Multispectral HDR Imaging , 2008 .

[16]  Alan C. Bovik,et al.  No-Reference Quality Assessment of Tone-Mapped HDR Pictures , 2017, IEEE Transactions on Image Processing.

[17]  Dominique Ginhac,et al.  Real-time ghost free HDR video stream generation using weight adaptation based method , 2016, ICDSC.

[18]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[19]  Masatoshi Okutomi,et al.  Simultaneous capturing of RGB and additional band images using hybrid color filter array , 2014, Electronic Imaging.

[20]  W. Jino Hans,et al.  FPGA implementation of bilinear interpolation algorithm for CFA demosaicing , 2013, 2013 International Conference on Communication and Signal Processing.

[21]  W.E. Snyder,et al.  Color image processing pipeline , 2005, IEEE Signal Processing Magazine.

[22]  Christophe Charrier,et al.  Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain , 2012, IEEE Transactions on Image Processing.

[23]  Marcus Larsson,et al.  Spatial and temporal skin blood volume and saturation estimation using a multispectral snapshot imaging camera , 2017, BiOS.

[24]  Jon Yngve Hardeberg,et al.  Illuminant estimation in multispectral imaging. , 2017, Journal of the Optical Society of America. A, Optics, image science, and vision.

[25]  K. Hirakawa,et al.  SINGLE SHOT HIGH DYNAMIC RANGE AND MULTISPECTRAL IMAGING BASED ON PROPERTIES OF COLOR FILTER ARRAYS , 2011 .

[26]  Pierre Gouton,et al.  Multispectral Filter Arrays: Recent Advances and Practical Implementation , 2014, Sensors.

[27]  Hans-Peter Seidel,et al.  Lightness Perception in Tone Reproduction for High Dynamic Range Images , 2005, Comput. Graph. Forum.

[28]  Wen-Chung Kao,et al.  Design considerations of color image processing pipeline for digital cameras , 2006, IEEE Transactions on Consumer Electronics.

[29]  Shree K. Nayar,et al.  Radiometric self calibration , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[30]  Martin Rehak,et al.  Detection of crop properties by means of hyperspectral remote sensing from a micro UAV , 2015 .

[31]  Nam Ik Cho,et al.  Probabilistic motion pixel detection for the reduction of ghost artifacts in high dynamic range images from multiple exposures , 2014, EURASIP J. Image Video Process..

[32]  Robert L. Stevenson,et al.  Dynamic range improvement through multiple exposures , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[33]  Takeo Kanade,et al.  Statistical Calibration of the CCD Imaging Process , 2001, ICCV.

[34]  Dominique Ginhac,et al.  HDR-ARtiSt: an adaptive real-time smart camera for high dynamic range imaging , 2014, Journal of Real-Time Image Processing.

[35]  Jon Y. Hardeberg,et al.  Multispectral Constancy Based on Spectral Adaptation Transform , 2017, SCIA.

[36]  Carolina Blanch,et al.  A tiny VIS-NIR snapshot multispectral camera , 2015, Photonics West - Optoelectronic Materials and Devices.

[37]  Pierre Gouton,et al.  Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition , 2016, Sensors.

[38]  Masatoshi Okutomi,et al.  A Practical One-Shot Multispectral Imaging System Using a Single Image Sensor , 2015, IEEE Transactions on Image Processing.

[39]  Pierre Gouton,et al.  HDR Imaging Pipeline for Spectral Filter Array Cameras , 2017, SCIA.

[40]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[41]  Steve Mann,et al.  ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES , 1995 .

[42]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[43]  Wesley E. Snyder,et al.  Binary Tree-based Generic Demosaicking Algorithm for Multispectral Filter Arrays , 2006, IEEE Transactions on Image Processing.

[44]  Thomas Bashford-Rogers,et al.  Dynamic range compression by differential zone mapping based on psychophysical experiments , 2012, SAP.

[45]  Erik Reinhard,et al.  Photographic tone reproduction for digital images , 2002, ACM Trans. Graph..

[46]  Kurt Debattista,et al.  Advanced High Dynamic Range Imaging: Theory and Practice , 2011 .

[47]  Dingrong Yi,et al.  Fabrication of multispectral imaging technology driven MEMS-based micro-arrayed multichannel optical filter mosaic , 2011, MOEMS-MEMS.

[48]  Dominique Ginhac,et al.  3D-IC: New Perspectives for a Digital Pixel Sensor , 2016, ICDSC.

[49]  Steve Mann,et al.  Realtime HDR (High Dynamic Range) video for eyetap wearable computers, FPGA-based seeing aids, and glasseyes (EyeTaps) , 2012, 2012 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE).

[50]  Sarah E Bohndiek,et al.  Fluorescence hyperspectral imaging (fHSI) using a spectrally resolved detector array , 2017, Journal of biophotonics.