Demosaicing of CFA 3.0 with Applications to Low Lighting Images

Low lighting images usually contain Poisson noise, which is pixel amplitude-dependent. More panchromatic or white pixels in a color filter array (CFA) are believed to help the demosaicing performance in dark environments. In this paper, we first introduce a CFA pattern known as CFA 3.0 that has 75% white pixels, 12.5% green pixels, and 6.25% of red and blue pixels. We then present algorithms to demosaic this CFA, and demonstrate its performance for normal and low lighting images. In addition, a comparative study was performed to evaluate the demosaicing performance of three CFAs, namely the Bayer pattern (CFA 1.0), the Kodak CFA 2.0, and the proposed CFA 3.0. Using a clean Kodak dataset with 12 images, we emulated low lighting conditions by introducing Poisson noise into the clean images. In our experiments, normal and low lighting images were used. For the low lighting conditions, images with signal-to-noise (SNR) of 10 dBs and 20 dBs were studied. We observed that the demosaicing performance in low lighting conditions was improved when there are more white pixels. Moreover, denoising can further enhance the demosaicing performance for all CFAs. The most important finding is that CFA 3.0 performs better than CFA 1.0, but is slightly inferior to CFA 2.0, in low lighting images.

[1]  トーマス コンプトン,ジョン,et al.  Processing of color and panchromatic pixels , 2006 .

[2]  Chiman Kwan,et al.  Demosaicing enhancement using pixel-level fusion , 2018, Signal Image Video Process..

[3]  J. Astola,et al.  ON BETWEEN-COEFFICIENT CONTRAST MASKING OF DCT BASIS FUNCTIONS , 2007 .

[4]  Chiman Kwan,et al.  Further Improvement of Debayering Performance of RGBW Color Filter Arrays Using Deep Learning and Pansharpening Techniques , 2019, J. Imaging.

[5]  Feng Gao,et al.  A Hybrid Color Mapping Approach to Fusing MODIS and Landsat Images for Forward Prediction , 2018, Remote. Sens..

[6]  Ning Zhang,et al.  Primary-consistent soft-decision color demosaicking for digital cameras (patent pending) , 2004, IEEE Transactions on Image Processing.

[7]  Masatoshi Okutomi,et al.  Pseudo four-channel image denoising for noisy CFA raw data , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[8]  Jun Ohta,et al.  Smart CMOS Image Sensors and Applications , 2007 .

[9]  Radu Timofte,et al.  Demosaicing Based on Directional Difference Regression and Efficient Regression Priors , 2016, IEEE Transactions on Image Processing.

[10]  Andrew F. Siegel,et al.  Practical Business Statistics , 1994 .

[11]  Wangmeng Zuo,et al.  COLOR IMAGE DEMOSAICKING VIA DEEP RESIDUAL LEARNING , 2017 .

[12]  Masatoshi Okutomi,et al.  Beyond Color Difference: Residual Interpolation for Color Image Demosaicking , 2016, IEEE Transactions on Image Processing.

[13]  Jocelyn Chanussot,et al.  Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening , 2014, IEEE Geoscience and Remote Sensing Letters.

[14]  Masatoshi Okutomi,et al.  Adaptive Residual Interpolation for Color and Multispectral Image Demosaicking † , 2017, Sensors.

[15]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Oscar C. Au,et al.  Exploitation of inter-color correlation for color image demosaicking , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[17]  Chiman Kwan,et al.  Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images , 2019, Electronics.

[18]  Brian A. Wandell,et al.  A spatial extension of CIELAB for digital color‐image reproduction , 1997 .

[19]  Chiman Kwan,et al.  Debayering RGBW color filter arrays: A pansharpening approach , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[20]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[21]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Giancarlo Calvagno,et al.  Regularization Approaches to Demosaicking , 2009, IEEE Transactions on Image Processing.

[23]  Chiman Kwan,et al.  Resolution enhancement for hyperspectral images: A super-resolution and fusion approach , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[25]  Chao Zhang,et al.  Universal Demosaicking of Color Filter Arrays , 2016, IEEE Transactions on Image Processing.

[26]  Yap-Peng Tan,et al.  Color filter array demosaicking: new method and performance measures , 2003, IEEE Trans. Image Process..

[27]  Gwanggil Jeon,et al.  Least-Squares Luma–Chroma Demultiplexing Algorithm for Bayer Demosaicking , 2011, IEEE Transactions on Image Processing.

[28]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[29]  Chiman Kwan,et al.  Fusion of themis and TES for accurate Mars surface characterization , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[30]  Thomas Pock,et al.  Learning joint demosaicing and denoising based on sequential energy minimization , 2016, 2016 IEEE International Conference on Computational Photography (ICCP).

[31]  Chiman Kwan,et al.  Hyperspectral image super-resolution: a hybrid color mapping approach , 2016 .

[32]  Chiman Kwan,et al.  Pansharpening of Mastcam images , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[33]  Mark T. Lemmon,et al.  The Mars Science Laboratory Curiosity rover Mastcam instruments: Preflight and in‐flight calibration, validation, and data archiving , 2017 .

[34]  Frédo Durand,et al.  Deep joint demosaicking and denoising , 2016, ACM Trans. Graph..

[35]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[36]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Guizhong Liu,et al.  A high performance approach to local active noise reduction , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[38]  Eric Dubois,et al.  Frequency-domain methods for demosaicking of Bayer-sampled color images , 2005, IEEE Signal Processing Letters.

[39]  Suk Ho Lee,et al.  Colorization-Based RGB-White Color Interpolation using Color Filter Array with Randomly Sampled Pattern , 2017, Sensors.

[40]  Chiman Kwan,et al.  A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[41]  황준 Image sensor with improved light sensitivity and fabricating method of the same , 2001 .

[42]  Yuzhong Shen,et al.  Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images , 2018, Sensors.

[43]  M. Ioannides,et al.  Digital Heritage , 2010, Lecture Notes in Computer Science.

[44]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[45]  Laurent Condat,et al.  A generic variational approach for demosaicking from an arbitrary color filter array , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[46]  Aleksandra Pizurica,et al.  Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[47]  Chiman Kwan,et al.  Enhancing Mastcam Images for Mars Rover Mission , 2017, ISNN.

[48]  Yücel Altunbasak,et al.  Color plane interpolation using alternating projections , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[49]  J. Bednar,et al.  Alpha-trimmed means and their relationship to median filters , 1984 .

[50]  Lei Zhang,et al.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding , 2011, J. Electronic Imaging.