Comparison of Deep Learning and Conventional Demosaicing Algorithms for Mastcam Images

Bayer pattern filters have been used in many commercial digital cameras. In National Aeronautics and Space Administration’s (NASA) mast camera (Mastcam) imaging system, onboard the Mars Science Laboratory (MSL) rover Curiosity, a Bayer pattern filter is being used to capture the RGB (red, green, and blue) color of scenes on Mars. The Mastcam has two cameras: left and right. The right camera has three times better resolution than that of the left. It is well known that demosaicing introduces color and zipper artifacts. Here, we present a comparative study of demosaicing results using conventional and deep learning algorithms. Sixteen left and 15 right Mastcam images were used in our experiments. Due to a lack of ground truth images for Mastcam data from Mars, we compared the various algorithms using a blind image quality assessment model. It was observed that no one algorithm can work the best for all images. In particular, a deep learning-based algorithm worked the best for the right Mastcam images and a conventional algorithm achieved the best results for the left Mastcam images. Moreover, subjective evaluation of five demosaiced Mastcam images was also used to compare the various algorithms.

[1]  Yap-Peng Tan,et al.  Color filter array demosaicking: new method and performance measures , 2003, IEEE Trans. Image Process..

[2]  Radu Timofte,et al.  Demosaicing Based on Directional Difference Regression and Efficient Regression Priors , 2016, IEEE Transactions on Image Processing.

[3]  Masatoshi Okutomi,et al.  Adaptive Residual Interpolation for Color and Multispectral Image Demosaicking † , 2017, Sensors.

[4]  Masatoshi Okutomi,et al.  Beyond Color Difference: Residual Interpolation for Color Image Demosaicking , 2016, IEEE Transactions on Image Processing.

[5]  Chiman Kwan,et al.  Demosaicing enhancement using pixel-level fusion , 2018, Signal Image Video Process..

[6]  Wangmeng Zuo,et al.  COLOR IMAGE DEMOSAICKING VIA DEEP RESIDUAL LEARNING , 2017 .

[7]  Chiman Kwan,et al.  A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[8]  Hairong Qi,et al.  Revisiting the preprocessing procedures for elemental concentration estimation based on chemcam libs on mars rover , 2014, 2014 6th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[9]  Oscar C. Au,et al.  Exploitation of inter-color correlation for color image demosaicking , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[10]  B. Ayhan,et al.  On The Use of a Linear Spectral Unmixing Technique For Concentration Estimation of APXS Spectrum , 2015 .

[11]  Yücel Altunbasak,et al.  Color plane interpolation using alternating projections , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[12]  Chiman Kwan,et al.  Enhancing Mastcam Images for Mars Rover Mission , 2017, ISNN.

[13]  Lei Zhang,et al.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding , 2011, J. Electronic Imaging.

[14]  Michal C Malin,et al.  The Mars Science Laboratory (MSL) Mast cameras and Descent imager: Investigation and instrument descriptions , 2017, Earth and space science.

[15]  Eric Dubois,et al.  Frequency-domain methods for demosaicking of Bayer-sampled color images , 2005, IEEE Signal Processing Letters.

[16]  Brian A. Wandell,et al.  A spatial extension of CIELAB for digital color‐image reproduction , 1997 .

[17]  Mark T. Lemmon,et al.  The Mars Science Laboratory Curiosity rover Mastcam instruments: Preflight and in‐flight calibration, validation, and data archiving , 2017 .

[18]  Hairong Qi,et al.  A Novel and Effective Multivariate Method for Compositional Analysis using Laser Induced Breakdown Spectroscopy , 2014 .

[19]  J. Bednar,et al.  Alpha-trimmed means and their relationship to median filters , 1984 .

[20]  Lei Zhang,et al.  Image demosaicing: a systematic survey , 2008, Electronic Imaging.

[21]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[22]  Chiman Kwan,et al.  Debayering RGBW color filter arrays: A pansharpening approach , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[23]  Chiman Kwan,et al.  Anomaly detection in hyperspectral images through spectral unmixing and low rank decomposition , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[24]  L. Macaire,et al.  Comparison of color demosaicing methods , 2010 .

[25]  Thomas Pock,et al.  Learning joint demosaicing and denoising based on sequential energy minimization , 2016, 2016 IEEE International Conference on Computational Photography (ICCP).

[26]  Frédo Durand,et al.  Deep joint demosaicking and denoising , 2016, ACM Trans. Graph..

[27]  Chiman Kwan,et al.  Mastcam image enhancement using estimated point spread functions , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[28]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[29]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[30]  Chiman Kwan,et al.  Pansharpening of Mastcam images , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[31]  Henrique S. Malvar,et al.  High-quality linear interpolation for demosaicing of Bayer-patterned color images , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[32]  Chiman Kwan,et al.  Further Improvement of Debayering Performance of RGBW Color Filter Arrays Using Deep Learning and Pansharpening Techniques , 2019, J. Imaging.

[33]  Ning Zhang,et al.  Primary-consistent soft-decision color demosaicking for digital cameras (patent pending) , 2004, IEEE Transactions on Image Processing.