Demosaicing Mastcam Images using A New Color Filter Array

The two mast cameras (Mastcam) act as eyes of the NASA’s Mars rover Curiosity. They can work independently or together for near and long range (up to 1 km) rover guidance and rock sample selection. Currently, the Mastcams are using Bayer color filter array (CFA), also known as CFA 1.0, in generating the RGB images. Under normal lighting conditions, CFA 1.0 is sufficient. However, since Mastcam may need to collect images under different lighting conditions such as early morning and sunset hours or sand storm periods, the lighting conditions in those scenarios will be unfavorable. It will be good to investigate a CFA that is robust to various lighting conditions. In the past, we have compared CFA 1.0 and CFA 2.0 for normal and low lighting images. Recently, a new CFA known as CFA 3.0 has been proposed by our team. CFA 3.0 has 75% white pixels, which are believed to be able to enhance the sensitivity of cameras. In this paper, we will first review some past demosaicing results for Mastcams. We will then investigate the performance of CFA 3.0 for Mastcam images in normal lighting conditions. Experiments using actual Mastcam images show that the demosaicing image quality using CFA 3.0 is satisfactory based on objective and subjective evaluations.

[1]  Chiman Kwan,et al.  Resolution enhancement for hyperspectral images: A super-resolution and fusion approach , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Aleksandra Pizurica,et al.  Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[3]  Chiman Kwan,et al.  Demosaicing of CFA 3.0 with Applications to Low Lighting Images , 2020, Sensors.

[4]  Chiman Kwan,et al.  Enhancing Stereo Image Formation and Depth Map Estimation for Mastcam Images , 2018, 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON).

[5]  Chiman Kwan,et al.  Comparison of Deep Learning and Conventional Demosaicing Algorithms for Mastcam Images , 2019 .

[6]  Chiman Kwan,et al.  A comparative study of conventional and deep learning approaches for demosaicing Mastcam images , 2019, Defense + Commercial Sensing.

[7]  Chiman Kwan,et al.  A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[8]  Chao Zhang,et al.  Universal Demosaicking of Color Filter Arrays , 2016, IEEE Transactions on Image Processing.

[9]  Brian A. Wandell,et al.  A spatial extension of CIELAB for digital color‐image reproduction , 1997 .

[10]  Hairong Qi,et al.  Revisiting the preprocessing procedures for elemental concentration estimation based on chemcam libs on mars rover , 2014, 2014 6th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[11]  Giancarlo Calvagno,et al.  Regularization Approaches to Demosaicking , 2009, IEEE Transactions on Image Processing.

[12]  Yap-Peng Tan,et al.  Color filter array demosaicking: new method and performance measures , 2003, IEEE Trans. Image Process..

[13]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Feng Gao,et al.  A Hybrid Color Mapping Approach to Fusing MODIS and Landsat Images for Forward Prediction , 2018, Remote. Sens..

[15]  Yuzhong Shen,et al.  Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images , 2018, Sensors.

[16]  W. Marsden.  I and J , 2012 .

[17]  Frédo Durand,et al.  Deep joint demosaicking and denoising , 2016, ACM Trans. Graph..

[18]  Chiman Kwan,et al.  Mastcam Image Resolution Enhancement with Application to Disparity Map Generation for Stereo Images with Different Resolutions , 2019, Sensors.

[19]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[20]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[21]  Lei Zhang,et al.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding , 2011, J. Electronic Imaging.

[22]  Yücel Altunbasak,et al.  Color plane interpolation using alternating projections , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[23]  Ning Zhang,et al.  Primary-consistent soft-decision color demosaicking for digital cameras (patent pending) , 2004, IEEE Transactions on Image Processing.

[24]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[25]  トーマス コンプトン，ジョン,et al.  Processing of color and panchromatic pixels , 2006 .

[26]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[27]  J. Bednar,et al.  Alpha-trimmed means and their relationship to median filters , 1984 .

[28]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[29]  Chiman Kwan,et al.  Fusion of themis and TES for accurate Mars surface characterization , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[30]  Chiman Kwan,et al.  Hyperspectral image super-resolution: a hybrid color mapping approach , 2016 .

[31]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[32]  B. Ayhan,et al.  On The Use of a Linear Spectral Unmixing Technique For Concentration Estimation of APXS Spectrum , 2015 .

[33]  Jocelyn Chanussot,et al.  Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening , 2014, IEEE Geoscience and Remote Sensing Letters.

[34]  Chiman Kwan,et al.  Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images , 2019, Electronics.

[35]  황준.  Image sensor with improved light sensitivity and fabricating method of the same , 2001 .

[36]  Chiman Kwan,et al.  Demosaicing enhancement using pixel-level fusion , 2018, Signal Image Video Process..

[37]  Chiman Kwan,et al.  Compression Algorithm Selection for Multispectral Mastcam Images , 2019 .

[38]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[39]  J. Astola,et al.  ON BETWEEN-COEFFICIENT CONTRAST MASKING OF DCT BASIS FUNCTIONS , 2007 .

[40]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[41]  Laurent Condat,et al.  A generic variational approach for demosaicking from an arbitrary color filter array , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[42]  Chiman Kwan,et al.  Further Improvement of Debayering Performance of RGBW Color Filter Arrays Using Deep Learning and Pansharpening Techniques , 2019, J. Imaging.

[43]  Eric Dubois,et al.  Frequency-domain methods for demosaicking of Bayer-sampled color images , 2005, IEEE Signal Processing Letters.