A Brief Review of Some Interesting Mars Rover Image Enhancement Projects

The Curiosity rover has landed on Mars since 2012. One of the instruments onboard the rover is a pair of multispectral cameras known as Mastcams, which act as eyes of the rover. In this paper, we summarize our recent studies on some interesting image processing projects for Mastcams. In particular, we will address perceptually lossless compression of Mastcam images, debayering and resolution enhancement of Mastcam images, high resolution stereo and disparity map generation using fused Mastcam images, and improved performance of anomaly detection and pixel clustering using combined left and right Mastcam images. The main goal of this review paper is to raise public awareness about these interesting Mastcam projects and also stimulate interests in the research community to further develop new algorithms for those applications.

[1]  Xiaoli Yu,et al.  Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution , 1990, IEEE Trans. Acoust. Speech Signal Process..

[2]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[3]  Chiman Kwan,et al.  Further Improvement of Debayering Performance of RGBW Color Filter Arrays Using Deep Learning and Pansharpening Techniques , 2019, J. Imaging.

[4]  Peter Wonka,et al.  High Quality Monocular Depth Estimation via Transfer Learning , 2018, ArXiv.

[5]  Masatoshi Okutomi,et al.  Beyond Color Difference: Residual Interpolation for Color Image Demosaicking , 2016, IEEE Transactions on Image Processing.

[6]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[7]  Eric Dubois,et al.  Frequency-domain methods for demosaicking of Bayer-sampled color images , 2005, IEEE Signal Processing Letters.

[8]  Chiman Kwan,et al.  Demosaicing enhancement using pixel-level fusion , 2018, Signal Image Video Process..

[9]  Mark T. Lemmon,et al.  The Mars Science Laboratory Curiosity rover Mastcam instruments: Preflight and in‐flight calibration, validation, and data archiving , 2017 .

[10]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Trac D. Tran,et al.  Lapped transform via time-domain pre- and post-filtering , 2003, IEEE Trans. Signal Process..

[12]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[13]  Chiman Kwan,et al.  Comparison of Deep Learning and Conventional Demosaicing Algorithms for Mastcam Images , 2019 .

[14]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Masatoshi Okutomi,et al.  Adaptive Residual Interpolation for Color and Multispectral Image Demosaicking † , 2017, Sensors.

[16]  Yap-Peng Tan,et al.  Color filter array demosaicking: new method and performance measures , 2003, IEEE Trans. Image Process..

[17]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Andrea Garzelli,et al.  Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[19]  Chiman Kwan,et al.  Perceptually Lossless Compression for Mastcam Multispectral Images: A Comparative Study , 2019 .

[20]  Lei Zhang,et al.  Color demosaicking by local directional interpolation and nonlocal adaptive thresholding , 2011, J. Electronic Imaging.

[21]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[22]  Hairong Qi,et al.  Application of Deep Learning Approaches for Enhancing Mastcam Images , 2020, Recent Advances in Image Restoration with Applications to Real World Problems.

[23]  J. Bednar,et al.  Alpha-trimmed means and their relationship to median filters , 1984 .

[24]  Radu Timofte,et al.  Demosaicing Based on Directional Difference Regression and Efficient Regression Priors , 2016, IEEE Transactions on Image Processing.

[25]  Yücel Altunbasak,et al.  Color plane interpolation using alternating projections , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[26]  Ning Zhang,et al.  Primary-consistent soft-decision color demosaicking for digital cameras (patent pending) , 2004, IEEE Transactions on Image Processing.

[27]  Ning Xu,et al.  Wide Activation for Efficient and Accurate Image Super-Resolution , 2018, ArXiv.

[28]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[29]  Shuo Li,et al.  A Parameterization of Deformation Fields for Diffeomorphic Image Registration and Its Application to Myocardial Delineation , 2010, MICCAI.

[30]  James E. Fowler,et al.  Nearest Regularized Subspace for Hyperspectral Classification , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Kiyun Yu,et al.  A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[32]  L. Macaire,et al.  Comparison of color demosaicing methods , 2010 .

[33]  Paolo Maggiore,et al.  Analysis of a Moon outpost for Mars enabling technologies through a Virtual Reality environment , 2018 .

[34]  Hairong Qi,et al.  Unsupervised and Unregistered Hyperspectral Image Super-Resolution With Mutual Dirichlet-Net , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[35]  Frédo Durand,et al.  Deep joint demosaicking and denoising , 2016, ACM Trans. Graph..

[36]  Lei Zhang,et al.  Color demosaicking via directional linear minimum mean square-error estimation , 2005, IEEE Transactions on Image Processing.

[37]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[38]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[39]  Chiman Kwan,et al.  Mastcam Image Resolution Enhancement with Application to Disparity Map Generation for Stereo Images with Different Resolutions , 2019, Sensors.

[40]  Chiman Kwan,et al.  A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.