Mastcam Image Resolution Enhancement with Application to Disparity Map Generation for Stereo Images with Different Resolutions

In this paper, we introduce an in-depth application of high-resolution disparity map estimation using stereo images from Mars Curiosity rover’s Mastcams, which have two imagers with different resolutions. The left Mastcam has three times lower resolution as that of the right. The left Mastcam image’s resolution is first enhanced with three methods: Bicubic interpolation, pansharpening-based method, and a deep learning super resolution method. The enhanced left camera image and the right camera image are then used to estimate the disparity map. The impact of the left camera image enhancement is examined. The comparative performance analyses showed that the left camera enhancement results in getting more accurate disparity maps in comparison to using the original left Mastcam images for disparity map estimation. The deep learning-based method provided the best performance among the three for both image enhancement and disparity map estimation accuracy. A high-resolution disparity map, which is the result of the left camera image enhancement, is anticipated to improve the conducted science products in the Mastcam imagery such as 3D scene reconstructions, depth maps, and anaglyph images.

[1]  Jocelyn Chanussot,et al.  A Critical Comparison Among Pansharpening Algorithms , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Chiman Kwan,et al.  Stereo Image and Depth Map Generation for Images with Different Views and Resolutions , 2018, 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON).

[3]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[4]  Stefano Mattoccia,et al.  Learning Monocular Depth Estimation with Unsupervised Trinocular Assumptions , 2018, 2018 International Conference on 3D Vision (3DV).

[5]  Saad Hassan,et al.  Exploration of Mars Using Augmented Reality , 2018 .

[6]  Ning Xu,et al.  Wide Activation for Efficient and Accurate Image Super-Resolution , 2018, ArXiv.

[7]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[8]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[9]  Chiman Kwan,et al.  Enhancing Stereo Image Formation and Depth Map Estimation for Mastcam Images , 2018, 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON).

[10]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[11]  Chiman Kwan,et al.  A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[12]  Chiman Kwan,et al.  Pansharpening of Mastcam images , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[13]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[14]  Mark T. Lemmon,et al.  The Mars Science Laboratory Curiosity rover Mastcam instruments: Preflight and in‐flight calibration, validation, and data archiving , 2017 .

[15]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[16]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[17]  Nikolay N. Ponomarenko,et al.  A NEW FULL-REFERENCE QUALITY METRICS BASED ON HVS , 2006 .

[18]  Paolo Maggiore,et al.  Analysis of a Moon outpost for Mars enabling technologies through a Virtual Reality environment , 2018 .

[19]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[20]  Georgios Ch. Sirakoulis,et al.  Real-time disparity map computation module , 2008, Microprocess. Microsystems.

[21]  GasteratosAntonios,et al.  Real-time disparity map computation module , 2008 .

[22]  Tony Greicius.  Mars Virtual Reality Software Wins NASA Award , 2018 .

[23]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  J. Astola,et al.  ON BETWEEN-COEFFICIENT CONTRAST MASKING OF DCT BASIS FUNCTIONS , 2007 .

[25]  Hairong Qi,et al.  DOES multispectral / hyperspectral pansharpening improve the performance of anomaly detection? , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[26]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[27]  Mei-Juan Chen,et al.  A fast edge-oriented algorithm for image interpolation , 2005, Image Vis. Comput..

[28]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Peter Wonka,et al.  High Quality Monocular Depth Estimation via Transfer Learning , 2018, ArXiv.

[30]  Oisin Mac Aodha,et al.  Unsupervised Monocular Depth Estimation with Left-Right Consistency , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[32]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[33]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .