A Super-Resolution and Fusion Approach to Enhancing Hyperspectral Images

High resolution (HR) hyperspectral (HS) images have found widespread applications in terrestrial remote sensing applications, including vegetation monitoring, military surveillance and reconnaissance, fire damage assessment, and many others. They also find applications in planetary missions such as Mars surface characterization. However, resolutions of most HS imagers are limited to tens of meters. Existing resolution enhancement techniques either require additional multispectral (MS) band images or use a panchromatic (pan) band image. The former poses hardware challenges, whereas the latter may have limited performance. In this paper, we present a new resolution enhancement algorithm for HS images that only requires an HR color image and a low resolution (LR) HS image cube. Our approach integrates two newly developed techniques: (1) A hybrid color mapping (HCM) algorithm, and (2) A Plug-and-Play algorithm for single image super-resolution. Comprehensive experiments (objective (five performance metrics), subjective (synthesized fused images in multiple spectral ranges), and pixel clustering) using real HS images and comparative studies with 20 representative algorithms in the literature were conducted to validate and evaluate the proposed method. Results demonstrated that the new algorithm is very promising.

[1]  Chiman Kwan,et al.  New sparsity based pansharpening algorithms for hyperspectral images , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[2]  J. Hook Simon,et al.  Mineralogic mapping using Airborne Visible Infrared Imaging Spectrometer (AVIRIS) shortwave infrared (SWIR) data acquired over Cuprite, Nevada , 1990 .

[3]  Charles A. Bouman,et al.  Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation , 2015, IEEE Transactions on Computational Imaging.

[4]  Russell C. Hardie,et al.  MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor , 2004, IEEE Transactions on Image Processing.

[5]  D. A. Fish,et al.  Blind deconvolution by means of the Richardson-Lucy algorithm. , 1995 .

[6]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[7]  Chiman Kwan,et al.  Mastcam image enhancement using estimated point spread functions , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[8]  Chiman Kwan,et al.  Blind Quality Assessment of Fused WorldView-3 Images by Using the Combinations of Pansharpening and Hypersharpening Paradigms , 2017, IEEE Geoscience and Remote Sensing Letters.

[9]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[10]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[11]  Jocelyn Chanussot,et al.  Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening , 2014, IEEE Geoscience and Remote Sensing Letters.

[12]  Chiman Kwan,et al.  Resolution enhancement for hyperspectral images: A super-resolution and fusion approach , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Feng Gao,et al.  A Hybrid Color Mapping Approach to Fusing MODIS and Landsat Images for Forward Prediction , 2018, Remote. Sens..

[14]  Jean-Yves Tourneret,et al.  Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Truong Q. Nguyen,et al.  Single Image Superresolution Based on Gradient Profile Sharpness , 2015, IEEE Transactions on Image Processing.

[16]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[17]  J. G. Liu,et al.  Smoothing Filter-based Intensity Modulation : a spectral preserve image fusion technique for improving spatial details , 2001 .

[18]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[19]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[20]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[21]  Luciano Alparone,et al.  MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery , 2006 .

[22]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[23]  Jocelyn Chanussot,et al.  A Convex Formulation for Hyperspectral Image Superresolution via Subspace-Based Regularization , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Robert A. Schowengerdt,et al.  Remote sensing, models, and methods for image processing , 1997 .

[25]  Jocelyn Chanussot,et al.  Hyperspectral image superresolution: An edge-preserving convex formulation , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[26]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[27]  R. Green,et al.  An introduction to the NASA Hyperspectral InfraRed Imager (HyspIRI) mission and preparatory activities , 2015 .

[28]  Luis Grave de Peralta,et al.  Super-resolution imaging of photonic crystals using the dual-space microscopy technique. , 2016, Applied optics.

[29]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Stanley H. Chan,et al.  Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications , 2016, IEEE Transactions on Computational Imaging.

[31]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[32]  Chiman Kwan,et al.  On the generation of high-spatial and high-spectral resolution images using THEMIS and TES for Mars exploration , 2018, Defense + Security.

[33]  Chiman Kwan,et al.  Fusion of themis and TES for accurate Mars surface characterization , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[34]  Chiman Kwan,et al.  Hyperspectral image super-resolution: a hybrid color mapping approach , 2016 .

[35]  Chiman Kwan,et al.  Pansharpening of Mastcam images , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[36]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[38]  Chiman Kwan,et al.  A Novel Cluster Kernel RX Algorithm for Anomaly and Change Detection Using Hyperspectral Images , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Peyman Milanfar,et al.  RAISR: Rapid and Accurate Image Super Resolution , 2016, IEEE Transactions on Computational Imaging.

[40]  Chiman Kwan,et al.  Debayering RGBW color filter arrays: A pansharpening approach , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[41]  Yuzhong Shen,et al.  Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images , 2018, Sensors.

[42]  Aleksandra Pizurica,et al.  Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[43]  Chiman Kwan,et al.  Enhancing Mastcam Images for Mars Rover Mission , 2017, ISNN.