Fusion of hyperspectral and multispectral image by dual residual dense networks

Abstract. The spatial resolution of hyperspectral image (HSI) is severely limited as a cost for higher spectral resolution. We propose a deep learning-based HSI super resolution method named dual residual dense networks (DRDNs) to overcome the resolution limitation by fusing low-resolution (LR) HSI and high-resolution (HR) multispectral image (MSI). The proposed model uses two symmetric subnets based on residual dense block to fully exploit deep features from HR-MSI and LR-HSI, respectively. Then the features are fused through a feature fusion subnet to generate the final reconstructed HR-HSI. Our DRDNs have realized end-to-end mapping from the original input of LR-HSI and HR-MSI to the desired HR-HSI directly. All the parameters are trained through a unified framework. Some experiments are also carried out to evaluate the performance of our proposed method. The results show that our proposed method gains significant improvement in simultaneously enhancing spatial resolution and preserving spectral consistency when compared to other HSI/MSI fusion methods proposed recently.

[1]  Guangming Shi,et al.  Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation. , 2016, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[2]  Xiao Xiang Zhu,et al.  Unsupervised Spectral–Spatial Feature Learning via Deep Residual Conv–Deconv Network for Hyperspectral Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Jonathan Cheung-Wai Chan,et al.  Hyperspectral Imagery Super-Resolution by Spatial–Spectral Joint Nonlocal Similarity , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[4]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[5]  Mingyi He,et al.  Multi-spectral and hyperspectral image fusion using 3-D wavelet transform , 2007 .

[6]  Lei Wang,et al.  Image Superresolution Using Densely Connected Residual Networks , 2018, IEEE Signal Processing Letters.

[7]  Yunsong Li,et al.  Hyperspectral Image Super-Resolution by Spectral Difference Learning and Spatial Error Correction , 2017, IEEE Geoscience and Remote Sensing Letters.

[8]  Jie Li,et al.  Hyperspectral Image Super-Resolution by Spectral Mixture Analysis and Spatial–Spectral Group Sparsity , 2016, IEEE Geoscience and Remote Sensing Letters.

[9]  Menas Kafatos,et al.  Wavelet-based hyperspectral and multispectral image fusion , 2001, SPIE Defense + Commercial Sensing.

[10]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[11]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Jean-Yves Tourneret,et al.  Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Michael E. Schaepman,et al.  Unmixing-Based Landsat TM and MERIS FR Data Fusion , 2008, IEEE Geoscience and Remote Sensing Letters.

[15]  Lucien Wald,et al.  Quality of high resolution synthesised images: Is there a simple criterion ? , 2000 .

[16]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[18]  Qian Du,et al.  Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network , 2017, Remote. Sens..

[19]  Shutao Li,et al.  Hyperspectral Image Super-Resolution via Non-local Sparse Tensor Factorization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Silvio Savarese,et al.  Pose Estimation Errors, the Ultimate Diagnosis , 2016, ECCV.

[21]  Shutao Li,et al.  Deep Hyperspectral Image Sharpening , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[22]  J. Chanussot,et al.  Hyperspectral Remote Sensing Data Analysis and Future Challenges , 2013, IEEE Geoscience and Remote Sensing Magazine.

[23]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[24]  Ayan Chakrabarti,et al.  Statistics of real-world hyperspectral images , 2011, CVPR 2011.

[25]  Hongyi Liu,et al.  Spatial-spectral compressive sensing for hyperspectral images super-resolution over learned dictionary , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[26]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[27]  Xiangtao Zheng,et al.  Hyperspectral Image Superresolution by Transfer Learning , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[28]  Konrad Schindler,et al.  Hyperspectral Super-Resolution by Coupled Spectral Unmixing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[29]  Johannes R. Sveinsson,et al.  Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[30]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.