Hyperspectral Image Super Resolution Based on Multiscale Feature Fusion and Aggregation Network With 3-D Convolution

The spectral resolution of hyperspectral images (HSIs) is very high. Nevertheless, their spatial resolution is low due to various hardware limitations. Therefore, it is important to study HSI super resolution to improve their spatial resolution. In this article, for hyperspectral single-image super resolution, we propose a multiscale feature fusion and aggregation network with 3-D convolution (MFFA-3D) by cascading the MFFA-3D block. The MFFA-3D block includes a group multiscale feature fusion part and a multiscale feature aggregation part. In group multiscale feature fusion part, a novel group multiscale feature fusion method is proposed. Group feature fusion module and two-step multiscale module are proposed in multiscale feature aggregation part. In order to prevent spectral distortion, a spectral gradient loss function is proposed and combined with the mean square error loss function to form the final loss function. Since the proposed super-resolution (SR) network is a full 3-D convolutional network, our method can perform direct super-resolution transfer even if the number of the bands of test images is different from that of the training images. The experiments over simulated and real HSIs demonstrate the superiority of the proposed method in terms of qualitative and quantitative evaluation.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Naoto Yokoya,et al.  Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art , 2017, IEEE Geoscience and Remote Sensing Magazine.

[3]  Narendra Ahuja,et al.  Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Yongchao Zhao,et al.  Hyperspectral image super-resolution with spectral–spatial network , 2018, International Journal of Remote Sensing.

[5]  Steven C. H. Hoi,et al.  Deep Learning for Image Super-Resolution: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Shutao Li,et al.  Super-resolution of hyperspectral image via superpixel-based sparse representation , 2018, Neurocomputing.

[7]  Yunsong Li,et al.  Hyperspectral image super-resolution using deep convolutional neural network , 2017, Neurocomputing.

[8]  Ajmal S. Mian,et al.  Bayesian sparse representation for hyperspectral image super resolution , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Lei Zhang,et al.  Single Hyperspectral Image Super-Resolution with Grouped Deep Recursive Residual Network , 2018, 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM).

[10]  Qingxiang Wu,et al.  Image super-resolution using a dilated convolutional neural network , 2018, Neurocomputing.

[11]  Liang Xiao,et al.  A Multi-Scale Wavelet 3D-CNN for Hyperspectral Image Super-Resolution , 2019, Remote. Sens..

[12]  Anthony G. Christodoulou,et al.  Super-resolution hyperspectral imaging with unknown blurring by low-rank and group-sparse modeling , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[13]  Yunsong Li,et al.  Hyperspectral Image Super-Resolution Using Deep Feature Matrix Factorization , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Qian Du,et al.  Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network , 2017, Remote. Sens..

[15]  Liangpei Zhang,et al.  A super-resolution reconstruction algorithm for hyperspectral images , 2012, Signal Process..

[16]  Jing Yu,et al.  Super-resolution mapping via multi-dictionary based sparse representation , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Ajmal S. Mian,et al.  Sparse Spatio-spectral Representation for Hyperspectral Image Super-resolution , 2014, ECCV.

[18]  Xiangtao Zheng,et al.  Hyperspectral Image Superresolution by Transfer Learning , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[19]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Hairong Qi,et al.  Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Qi Xie,et al.  Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Zhi Tang,et al.  CBNet: A Novel Composite Backbone Network Architecture for Object Detection , 2019, AAAI.

[24]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Xiang Zhu,et al.  Automatic Parameter Selection for Denoising Algorithms Using a No-Reference Measure of Image Content , 2010, IEEE Transactions on Image Processing.

[26]  Yen-Wei Chen,et al.  Multi-Level and Multi-Scale Spatial and Spectral Fusion CNN for Hyperspectral Image Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[27]  Xing Zhao,et al.  Spectral–Spatial Classification of Hyperspectral Data Based on Deep Belief Network , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[28]  Shutao Li,et al.  Deep Hyperspectral Image Sharpening , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Peng Liu,et al.  Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing 1 Active Deep Learning for Classification of Hyperspectral Images , 2022 .