Hyperspectral Super-Resolution Reconstruction Network Based on Hybrid Convolution and Spectral Symmetry Preservation

Hyperspectral images (HSI) have high-dimensional and complex spectral characteristics, with dozens or even hundreds of bands covering the same area of pixels. The rich information of the ground objects makes hyperspectral images widely used in satellite remote sensing. Due to the limitations of remote sensing satellite sensors, hyperspectral images suffer from insufficient spatial resolution. Therefore, utilizing software algorithms to improve the spatial resolution of hyperspectral images has become an urgent problem that needs to be solved. The spatial information and spectral information of hyperspectral images are strongly correlated. If only the spatial resolution is improved, it often damages the spectral information. Inspired by the high correlation between spectral information in adjacent spectral bands of hyperspectral images, a hybrid convolution and spectral symmetry preservation network has been proposed for hyperspectral super-resolution reconstruction. This includes a model to integrate information from neighboring spectral bands to supplement target band feature information. The proposed model introduces flexible spatial-spectral symmetric 3D convolution in the network structure to extract low-resolution and neighboring band features. At the same time, a combination of deformable convolution and attention mechanisms is used to extract information from low-resolution bands. Finally, multiple bands are fused in the reconstruction module, and the high-resolution hyperspectral image containing global information is obtained by Fourier transform upsampling. Experiments were conducted on the indoor hyperspectral image dataset CAVE, the airborne hyperspectral dataset Pavia Center, and Chikusei. In the X2 super-resolution task, the PSNR values achieved on the CAVE, Pavia Center, and Chikusei datasets were 46.335, 36.321, and 46.310, respectively. In the X4 super-resolution task, the PSNR values achieved on the CAVE, Pavia Center, and Chikusei datasets were 41.218, 30.377, and 38.365, respectively. The results show that our method outperforms many advanced algorithms in objective indicators such as PSNR and SSIM while maintaining the spectral characteristics of hyperspectral images.

[1]  Mingju Chen,et al.  Scene Reconstruction Algorithm for Unstructured Weak-Texture Regions Based on Stereo Vision , 2023, Applied Sciences.

[2]  Junhui Hou,et al.  Deep Diversity-Enhanced Feature Representation of Hyperspectral Images , 2023, ArXiv.

[3]  Xiaohui Liu,et al.  Hyperspectral image super-resolution based on attention ConvBiLSTM network , 2022, International Journal of Remote Sensing.

[4]  Qiang Li,et al.  Hyperspectral Image Superresolution Using Spectrum and Feature Context , 2021, IEEE Transactions on Industrial Electronics.

[5]  Jiayi Ma,et al.  Hyperspectral Image Super-Resolution via Recurrent Feedback Embedding and Spatial–Spectral Consistency Regularization , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Huan Liu,et al.  Satellite Video Super-Resolution Based on Adaptively Spatiotemporal Neighbors and Nonlocal Similarity Regularization , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[7]  Guoyong Zhang,et al.  Research on Secondary Analysis Method of Synchronous Satellite Monitoring Data of Power Grid Wildfire , 2020, 2020 IEEE International Conference on Information Technology,Big Data and Artificial Intelligence (ICIBA).

[8]  N. Kussul,et al.  Satellite Agricultural Monitoring in Ukraine at Country Level: World Bank Project , 2020, IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium.

[9]  Lianru Gao,et al.  Coupled Convolutional Neural Network With Adaptive Response Function Learning for Unsupervised Hyperspectral Super Resolution , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Hong Yan,et al.  A Truncated Matrix Decomposition for Hyperspectral Image Super-Resolution , 2020, IEEE Transactions on Image Processing.

[11]  Qi Wang,et al.  Mixed 2D/3D Convolutional Network for Hyperspectral Image Super-Resolution , 2020, Remote. Sens..

[12]  Junjun Jiang,et al.  Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery , 2020, IEEE Transactions on Computational Imaging.

[13]  Gang He,et al.  Hyperspectral Image Super-Resolution via Intrafusion Network , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Mahbubur Rahman,et al.  Toward Efficient Land Cover Mapping: An Overview of the National Land Representation System and Land Cover Map 2015 of Bangladesh , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[15]  Muna Darweesh,et al.  Simple Roads Extraction Algorithm Based on Edge Detection Using Satellite Images , 2019, 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC).

[16]  Shu-Tao Xia,et al.  Second-Order Attention Network for Single Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Jing Hu,et al.  Hyperspectral Image Super-Resolution by Deep Spatial-Spectral Exploitation , 2019, Remote. Sens..

[18]  Jocelyn Chanussot,et al.  Nonlocal Patch Tensor Sparse Representation for Hyperspectral Image Super-Resolution , 2019, IEEE Transactions on Image Processing.

[19]  Jie Liu,et al.  3D separable convolutional neural network for dynamic hand gesture recognition , 2018, Neurocomputing.

[20]  Lei Zhang,et al.  Single Hyperspectral Image Super-Resolution with Grouped Deep Recursive Residual Network , 2018, 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM).

[21]  Xiang Zhang,et al.  Monitoring of Drought Change in the Middle Reach of Yangtze River , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[22]  Shutao Li,et al.  Fusing Hyperspectral and Multispectral Images via Coupled Sparse Tensor Factorization , 2018, IEEE Transactions on Image Processing.

[23]  Hairong Qi,et al.  Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Gregory Shakhnarovich,et al.  Deep Back-Projection Networks for Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Xiai Chen,et al.  Hyperspectral Image Super-Resolution via Nonlocal Low-Rank Tensor Approximation and Total Variation Regularization , 2017, Remote. Sens..

[26]  Qian Du,et al.  Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network , 2017, Remote. Sens..

[27]  Tao Mei,et al.  Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Yunsong Li,et al.  Hyperspectral Image Super-Resolution by Spectral Difference Learning and Spatial Error Correction , 2017, IEEE Geoscience and Remote Sensing Letters.

[29]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[30]  Johannes R. Sveinsson,et al.  Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network , 2017, IEEE Geoscience and Remote Sensing Letters.

[31]  Xiangtao Zheng,et al.  Hyperspectral Image Superresolution by Transfer Learning , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[32]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Ajmal S. Mian,et al.  Bayesian sparse representation for hyperspectral image super resolution , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Jean-Yves Tourneret,et al.  Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Jing Yu,et al.  Super-resolution mapping via multi-dictionary based sparse representation , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[37]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Shree K. Nayar,et al.  Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum , 2010, IEEE Transactions on Image Processing.

[39]  Gunter Menz,et al.  Monitoring and Modeling Urban Land-Use Change with Multitemporal Satellite Data , 2008, IGARSS 2008 - 2008 IEEE International Geoscience and Remote Sensing Symposium.

[40]  Bruno Aiazzi,et al.  Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Yücel Altunbasak,et al.  Super-resolution reconstruction of hyperspectral images , 2004, IEEE Transactions on Image Processing.

[42]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[43]  Menas Kafatos,et al.  Wavelet-based hyperspectral and multispectral image fusion , 2001, SPIE Defense + Commercial Sensing.

[44]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[45]  Meng Xu,et al.  Diffused Convolutional Neural Network for Hyperspectral Image Super-Resolution , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Yinnian Liu.  Development of hyperspectral imaging remote sensing technology , 2021, National Remote Sensing Bulletin.