Dilated projection correction network based on autoencoder for hyperspectral image super-resolution

This paper focuses on improving the spatial resolution of the hyperspectral image (HSI) by taking the prior information into consideration. In recent years, single HSI super-resolution methods based on deep learning have achieved good performance. However, most of them only simply apply general image super-resolution deep networks to hyperspectral data, thus ignoring some specific characteristics of hyperspectral data itself. In order to make full use of spectral information of the HSI, we transform the HSI SR problem from the image domain into the abundance domain by the dilated projection correction network with an autoencoder, termed as aeDPCN. In particular, we first encode the low-resolution HSI to abundance representation and preserve the spectral information in the decoder network, which could largely reduce the computational complexity. Then, to enhance the spatial resolution of the abundance embedding, we super-resolve the embedding in a coarse-to-fine manner by the dilated projection correction network where the back-projection strategy is introduced to further eliminate spectral distortion. Finally, the predictive images are derived by the same decoder, which increases the stability of our method, even at a large upscaling factor. Extensive experiments on real hyperspectral image scenes demonstrate the superiority of our method over the state-of-the-art, in terms of accuracy and efficiency.

[1]  Yunsong Li,et al.  Hyperspectral Image Super-Resolution by Spectral Difference Learning and Spatial Error Correction , 2017, IEEE Geoscience and Remote Sensing Letters.

[2]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[3]  Bo Du,et al.  A Discriminative Metric Learning Based Anomaly Detection Method , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Yunsong Li,et al.  Hyperspectral Image Super-Resolution Using Deep Feature Matrix Factorization , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[5]  Jing Yu,et al.  Super-resolution mapping via multi-dictionary based sparse representation , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Gregory Shakhnarovich,et al.  Deep Back-Projection Networks for Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[8]  Qian Du,et al.  Foreword to the Special Issue on Spectral Unmixing of Remotely Sensed Data , 2011 .

[9]  Naoto Yokoya,et al.  Hyperspectral Super-Resolution of Locally Low Rank Images From Complementary Multisource Data , 2014, IEEE Transactions on Image Processing.

[10]  Guangming Shi,et al.  Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation. , 2016, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[11]  Jocelyn Chanussot,et al.  Multimorphological Superpixel Model for Hyperspectral Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Yuanchao Su,et al.  DAEN: Deep Autoencoder Networks for Hyperspectral Unmixing , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[14]  Konrad Schindler,et al.  Hyperspectral Super-Resolution by Coupled Spectral Unmixing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  Fred A. Kruse,et al.  The Spectral Image Processing System (SIPS) - Interactive visualization and analysis of imaging spectrometer data , 1993 .

[16]  Feng Zhao,et al.  Unsupervised/Supervised Hybrid Deep Learning Framework for Low Dose Phase Contrast Imaging , 2020, Journal of Physics: Conference Series.

[17]  Zhi Han,et al.  Super-resolution reconstruction of hyperspectral images via low rank tensor modeling and total variation regularization , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[18]  Qian Du,et al.  Hyperspectral Image Spatial Super-Resolution via 3D Full Convolutional Neural Network , 2017, Remote. Sens..

[19]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Hao Wu,et al.  Semi-Supervised Deep Learning Using Pseudo Labels for Hyperspectral Image Classification , 2018, IEEE Transactions on Image Processing.

[22]  Wei Huang,et al.  Fusion of satellite images in urban area: Assessing the quality of resulting images , 2010, 2010 18th International Conference on Geoinformatics.

[23]  Hassan Mansour,et al.  Deepcasd: An End-to-End Approach for Multi-Spectral Image Super-Resolution , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[24]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[25]  Hairong Qi,et al.  Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Lianru Gao,et al.  Separable-spectral convolution and inception network for hyperspectral image super-resolution , 2019, Int. J. Mach. Learn. Cybern..

[27]  Naoto Yokoya,et al.  Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Jiayi Ma,et al.  Hyperspectral Image Super-Resolution via Recurrent Feedback Embedding and Spatial–Spectral Consistency Regularization , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Qi Wang,et al.  Mixed 2D/3D Convolutional Network for Hyperspectral Image Super-Resolution , 2020, Remote. Sens..

[30]  Yunsong Li,et al.  Trainable spectral difference learning with spatial starting for hyperspectral image denoising , 2018, Neural Networks.

[31]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[32]  Xiuping Jia,et al.  Unsupervised spectral mapping and feature selection for hyperspectral anomaly detection , 2020, Neural Networks.

[33]  Jian Yang,et al.  MemNet: A Persistent Memory Network for Image Restoration , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[34]  Lei Zhang,et al.  Single Hyperspectral Image Super-Resolution with Grouped Deep Recursive Residual Network , 2018, 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM).

[35]  Jing Hu,et al.  Hyperspectral Image Super-Resolution by Deep Spatial-Spectral Exploitation , 2019, Remote. Sens..

[36]  Ajmal S. Mian,et al.  Bayesian sparse representation for hyperspectral image super resolution , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Johannes R. Sveinsson,et al.  Hyperspectral Unmixing Using a Neural Network Autoencoder , 2018, IEEE Access.

[38]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[39]  Gozde Bozdagi Akar,et al.  A MAP-Based Approach for Hyperspectral Imagery Super-Resolution , 2018, IEEE Transactions on Image Processing.

[40]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Junjun Jiang,et al.  Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery , 2020, IEEE Transactions on Computational Imaging.

[42]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Jean-Yves Tourneret,et al.  Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Garrison W. Cottrell,et al.  Understanding Convolution for Semantic Segmentation , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[45]  Yunsong Li,et al.  Hyperspectral image super-resolution using deep convolutional neural network , 2017, Neurocomputing.

[46]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[47]  Thomas S. Huang,et al.  Non-Local Recurrent Network for Image Restoration , 2018, NeurIPS.

[48]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[49]  Xiangtao Zheng,et al.  Hyperspectral Image Superresolution by Transfer Learning , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[50]  Jocelyn Chanussot,et al.  A Convex Formulation for Hyperspectral Image Superresolution via Subspace-Based Regularization , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[51]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Naoto Yokoya,et al.  Hyperspectral and Multispectral Data Fusion: A comparative review of the recent literature , 2017, IEEE Geoscience and Remote Sensing Magazine.

[54]  Hairong Qi,et al.  uDAS: An Untied Denoising Autoencoder With Sparsity for Spectral Unmixing , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[55]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[56]  Francesco Carlo Morabito,et al.  A novel automatic classification system based on hybrid unsupervised and supervised machine learning for electrospun nanofibers , 2021, IEEE/CAA Journal of Automatica Sinica.

[57]  Hairong Qi,et al.  Endmember Extraction From Highly Mixed Data Using Minimum Volume Constrained Nonnegative Matrix Factorization , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[58]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[59]  Xiuping Jia,et al.  Spectral constraint adversarial autoencoders approach to feature representation in hyperspectral anomaly detection , 2019, Neural Networks.

[60]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).