Hyperspectral Image Denoising via Spatial–Spectral Recurrent Transformer

Hyperspectral images (HSIs) often suffer from noise arising from both intra-imaging mechanisms and environmental factors. Leveraging domain knowledge specific to HSIs, such as global spectral correlation (GSC) and non-local spatial self-similarity (NSS), is crucial for effective denoising. Existing methods tend to independently utilize each of these knowledge components with multiple blocks, overlooking the inherent 3D nature of HSIs where domain knowledge is strongly interlinked, resulting in suboptimal performance. To address this challenge, this paper introduces a spatial-spectral recurrent transformer U-Net (SSRT-UNet) for HSI denoising. The proposed SSRT-UNet integrates NSS and GSC properties within a single SSRT block. This block consists of a spatial branch and a spectral branch. The spectral branch employs a combination of transformer and recurrent neural network to perform recurrent computations across bands, allowing for GSC exploitation beyond a fixed number of bands. Concurrently, the spatial branch encodes NSS for each band by sharing keys and values with the spectral branch under the guidance of GSC. This interaction between the two branches enables the joint utilization of NSS and GSC, avoiding their independent treatment. Experimental results demonstrate that our method outperforms several alternative approaches. The source code will be available at https://github.com/lronkitty/SSRT.

[1]  Bing Tu,et al.  Graph Evolution-Based Vertex Extraction for Hyperspectral Anomaly Detection. , 2023, IEEE transactions on neural networks and learning systems.

[2]  Q. Yuan,et al.  Hyperspectral Image Denoising: From Model-Driven, Data-Driven, to Model-Data-Driven. , 2023, IEEE transactions on neural networks and learning systems.

[3]  D. Dou,et al.  Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  D. Tao,et al.  DDS2M: Self-Supervised Denoising Diffusion Spatio-Spectral Model for Hyperspectral Image Restoration , 2023, 2023 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  Yulun Zhang,et al.  Spatial-Spectral Transformer for Hyperspectral Image Denoising , 2022, AAAI.

[6]  Hongyan Zhang,et al.  Hider: A Hyperspectral Image Denoising Transformer With Spatial-Spectral Constraints for Hybrid Noise Removal. , 2022, IEEE transactions on neural networks and learning systems.

[7]  Q. Yuan,et al.  Cooperated Spectral Low-Rankness Prior and Deep Spatial Prior for HSI Unsupervised Denoising , 2022, IEEE Transactions on Image Processing.

[8]  Xiangyong Cao,et al.  TRQ3DNet: A 3D Quasi-Recurrent and Transformer Based Network for Hyperspectral Image Denoising , 2022, Remote. Sens..

[9]  Y. Duan,et al.  MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer , 2022, IEEE Transactions on Image Processing.

[10]  J. Chanussot,et al.  Spectral-Spatial Transformer for Hyperspectral Image Sharpening , 2022, IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium.

[11]  J. Chanussot,et al.  A survey on hyperspectral image restoration: from the view of low-rank tensor approximation , 2022, Science China Information Sciences.

[12]  Zhuqing He,et al.  Vision Transformers for Single Image Dehazing , 2022, IEEE Transactions on Image Processing.

[13]  Yuhuai Wu,et al.  Block-Recurrent Transformers , 2022, NeurIPS.

[14]  L. Gool,et al.  Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction , 2022, ECCV.

[15]  Jocelyn Chanussot,et al.  A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration , 2021, NeurIPS.

[16]  L. Gool,et al.  Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Guangyi Yang,et al.  LR-Net: Low-Rank Spatial-Spectral Network for Hyperspectral Image Denoising , 2021, IEEE Transactions on Image Processing.

[18]  Luc Van Gool,et al.  SwinIR: Image Restoration Using Swin Transformer , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[19]  Lianru Gao,et al.  SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Stephen Lin,et al.  Video Swin Transformer , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Shiwei Zhang,et al.  End-to-End Temporal Action Detection With Transformer , 2021, IEEE Transactions on Image Processing.

[22]  Yushi Chen,et al.  Spatial-Spectral Transformer for Hyperspectral Image Classification , 2021, Remote. Sens..

[23]  Jiantao Zhou,et al.  SMDS-Net: Model Guided Spectral-Spatial Network for Hyperspectral Image Denoising , 2020, IEEE Transactions on Image Processing.

[24]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[25]  Naoto Yokoya,et al.  Non-Local Meets Global: An Iterative Paradigm for Hyperspectral Image Restoration , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Yao Wang,et al.  Enhanced 3DTV Regularization and Its Applications on HSI Denoising and Compressed Sensing , 2020, IEEE Transactions on Image Processing.

[27]  Javier Plaza,et al.  A Single Model CNN for Hyperspectral Image Denoising , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Ying Fu,et al.  3-D Quasi-Recurrent Neural Network for Hyperspectral Image Denoising , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Wei Li,et al.  HSI-BERT: Hyperspectral Image Classification Using the Bidirectional Encoder Representation From Transformers , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Jun Zhou,et al.  Hyperspectral Restoration via $L_0$ Gradient Regularized Low-Rank Tensor Factorization , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Cuiling Lan,et al.  EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks , 2019, IEEE Transactions on Image Processing.

[32]  Ying Fu,et al.  Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Joonwhoan Lee,et al.  A 3-D Atrous Convolution Neural Network for Hyperspectral Image Denoising , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[34]  Yiming Yang,et al.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context , 2019, ACL.

[35]  Qiang Zhang,et al.  Hyperspectral Image Denoising Employing a Spatial–Spectral Deep Residual Convolutional Neural Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[36]  José M. Bioucas-Dias,et al.  Fast Hyperspectral Image Denoising and Inpainting Based on Low-Rank and Sparse Representations , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[37]  Jun Zhou,et al.  Nonlocal Similarity Based Nonnegative Tucker Decomposition for Hyperspectral Image Denoising , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[38]  Yu Zhang,et al.  Simple Recurrent Units for Highly Parallelizable Recurrence , 2017, EMNLP.

[39]  Sheng Zhong,et al.  Hyper-Laplacian Regularized Unidirectional Low-Rank Tensor Recovery for Multispectral Image Denoising , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Deyu Meng,et al.  Denoising Hyperspectral Image With Non-i.i.d. Noise Structure , 2017, IEEE Transactions on Cybernetics.

[41]  Bo Du,et al.  Beyond the Sparsity-Based Target Detector: A Hybrid Sparsity and Statistics-Based Detector for Hyperspectral Images , 2016, IEEE Transactions on Image Processing.

[42]  Heesung Kwon,et al.  Going Deeper With Contextual CNN for Hyperspectral Image Classification , 2016, IEEE Transactions on Image Processing.

[43]  Jun Zhou,et al.  Multitask Sparse Nonnegative Matrix Factorization for Joint Spectral–Spatial Hyperspectral Imagery Denoising , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[44]  Liangpei Zhang,et al.  Hyperspectral Image Denoising via Noise-Adjusted Iterative Low-Rank Matrix Approximation , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[45]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[46]  Arif Mahmood,et al.  Hyperspectral Face Recognition With Spatiospectral Information Fusion and PLS Regression , 2015, IEEE Transactions on Image Processing.

[47]  Liangpei Zhang,et al.  Hyperspectral Image Restoration Using Low-Rank Matrix Recovery , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Yi Yang,et al.  Decomposable Nonlocal Tensor Dictionary Learning for Multispectral Image Denoising , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[50]  Hanlin Qin,et al.  Spatial–Spectral Oriented Triple Attention Network for Hyperspectral Image Denoising , 2024, IEEE Transactions on Geoscience and Remote Sensing.

[51]  S. Jia,et al.  Texture-Aware Self-Attention Model for Hyperspectral Tree Species Classification , 2024, IEEE Transactions on Geoscience and Remote Sensing.

[52]  Yilong Lu,et al.  Nonlocal Structured Sparsity Regularization Modeling for Hyperspectral Image Denoising , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[53]  Xiyou Fu,et al.  Hyperspectral Image Denoising via Robust Subspace Estimation and Group Sparsity Constraint , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[54]  Jiangtao Peng,et al.  Cross-Channel Dynamic Spatial–Spectral Fusion Transformer for Hyperspectral Image Classification , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[55]  Xin Su,et al.  Fast Hyperspectral Image Denoising and Destriping Method Based on Graph Laplacian Regularization , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[56]  Xile Zhao,et al.  Hyperspectral Image Denoising: Reconciling Sparse and Low-Tensor-Ring-Rank Priors in the Transformed Domain , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[57]  Fulin Luo,et al.  Multiscale Diff-Changed Feature Fusion Network for Hyperspectral Image Change Detection , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[58]  Jing Zhao,et al.  Abundance Matrix Correlation Analysis Network Based on Hierarchical Multihead Self-Cross-Hybrid Attention for Hyperspectral Change Detection , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Y. Chen,et al.  Hyperspectral Image Denoising Via Texture-Preserved Total Variation Regularizer , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[60]  Lianru Gao,et al.  Hyperspectral Anomaly Detection Based on Chessboard Topology , 2023, IEEE Transactions on Geoscience and Remote Sensing.

[61]  Y. Qian,et al.  Nonlocal Spatial–Spectral Neural Network for Hyperspectral Image Denoising , 2022, IEEE Transactions on Geoscience and Remote Sensing.

[62]  Q. Yuan,et al.  Local–Global Feature-Aware Transformer Based Residual Network for Hyperspectral Image Denoising , 2022, IEEE Transactions on Geoscience and Remote Sensing.

[63]  Chengjun Wang,et al.  Translution-SNet: A Semisupervised Hyperspectral Image Stripe Noise Removal Based on Transformer and CNN , 2022, IEEE Transactions on Geoscience and Remote Sensing.

[64]  Y. Qian,et al.  MAC-Net: Model Aided Nonlocal Neural Network for Hyperspectral Image Denoising , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[65]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[66]  A. Foi,et al.  Nonlocal Transform-Domain Filter for Volumetric Data Denoising and Reconstruction , 2013, IEEE Transactions on Image Processing.