Improved U-Net3+ With Spatial–Spectral Transformer for Multispectral Image Reconstruction

Multispectral image reconstruction, which aims to recover a three-dimensional (3D) spatial–spectral signal from a two-dimensional measurement in a spectral camera based on ghost imaging via sparsity constraint (GISC), has been attracting much attention recently. However, faced with abundant 3D spectral data, the reconstruction quality cannot meet the visual requirements. Based on the robust data processing capability of deep learning, a novel network called SSTU-Net3+ is constructed by improving U-Net3+ with a spatial–spectral transformer (SST). To enhance the feature representation of images during reconstruction, mixed pooling modules and new convolution processes are proposed to improve the performance of the encoder and decoder, with U-Net3+ as the backbone. To boost the quality of reconstructed images, with split and concatenate (Concat) operations, we construct SST modules by exploiting both spatial and spectral correlations of multispectral images to refine the spatial and spectral features. Furthermore, we employ the SST in the decoder to reconstruct the desired 3D cube. Given similar network parameters, experiments on GISC spectral imaging data show that, compared to convolutional neural network–based methods, the average peak signal-to-noise ratio of images reconstructed using SSTU-Net3+ is improved by 3%, the structural similarity is enhanced by 3%, and the spectral angle mapping is cut by 12%. Particularly, compared to differential ghost imaging and compressed sensing, the reconstruction quality of SSTU-Net3+ has been significantly improved. SSTU-Net3+ can process a large amount of 3D multispectral image data more efficiently and construct the target image more accurately than the abovementioned methods.

[1]  Shensheng Han,et al.  Hyperspectral Image Reconstruction for Spectral Camera Based on Ghost Imaging Via Sparsity Constraints Using V-Dunet , 2022, Social Science Research Network.

[2]  L. Gool,et al.  MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  L. Gool,et al.  Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis , 2022, ArXiv.

[4]  L. Gool,et al.  Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Tao Mei,et al.  Contextual Transformer Networks for Visual Recognition , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Xin Yuan,et al.  Spectral Compressive Imaging Reconstruction Using Convolution and Spectral Contextual Transformer , 2022, ArXiv.

[7]  Shensheng Han,et al.  Spectral polarization camera based on ghost imaging via sparsity constraints. , 2021, Applied optics.

[8]  Genping Zhao,et al.  Hybrid neural network-based adaptive computational ghost imaging , 2021 .

[9]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[10]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Xin Yuan,et al.  End-to-End Low Cost Compressive Spectral Imaging with Spatial-Spectral Self-Attention , 2020, ECCV.

[12]  Lanfen Lin,et al.  UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Shensheng Han,et al.  Ghost imaging based on Y-net: a dynamic coding and conjugate-decoding approach , 2020, 2002.03824.

[14]  Xianmin Zhang,et al.  Sub-Nyquist computational ghost imaging with deep learning. , 2020, Optics express.

[15]  Vassilis Athitsos,et al.  lambda-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Tao Zhang,et al.  Hyperspectral Image Reconstruction Using Deep External and Internal Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Fei Wang,et al.  Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging. , 2019, Optics express.

[18]  Jian Yang,et al.  Selective Kernel Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[20]  Boaz Arad,et al.  Sparse Recovery of Hyperspectral Signal from Natural RGB Images , 2016, ECCV.

[21]  Zhishen Tong,et al.  Spectral Camera based on Ghost Imaging via Sparsity Constraints , 2015, Scientific Reports.

[22]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[23]  Zhihua Wei,et al.  Mixed Pooling for Convolutional Neural Networks , 2014, RSKT.

[24]  Henry Arguello,et al.  Compressive Coded Aperture Spectral Imaging: An Introduction , 2014, IEEE Signal Processing Magazine.

[25]  Marzio Giglio,et al.  X-ray-scattering information obtained from near-field speckle , 2008 .

[26]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[27]  Shih,et al.  Optical imaging by means of two-photon quantum entanglement. , 1995, Physical review. A, Atomic, molecular, and optical physics.

[28]  Shih,et al.  Observation of two-photon "ghost" interference and diffraction. , 1995, Physical review letters.

[29]  Fred A. Kruse,et al.  The Spectral Image Processing System (SIPS) - Interactive visualization and analysis of imaging spectrometer data , 1993 .