Hyperspectral Pansharpening With Adaptive Feature Modulation-Based Detail Injection Network

Recently, deep learning-based methodologies have attained unprecedented performance in hyperspectral (HS) pansharpening, which aims to improve the spatial quality of HS images (HSIs) by making use of details extracted from the high-resolution panchromatic (HR-PAN) image. However, it remains challenging to incorporate the details into the pansharpened image effectively, while alleviating the spectral distortion simultaneously. To tackle this problem, in this article, we propose an adaptive feature modulation-based detail injection network (AFM-DIN) for HS pansharpening, which mainly consists of four phases: high-frequency details generation of the HR-PAN image, multiscale feature extraction of the upsampled HSI, AFM-based detail injection, and reconstruction of the HR-HSI. First, a novel octave convolution unit is employed to decompose the HR-PAN image into high and low frequencies, and then merge the high-frequency features together to generate the comprehensive PAN-details. Second, the spatial and spectral separable 3-D convolution units with multiple kernel sizes are designed to extract multiscale features of the upsampled HSI in a computationally efficient manner. Subsequently, by taking the critical PAN-details as prior, the proposed AFM module is able to not only incorporate the detail information effectively, but also adjust the injected details adaptively to ensure the spectral fidelity. Finally, the anticipated HR-HSI is obtained through adding the upsampled HSI to the predicted HSI-details reconstructed from informative modulated features. Extensive comparison experiments with several state-of-the-arts conducted on simulated and real HS data sets demonstrate that our proposed AFM-DIN can achieve superior pansharpening accuracy in both spatial and spectral aspects.