MetaPan: Unsupervised Adaptation With Meta-Learning for Multispectral Pansharpening

Multispectral (MS) pansharpening aims to improve the spatial resolution of MS images (MSIs) using the spatial details of panchromatic (PAN) images. Due to the gap of prior knowledge between the simulated data and real-world cases, unsupervised learning-based approaches have grown increasing interest. However, some key hyper-parameters, such as the initial weights of the networks, are set manually, which significantly impacts the fusion performance. To tackle this problem, we propose a novel unsupervised adaptation method with meta-learning for MS pansharpening (MetaPan), in which the meta-learning aims to automatically learn the initial parameters of a three-stream fusion network (TSFNet) for unsupervised adaptation learning (UAL). Specifically, the TSFNet consists of a PAN stream, an MS stream, and a fusion stream, where the fusion stream implicitly leverages domain-specific knowledge of input image pairs while the other two streams explicitly inject spatial details and spectral information into the fusion stream. The MetaPan consists of a pretraining stage, a meta-learning stage, and a UAL stage. At the pretraining stage, the TSFNet is trained with the supervision of simulated ground truth such that it is universal for all image pairs. Then, the process of meta-learning optimizes for an internal representation of network parameters that can adapt to a specific image pair with UAL through only a few steps. Finally, the learned internal representation is fine-tuned to a real-world image pair (a test image pair) with UAL. Experiments on two datasets show that our method performs better than state-of-the-art methods in both quantitative metrics and visual appearance.