GTP-PNet: A residual learning network based on gradient transformation prior for pansharpening

Abstract Pansharpening aims to fuse low-resolution multi-spectral image and high-resolution panchromatic (PAN) image to produce a high-resolution multi-spectral (HRMS) image. In this paper, a new residual learning network based on gradient transformation prior, termed as GTP-PNet, is proposed to generate the high-quality HRMS image with accurate spectral distribution as well as reasonable spatial structure. Different from previous deep models that only rely on supervision of the HRMS reference image, we introduce the gradient transformation prior to the deep model, so as to improve the solution accuracy. Our model consists of two networks, namely gradient transformation network (TNet) and pansharpening network (PNet). TNet is committed to seeking the nonlinear mapping between gradients of PAN and HRMS images, which is essentially a spatial relationship regression of imaging bands in different ranges. PNet is the residual learning network used to generate the HRMS image, which is not only supervised by the HRMS reference image, but also constrained by the trained TNet. As a result, the HRMS image generated by PNet not only approximates the HRMS reference image in the spectral distribution, but also conforms to the gradient transformation prior in the spatial structure. Experimental results demonstrate the significant superiority of our method over the current state-of-the-arts in terms of both subjective visual effect and quantitative metrics. We also apply our method to produce the HR normalized difference vegetation index in remote sensing, which can achieve the best performance. Moreover, our method is much competitive compared with the state-of-the-art alternatives in running efficiency.