PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion

The Transformer architecture has achieved rapid development in recent years, outperforming the CNN architectures in many computer vision tasks, such as the Vision Transformers (ViT) for image classification. However, existing visual transformer models aim to extract semantic information for high-level tasks such as classification and detection, distorting the spatial resolution of the input image, without the capacity in reconstructing the input or generating high-resolution image. In this paper, therefore, we propose a Patch Pyramid Transformer(PPT) to effectively address the above issues. Specifically, we first design a Patch Transformer to transform the image into a sequence of patches, where transformer encoding is performed for each patch to extract local representations. In addition, we construct a Pyramid Transformer to effectively extract the nonlocal information from the entire image. After obtaining a set of multi-scale, multi-dimensional, and multi-angle features of the original image, we design the image reconstruction network to ensure that the features can be reconstructed into the input image. To validate the effectiveness, we apply the proposed Patch Pyramid Transformer to the image fusion task. The experimental results demonstrate its superior performance against the stateof-the-art fusion approaches, achieving the best results on several evaluation indicators. The underlying capacity of the PPT network is reflected by its universal power in feature extraction and image reconstruction, which can be directly applied to different image fusion tasks without redesigning or retraining the network.