Multi-scale attention network for image inpainting

Abstract Recently, deep learning based inpainting methods have shown promising performance, in which some multi-scale networks are introduced to characterize image content in both details and structures. However, few of these networks explore local spatial components under different receptive fields and internal connection between multi-scale feature maps. In this paper, we propose a novel multi-scale attention network (MSA-Net) to fill the irregular missing regions, in which a multi-scale attention group (MSAG) with several multi-scale attention units (MSAUs) is introduced for fully analysing the features from shallow details to high-level semantics. In each MSAU, an attention based spatial pyramid structure is designed to capture the deep features from different receptive fields. In this network, attention mechanisms are explored for boosting the representation power of MSAU, where spatial attention is combined with each scale to highlight the most probably attentive spatial components and the channel attention is used as a globally semantic detector to build the connection between the multiple scales. Furthermore, for better inpainting results, a max pooling based mask update method is utilized to predict the missing parts from the border regions to the inside. Finally, experiments on Places2 dataset and CelebA dataset demonstrate that the proposed method can achieve better results than the previous inpainting methods.