Multi-stage Progressive Reasoning for Dunhuang Murals Inpainting

Dunhuang murals suffer from fading, breakage, surface brittleness and extensive peeling affected by prolonged environmental erosion. Image inpainting techniques are widely used in the field of digital mural inpainting. Generally speaking, for mural inpainting tasks with large area damage, it is challenging for any image inpainting method. In this paper, we design a multi-stage progressive reasoning network (MPR-Net) containing global to local receptive fields for murals inpainting. This network is capable of recursively inferring the damage boundary and progressively tightening the regional texture constraints. Moreover, to adaptively fuse plentiful information at various scales of murals, a multi-scale feature aggregation module (MFA) is designed to empower the capability to select the significant features. The execution of the model is similar to the process of a mural restorer (i.e., inpainting the structure of the damaged mural globally first and then adding the local texture details further). Our method has been evaluated through both qualitative and quantitative experiments, and the results demonstrate that it outperforms state-of-the-art image inpainting methods.

[1]  Wenyin Liu,et al.  Multi-scale patch-GAN with edge detection for image inpainting , 2022, Applied Intelligence.

[2]  C. Qin,et al.  Region-aware Attention for Image Inpainting , 2022, ArXiv.

[3]  Zhifeng Li,et al.  Image Inpainting With Local and Global Refinement , 2022, IEEE Transactions on Image Processing.

[4]  Hongyu Yang,et al.  Image Inpainting via Conditional Texture and Structure Dual Generation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  Felix Juefei-Xu,et al.  JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting , 2021, ACM Multimedia.

[6]  Dacheng Tao,et al.  Recurrent Feature Reasoning for Image Inpainting , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Zhe L. Lin,et al.  High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling , 2020, ECCV.

[8]  Bo Du,et al.  Progressive Reconstruction of Visual Structure for Image Inpainting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Faisal Z. Qureshi,et al.  EdgeConnect: Structure Guided Image Inpainting using Edge Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[10]  Wangmeng Zuo,et al.  Image Inpainting With Learnable Bidirectional Attention Maps , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Seoung Wug Oh,et al.  Onion-Peel Networks for Deep Video Completion , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Thomas H. Li,et al.  StructureFlow: Image Inpainting via Structure-Aware Appearance Flow , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Sen Liu,et al.  Progressive Image Inpainting with Full-Resolution Residual Network , 2019, ACM Multimedia.

[14]  Bin Jiang,et al.  Coherent Semantic Attention for Image Inpainting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  Baining Guo,et al.  Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Wei Xiong,et al.  Foreground-Aware Image Inpainting , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Meng Wang,et al.  Semantic Image Inpainting with Progressive Generative Networks , 2018, ACM Multimedia.

[18]  Thomas S. Huang,et al.  Free-Form Image Inpainting With Gated Convolution , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[20]  Shiguang Shan,et al.  Shift-Net: Image Inpainting via Deep Feature Rearrangement , 2018, ECCV.

[21]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Chao Yang,et al.  Contextual-Based Image Inpainting: Infer, Match, and Translate , 2017, ECCV.

[23]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[25]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Qi Wang,et al.  Virtual Completion of Facial Image in Ancient Murals , 2011, 2011 Workshop on Digital Media and Digital Content Management.

[27]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2008, Commun. ACM.

[28]  Jiaya Jia,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[29]  Rachid Deriche,et al.  Vector-valued image regularization with PDEs: a common framework for different applications , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[31]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[32]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[33]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[34]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[35]  Wenjie Liu,et al.  Dunhuang Mural Line Drawing Based on Bi-Dexined Network and Adaptive Weight Learning , 2022, PRCV.