Virtual restoration of the colored paintings on weathered beams in the Forbidden City using multiple deep learning algorithms

Abstract The colored paintings on the surfaces of ancient Chinese buildings have suffered from the wind and the sun and produced many defects such as paint loss, blurring, and color distortion. Previous methods have not been able to virtually repair them well. Therefore, this paper proposed a virtual restoration method for the weathered beams in the Forbidden City using multiple deep learning algorithms. Instead of using only one technology to restore paintings, this paper divided the painting into 3 parts, i.e., the background, the golden edges, and the dragon patterns, and restored them in different technology. For the background, this paper transformed the problem of unrecognizable color restoration into a semantic segmentation problem using U-Net MobileNet. For the golden edges, this paper used traditional image processing technology to obtain them from the color maps generated by the semantic segmentation algorithm. For the dragon patterns, after sketching the skeletons according to the dragon patterns, the image translation algorithm Pix2pix was applied to generate a realistic dragon pattern. Finally, the three repair results are superimposed to complete the repair. The virtually restored paintings can provide reference and guidance for traditional manual restoration and help the restorers to imagine what they looked like before oxidation, thus alleviating some repetitive work and reducing the complexity of restoration. Besides, each step of the method could form a layer, so that the restorers could overlay or modify them as they wish.

[1]  Roberto Pierdicca,et al.  Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage , 2020, Remote. Sens..

[2]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Peter E. D. Love,et al.  Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach , 2018, Adv. Eng. Informatics.

[4]  Lemonia Ragia,et al.  3D Building Façade Reconstruction Using Deep Learning , 2020, ISPRS Int. J. Geo Inf..

[5]  Ming-Sui Lee,et al.  Dunhuang mural restoration using deep learning , 2018, SIGGRAPH Asia Technical Briefs.

[6]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Fahad Shahbaz Khan,et al.  Semi-Supervised Learning for Few-Shot Image-to-Image Translation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jiuyong Li,et al.  Restoration of non-structural damaged murals in Shenzhen Bao’an based on a generator–discriminator network , 2021, Heritage Science.

[9]  Heung-Il Suk,et al.  Deep Learning in Medical Image Analysis. , 2017, Annual review of biomedical engineering.

[10]  Peter E.D. Love,et al.  Computer vision for behaviour-based safety in construction: A review and future directions , 2020, Adv. Eng. Informatics.

[11]  Yassine Ruichek,et al.  Survey on semantic segmentation using deep learning techniques , 2019, Neurocomputing.

[12]  Peng Zhao,et al.  Damage Classification for Masonry Historic Structures Using Convolutional Neural Networks Based on Still Images , 2018, Comput. Aided Civ. Infrastructure Eng..

[13]  Jenny Benois-Pineau,et al.  Connoisseur: classification of styles of Mexican architectural heritage with deep learning and visual attention prediction , 2017, CBMI.

[14]  Abdelhak Belhi,et al.  A machine learning framework for enhancing digital experiences in cultural heritage , 2020, J. Enterp. Inf. Manag..

[15]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[16]  Miriam A. M. Capretz,et al.  A systematic review of convolutional neural network-based structural condition assessment techniques , 2021 .

[17]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[18]  Peng Zhao,et al.  CNN-based statistics and location estimation of missing components in routine inspection of historic buildings , 2019, Journal of Cultural Heritage.

[19]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Yongqi Zhang,et al.  XOGAN: One-to-Many Unsupervised Image-to-Image Translation , 2018, ArXiv.

[21]  Qi Zhang,et al.  Ancient mural restoration based on a modified generative adversarial network , 2020, Heritage Science.

[22]  Ping Tan,et al.  DualGAN: Unsupervised Dual Learning for Image-to-Image Translation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Soo-Chang Pei,et al.  Virtual restoration of ancient Chinese paintings using color contrast enhancement and lacuna texture synthesis , 2004, IEEE Transactions on Image Processing.

[24]  Peter E. D. Love,et al.  A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network , 2019, Adv. Eng. Informatics.

[25]  Xuefeng Zhao,et al.  Automatic segmentation, inpainting, and classification of defective patterns on ancient architecture using multiple deep learning algorithms , 2021, Structural Control and Health Monitoring.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Taesung Park,et al.  Semantic Image Synthesis With Spatially-Adaptive Normalization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[29]  Hee-Jun Kang,et al.  A survey on Deep Learning based bearing fault diagnosis , 2019, Neurocomputing.

[30]  Subhajit Chaudhury,et al.  Can fully convolutional networks perform well for general image restoration problems? , 2016, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[31]  Tarek Hegazy,et al.  Review of image-based analysis and applications in construction , 2021 .

[32]  Xiangrui Zeng,et al.  Controllable digital restoration of ancient paintings using convolutional neural network and nearest neighbor , 2020, Pattern Recognit. Lett..

[33]  P. Praveen Kumar,et al.  Restoration of artwork using deep neural networks , 2019, Evol. Syst..

[34]  Roberto Medina,et al.  Classification of Architectural Heritage Images Using Deep Learning Techniques , 2017 .

[35]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Jiaya Jia,et al.  Homomorphic Latent Space Interpolation for Unpaired Image-To-Image Translation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Youngjung Uh,et al.  Rethinking the Truly Unsupervised Image-to-Image Translation , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[38]  Andreas Kamilaris,et al.  Deep learning in agriculture: A survey , 2018, Comput. Electron. Agric..

[39]  Bruno Cornelis,et al.  Crack Detection in Paintings Using Convolutional Neural Networks , 2020, IEEE Access.

[40]  Bob D. de Vos,et al.  State-of-the-Art Deep Learning in Cardiovascular Image Analysis. , 2019, JACC. Cardiovascular imaging.

[41]  Eduardo Zalama Casanova,et al.  Applying Deep Learning Techniques to Cultural Heritage Images Within the INCEPTION Project , 2016, EuroMed.

[42]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Aleksandra Pizurica,et al.  Crack detection and inpainting for virtual restoration of paintings: The case of the Ghent Altarpiece , 2013, Signal Process..

[44]  Ran Xu,et al.  Application of Color Transfer Algorithm in the Virtual Color Restoration of Ancient Architecture , 2013 .

[45]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[46]  Rongrong Ji,et al.  Attribute Guided Unpaired Image-to-Image Translation with Semi-supervised Learning , 2019, ArXiv.