Inpainting of Irregular Holes in a Manuscript using UNet and Partial Convolution

Image inpainting of irregular holes in the old handwritten manuscripts is a challenging task. It requires finding the missing part in the manuscript and then filling it with an appropriate pixel. It needs to fill it in such a way that words or letters in that manuscript can be recovered. There are many degraded manuscripts available in museums and libraries all across the globe. This degradation can be due to human mishandling or certain environmental factors (fire, heat, water, microorganisms, pests, and other vermin). Even if it is tired very hard to protect our written heritage from all this, since it is written on natural materials (wood, paper), it is prone to inherent vice. A novel method is presented to inpaint the irregular holes in manuscripts with appropriate pixels. Deep learning, which tends to solve many complex problems like image classification, object detection, image segmentation, helps in achieving image inpainting as well. UNet is proposed to generate the mask of an irregular hole part in a manuscript image and partial convolution, which takes two input masks and the image for inpainting.

[1]  Lambert Schomaker,et al.  BiNet: Degraded-Manuscript Binarization in Diverse Document Textures and Layouts using Deep Encoder-Decoder Networks , 2019, ArXiv.

[2]  Peter Peer,et al.  End-to-End Iris Segmentation Using U-Net , 2018, 2018 IEEE International Work Conference on Bioinspired Intelligence (IWOBI).

[3]  Lambert Schomaker,et al.  A Multiple-Label Guided Clustering Algorithm for Historical Document Dating and Localization , 2016, IEEE Transactions on Image Processing.

[4]  Sitti Rachmawati Yahya,et al.  Review on image enhancement methods of old manuscript with the damaged background , 2009, 2009 International Conference on Electrical Engineering and Informatics.

[5]  A. M. Raid,et al.  Image Restoration Based on Morphological Operations , 2014 .

[6]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Apostolos Antonacopoulos,et al.  Special issue on the analysis of historical documents , 2007, International Journal of Document Analysis and Recognition (IJDAR).

[8]  Palaiahnakote Shivakumara,et al.  Text segmentation in degraded historical document images , 2016 .

[9]  Suresha,et al.  Degraded Document Analysis and Extraction of Original Text Document: An Approach without Optical Character Recognition , 2017 .

[10]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Mikhail Belyaev,et al.  Multidomain CT Metal Artifacts Reduction Using Partial Convolution Based Inpainting , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[12]  Hannes Nickisch,et al.  Learning metal artifact reduction in cardiac CT images with moving pacemakers , 2020, Medical Image Anal..

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[15]  Thomas Brox,et al.  U-Net: deep learning for cell counting, detection, and morphometry , 2018, Nature Methods.

[16]  G. Leedham,et al.  Decompose algorithm for thresholding degraded historical document images , 2005 .

[17]  Mohamed Cheriet,et al.  Text extraction from degraded document images , 2010, 2010 2nd European Workshop on Visual Information Processing (EUVIP).

[18]  Mohamed Cheriet,et al.  Historical document image restoration using multispectral imaging system , 2013, Pattern Recognit..

[19]  Arunkumar Chinnaswamy,et al.  Content Restoration of Degraded Termite Bitten Document Images , 2014 .

[20]  Song Li,et al.  Bottleneck feature supervised U-Net for pixel-wise liver and tumor segmentation , 2020, Expert Syst. Appl..

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  Xudong Zhao,et al.  Image Inpainting for Digital Dunhuang Murals Using Partial Convolutions and Sliding Window Method , 2019 .

[23]  Vijayakumar T Dr,et al.  COMPARATIVE STUDY OF CAPSULE NEURAL NETWORK IN VARIOUS APPLICATIONS , 2019, Journal of Artificial Intelligence and Capsule Networks.