Image Inpainting: A Review

Although image inpainting, or the art of repairing the old and deteriorated images, has been around for many years, it has recently gained even more popularity, because of the recent development in image processing techniques. With the improvement of image processing tools and the flexibility of digital image editing, automatic image inpainting has found important applications in computer vision and has also become an important and challenging topic of research in image processing. This paper reviews the existing image inpainting approaches, that were classified into three subcategories, sequential-based, CNN-based, and GAN-based methods. In addition, for each category, a list of methods for different types of distortion on images are presented. Furthermore, the paper also presents available datasets. Last but not least, we present the results of real evaluations of the three categories of image inpainting methods performed on the used datasets, for different types of image distortion. We also present the evaluations metrics and discuss the performance of these methods in terms of these metrics. This overview can be used as a reference for image inpainting researchers, and it can also facilitate the comparison of the methods as well as the datasets used. The main contribution of this paper is the presentation of the three categories of image inpainting methods along with a list of available datasets that the researchers can use to evaluate their proposed methodology against.

[1]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[2]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Alexei A. Efros,et al.  What makes Paris look like Paris? , 2015, Commun. ACM.

[4]  Qing Zhang,et al.  Exemplar-Based Image Inpainting Using Color Distribution Analysis , 2012, J. Inf. Sci. Eng..

[5]  Vicent Caselles,et al.  Exemplar-Based Image Inpainting Using Multiscale Graph Cuts , 2013, IEEE Transactions on Image Processing.

[6]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[7]  Jon Almazán,et al.  ICDAR 2013 Robust Reading Competition , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[8]  Chin-Chen Chang,et al.  A novel joint data-hiding and compression scheme based on SMVQ and image inpainting. , 2014, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[9]  Wei Jiang Rate-distortion optimized image compression based on image inpainting , 2014, Multimedia Tools and Applications.

[10]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[11]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[12]  Chin-Chen Chang,et al.  A Novel Joint Data-Hiding and Compression Scheme Based on SMVQ and Image Inpainting , 2014, IEEE Transactions on Image Processing.

[13]  M. F. Baumgardner,et al.  220 Band AVIRIS Hyperspectral Image Data Set: June 12, 1992 Indian Pine Test Site 3 , 2015 .

[14]  Zhijing Yang,et al.  Blind inpainting using the fully convolutional neural network , 2017, The Visual Computer.

[15]  Alexei A. Efros,et al.  What makes Paris look like Paris? , 2015, Commun. ACM.

[16]  Aleksandra Pizurica,et al.  Context-Aware Patch-Based Image Inpainting Using Markov Random Field Modeling , 2015, IEEE Transactions on Image Processing.

[17]  Omar ElHarrouss,et al.  Moving objects detection based on thresholding operations for video surveillance systems , 2015, 2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA).

[18]  Meng Wang,et al.  Image-Based Three-Dimensional Human Pose Recovery by Multiview Locality-Sensitive Sparse Retrieval , 2015, IEEE Transactions on Industrial Electronics.

[19]  Meng Wang,et al.  Multimodal Deep Autoencoder for Human Pose Recovery , 2015, IEEE Transactions on Image Processing.

[20]  Y. Rui,et al.  Learning to Rank Using User Clicks and Visual Features for Image Retrieval , 2015, IEEE Transactions on Cybernetics.

[21]  Farzin Yaghmaee,et al.  Application of GRNN neural network in non-texture image inpainting and restoration , 2015, Pattern Recognit. Lett..

[22]  Wanquan Liu,et al.  Fast algorithm for color texture image inpainting using the non-local CTV model , 2015, J. Glob. Optim..

[23]  Jong Chul Ye,et al.  Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting , 2015, IEEE Transactions on Image Processing.

[24]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[25]  Stefan Leutenegger,et al.  SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth , 2016, ArXiv.

[26]  Tuanfeng Zhang,et al.  Structure- and Texture-Based Fullbore Image Reconstruction , 2017, Mathematical Geosciences.

[27]  Farzin Yaghmaee,et al.  Exemplar-based image inpainting using svd-based approximation matrix and multi-scale analysis , 2017, Multimedia Tools and Applications.

[28]  Yin-wei Wei,et al.  Domain-based structure-aware image inpainting , 2016, Signal Image Video Process..

[29]  Naokazu Yokoya,et al.  Diminished Reality Based on Image Inpainting Considering Background Geometry , 2016, IEEE Transactions on Visualization and Computer Graphics.

[30]  Omar ElHarrouss,et al.  Moving object detection using a background modeling based on entropy theory and quad-tree decomposition , 2016, J. Electronic Imaging.

[31]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Mårten Sjöström,et al.  Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting , 2016, EURASIP J. Image Video Process..

[33]  Zhen Yang,et al.  Image inpainting algorithm based on TV model and evolutionary algorithm , 2014, Soft Computing - A Fusion of Foundations, Methodologies and Applications.

[34]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Guijin Wang,et al.  High-Resolution Image Inpainting through Multiple Deep Networks , 2017, 2017 International Conference on Vision, Image and Signal Processing (ICVISP).

[36]  Deng Cai,et al.  Depth Image Inpainting: Improving Low Rank Matrix Completion With Low Gradient Regularization , 2017, IEEE Transactions on Image Processing.

[37]  Rajesh Pandurang Borole,et al.  Image Restoration using Prioritized Exemplar Inpainting with Automatic Patch Optimization , 2017 .

[38]  Qian Fan,et al.  A novel patch matching algorithm for exemplar-based image inpainting , 2018, Multimedia Tools and Applications.

[39]  Laurent Hoeltgen Understanding image inpainting with the help of the Helmholtz equation , 2017 .

[40]  Jianping Fan,et al.  iPrivacy: Image Privacy Protection by Identifying Sensitive Objects via Deep Multi-Task Learning , 2017, IEEE Transactions on Information Forensics and Security.

[41]  Fei Gao,et al.  Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking , 2017, IEEE Transactions on Cybernetics.

[42]  Keiji Yanai,et al.  Scene Text Eraser , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[43]  Hiroshi Ishikawa,et al.  Joint Gap Detection and Inpainting of Line Drawings , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Minghui Zhang,et al.  Gradient-based low rank method and its application in image inpainting , 2018, Multimedia Tools and Applications.

[45]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Weilan Wang,et al.  Damaged region filling and evaluation by symmetrical exemplar-based image inpainting for Thangka , 2017, EURASIP J. Image Video Process..

[47]  Jiwu Huang,et al.  Localization of Diffusion-Based Inpainting in Digital Images , 2017, IEEE Transactions on Information Forensics and Security.

[48]  Jamal Riffi,et al.  A block-based background model for moving object detection , 2017 .

[49]  Xingming Sun,et al.  A robust forgery detection algorithm for object removal by exemplar-based image inpainting , 2017, Multimedia Tools and Applications.

[50]  Lei Wang,et al.  A deep network architecture for image inpainting , 2017, 2017 3rd IEEE International Conference on Computer and Communications (ICCC).

[51]  Ian D. Reid,et al.  Just-in-Time Reconstruction: Inpainting Sparse Maps Using Single View Depth Predictors as Priors , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[52]  2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD) , 2018 .

[53]  Hazem Abbas,et al.  Semantic Image Inpainting Vsing Self-Learning Encoder-Decoder and Adversarial Loss , 2018, 2018 13th International Conference on Computer Engineering and Systems (ICCES).

[54]  Yucai Zhou,et al.  The research of image inpainting algorithm using self-adaptive group structure and sparse representation , 2019, Cluster Computing.

[55]  Jun Yu,et al.  Local Deep-Feature Alignment for Unsupervised Dimension Reduction , 2018, IEEE Transactions on Image Processing.

[56]  Omar ElHarrouss,et al.  Moving object detection zone using a block-based background model , 2018, IET Comput. Vis..

[57]  Bin Song,et al.  Semantic object removal with convolutional neural network feature-based inpainting approach , 2018, Multimedia Systems.

[58]  Jianwu Li,et al.  Occluded offline handwritten Chinese character recognition using deep convolutional generative adversarial network and improved GoogLeNet , 2018, Neural Computing and Applications.

[59]  Haifeng Hu,et al.  An Improved Method for Semantic Image Inpainting with GANs: Progressive Inpainting , 2018, Neural Processing Letters.

[60]  Honghai Li,et al.  Driving style estimation by fusing multiple driving behaviors: a case study of freeway in China , 2018, Cluster Computing.

[61]  Zhiqiang Chen,et al.  Unsupervised Learnable Sinogram Inpainting Network (SIN) for Limited Angle CT reconstruction. , 2018, 1811.03911.

[62]  Omar ElHarrouss,et al.  Visual object tracking via the local soft cosine similarity , 2018, Pattern Recognit. Lett..

[63]  Dacheng Tao,et al.  Perceptual Adversarial Networks for Image-to-Image Transformation , 2017, IEEE Transactions on Image Processing.

[64]  Xiao Jin,et al.  Sparsity-Based Image Inpainting Detection via Canonical Correlation Analysis With Low-Rank Constraints , 2018, IEEE Access.

[65]  Shiguang Shan,et al.  Shift-Net: Image Inpainting via Deep Feature Rearrangement , 2018, ECCV.

[66]  Yilong Yin,et al.  Patch-Based Image Inpainting via Two-Stage Low Rank Approximation , 2018, IEEE Transactions on Visualization and Computer Graphics.

[67]  Shuai Yang,et al.  Structure-Guided Image Inpainting Using Homography Transformation , 2018, IEEE Transactions on Multimedia.

[68]  Fan Yao,et al.  Damaged region filling by improved criminisi image inpainting algorithm for thangka , 2018, Cluster Computing.

[69]  Kosuke Sato,et al.  Mask Optimization for Image Inpainting , 2018, IEEE Access.

[70]  Lu Leng,et al.  Image Inpainting Algorithm Based on Saliency Map and Gray Entropy , 2018, Arabian Journal for Science and Engineering.

[71]  Xianfeng Zhao,et al.  A deep learning approach to patch-based image inpainting forensics , 2018, Signal Process. Image Commun..

[72]  Cheng Wang,et al.  Preliminary Investigation on Single Remote Sensing Image Inpainting Through a Modified GAN , 2018, 2018 10th IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS).

[73]  Hao Wu,et al.  New Inpainting Algorithm Based on Simplified Context Encoders and Multi-Scale Adversarial Network , 2018, IIKI.

[74]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[75]  Kotagiri Ramamohanarao,et al.  Generative Image Inpainting with Submanifold Alignment , 2019, IJCAI.

[76]  Nassir Navab,et al.  Peeking Behind Objects: Layered Depth Prediction from a Single Image , 2018, Pattern Recognit. Lett..

[77]  Coloma Ballester,et al.  Semantic Image Inpainting Through Improved Wasserstein Generative Adversarial Networks , 2018, VISIGRAPP.

[78]  Omar ElHarrouss,et al.  Application of MEEMD in post-processing of dimensionality reduction methods for face recognition , 2019, IET Biom..

[79]  Sekhar Mandal,et al.  Multiple Pyramids Based Image Inpainting Using Local Patch Statistics and Steering Kernel Feature , 2019, IEEE Transactions on Image Processing.

[80]  Jun Yu,et al.  Multimodal Face-Pose Estimation With Multitask Manifold Deep Learning , 2019, IEEE Transactions on Industrial Informatics.

[81]  S. Srinivas Kumar,et al.  Image Inpainting Based on Fractional-Order Nonlinear Diffusion for Image Reconstruction , 2019, Circuits Syst. Signal Process..

[82]  Ang Li,et al.  Boosted GAN with Semantically Interpretable Information for Image Inpainting , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[83]  Samad Wali,et al.  A new adaptive boosting total generalized variation (TGV) technique for image denoising and inpainting , 2019, J. Vis. Commun. Image Represent..

[84]  Winston H. Hsu,et al.  VORNet: Spatio-Temporally Consistent Video Inpainting for Object Removal , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[85]  Jeffrey J. Rodríguez,et al.  Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering , 2019, IEEE Transactions on Image Processing.

[86]  Hao Wu,et al.  Multi-scale semantic image inpainting with residual learning and GAN , 2019, Neurocomputing.

[87]  Sung-Jea Ko,et al.  PEPSI : Fast Image Inpainting With Parallel Decoding Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Larry S. Davis,et al.  FiNet: Compatible and Diverse Fashion Image Inpainting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[89]  Bin Yang,et al.  Adversarial Inpainting of Medical Image Modalities , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[90]  Wei Xiong,et al.  Foreground-Aware Image Inpainting , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[91]  Abdelhak Belhi,et al.  Towards an Inpainting Framework for Visual Cultural Heritage , 2019, 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT).

[92]  Soo-Chang Pei,et al.  Image Inpainting For Random Areas Using Dense Context Features , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[93]  Omar ElHarrouss,et al.  Video Summarization based on Motion Detection for Surveillance Systems , 2019, 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC).

[94]  Scott Cohen,et al.  Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[95]  Bin Jiang,et al.  Coherent Semantic Attention for Image Inpainting , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[96]  A Novel Approach for Robust Multi Human Action Detection and Recognition based on 3-Dimentional Convolutional Neural Networks , 2019, ArXiv.

[97]  Jing Gao,et al.  An Image Inpainting Method for Interleaved 3D Stacked Image Sensor , 2019, IEEE Sensors Journal.

[98]  Haifeng Hu,et al.  Image Inpainting Based on Patch-GANs , 2019, IEEE Access.

[99]  Daniel Cohen-Or,et al.  Blind Visual Motif Removal From a Single Image , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[100]  Tao Yu,et al.  Interleaved Zooming Network for Image Inpainting , 2019, 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[101]  Jing Xiao,et al.  Artist-Net: Decorating the Inferred Content With Unified Style for Image Inpainting , 2019, IEEE Access.

[102]  Larry S. Davis,et al.  Compatible and Diverse Fashion Image Inpainting , 2019, ArXiv.

[103]  Baining Guo,et al.  Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[104]  Jon Y. Hardeberg,et al.  Deep Hyperspectral Prior: Denoising, Inpainting, Super-Resolution , 2019, ArXiv.

[105]  Sen Liu,et al.  Progressive Image Inpainting with Full-Resolution Residual Network , 2019, ACM Multimedia.

[106]  Jing Ke,et al.  Noise reduction with image inpainting: an application in clinical data diagnosis , 2019, SIGGRAPH Posters.

[107]  Lei Wang,et al.  Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation , 2019, IJCAI.

[108]  Mehran Ebrahimi,et al.  EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning , 2019, ArXiv.

[109]  Junyu Dong,et al.  Inpainting of Remote Sensing SST Images With Deep Convolutional Generative Adversarial Network , 2019, IEEE Geoscience and Remote Sensing Letters.

[110]  Yi Jiang,et al.  Image Inpainting Based on Generative Adversarial Networks , 2020, IEEE Access.

[111]  Qingming Huang,et al.  Spatial Pyramid-Enhanced NetVLAD With Weighted Triplet Loss for Place Recognition , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[112]  Sung-Jea Ko,et al.  PEPSI++: Fast and Lightweight Network for Image Inpainting , 2019, IEEE Transactions on Neural Networks and Learning Systems.