Single-shot Semantic Image Inpainting with Densely Connected Generative Networks

Semantic image inpainting - a task to speculate and fill in large missing areas of a natural image, has shown exciting progress with the introduction of generative adversarial networks (GANs). But due to lack of sufficient understanding of semantic and spatial context, existing methods easily generate blurred boundary and distorted structure, which are inconsistent with the surrounding area. In this paper, we propose a new end-to-end framework named Single-shot Densely Connected Generative Network (SSDCGN), which generates visually realistic and semantically distinct pixels for the missing content by a battery of symmetric encoder-decoder groups. To maximize semantic extraction and realize precise spatial context localization, we involve a deeper densely skip connection in our network. Extensive experiments on Paris StreetView and ImageNet datasets show the superiority of our method.

[1]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[3]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[4]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[5]  Meng Wang,et al.  Semantic Image Inpainting with Progressive Generative Networks , 2018, ACM Multimedia.

[6]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[7]  Joachim Weickert,et al.  Coherence-Enhancing Diffusion Filtering , 1999, International Journal of Computer Vision.

[8]  Shih-Fu Chang,et al.  Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification , 2017, IEEE Transactions on Multimedia.

[9]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[10]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[11]  Alexei A. Efros,et al.  What makes Paris look like Paris? , 2015, Commun. ACM.

[12]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[14]  Zongben Xu,et al.  Image Inpainting by Patch Propagation Using Patch Sparsity , 2010, IEEE Transactions on Image Processing.

[15]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[17]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[18]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[19]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[20]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[21]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[22]  Wei Liu,et al.  DeepProduct: Mobile Product Search With Portable Deep Features , 2018, ACM Trans. Multim. Comput. Commun. Appl..

[23]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[25]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[26]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[27]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[28]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[29]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[31]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..