Photo-Realistic and Robust Inpainting of Faces Using Refinement GANs

Image inpainting faces many different challenges when filling holes or restoring damaged pixels. One of the most difficult tasks is inpainting central and eye-catching parts, where already small artifacts are fatal. Such a challenge is inpainting missing parts of human faces in portrait photographs. In this paper we propose, design and evaluate a refinement generative adversarial network (GAN), that we apply within an inpainting pipeline to robustly generate photo-realistic restorations. The key aspects of our refiner are the global consensus and the inpainting loss, that ensure close generated and damaged images. Furthermore, the content losses lead to a reliable prediction of the perception score, that examines the generated images upon their authenticity. We show typical problems of state-of-the-art semantic inpainting that omits such a global consensus loss and suffers from visually disruptive artifacts such as inpainting a mustache in a female face or inpainting disagreeing skin or eye color. We validate our method on the CelebA dataset and report four different traditional quality metrics using six different hole patterns. We conclude, that refinement considerably improves for each setting the numerical and visual quality and preserves already good restorations.