Face inpainting based on high-level facial attributes

A novel approach is proposed for facial inpainting based on a guidance face.The guidance face matches with the high-level facial attributes of the target face.The facial attributes considered include gender, ethnicity, expression and skin tone.Inpainting is done on intrinsic image layers to cope with illumination differences.An MRF optimization is applied to effectively solve the proposed problem. Our approach selects a guidance face that matches the targeted attributes and utilizes it together with the observable input face regions to inpaint the missing areas. Sources of information are balanced using an adaptive optimization, and the inpainting is performed on the intrinsic image layers instead of the RGB color space to handle the illumination dierences between the target face and the guidance face to further enhance the resulting visual quality. Display Omitted We introduce a novel data-driven approach for face inpainting, which makes use of the observable region of an occluded face as well as its inferred high-level facial attributes, namely gender, ethnicity, and expression. Based on the intuition that the realism of a face inpainting result depends significantly on its overall consistency with respect to these high-level attributes, our approach selects a guidance face that matches the targeted attributes and utilizes it together with the observable input face regions to inpaint the missing areas. These two sources of information are balanced using an adaptive optimization, and the inpainting is performed on the intrinsic image layers instead of the RGB color space to handle the illumination differences between the target face and the guidance face to further enhance the resulting visual quality. Our experiments demonstrate that this approach is effective in inpainting facial components such as the mouth or the eyes that could be partially or completely occluded in the input face. A perceptual study shows that our approach generates more natural facial appearances by accounting for high-level facial attributes.

[1]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[2]  Narendra Ahuja,et al.  Image completion using planar structure guidance , 2014, ACM Trans. Graph..

[3]  Terrance E. Boult,et al.  Multi-attribute spaces: Calibration for attribute fusion and similarity search , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Takashi Shibata,et al.  Image Inpainting Based on Probabilistic Structure Estimation , 2010, ACCV.

[5]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[6]  Harald Grossauer,et al.  A Combined PDE and Texture Synthesis Approach to Inpainting , 2004, ECCV.

[7]  Pietro Perona,et al.  Robust Face Landmark Estimation under Occlusion , 2013, 2013 IEEE International Conference on Computer Vision.

[8]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Ding Xing-hao.  Image inpainting based on 1 Norm , 2010 .

[10]  Xiangyu Zhu,et al.  High-fidelity Pose and Expression Normalization for face recognition in the wild , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Karthikeyan Umapathy,et al.  Audio Signal Processing Using Time-Frequency Approaches: Coding, Classification, Fingerprinting, and Watermarking , 2010, EURASIP J. Adv. Signal Process..

[12]  Hossein Mobahi,et al.  Data-driven image completion by image patch subspaces , 2009, 2009 Picture Coding Symposium.

[13]  Heloir,et al.  The Uncanny Valley , 2019, The Animation Studies Reader.

[14]  Ming Zhao,et al.  Image inpainting with salient structure completion and texture propagation , 2011, Pattern Recognit. Lett..

[15]  Nikos Komodakis,et al.  Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning , 2007, IEEE Transactions on Image Processing.

[16]  Vincent Lepetit,et al.  Pose-specific non-linear mappings in feature space towards multiview facial expression recognition , 2017, Image Vis. Comput..

[17]  Xiaoou Tang,et al.  Image inpainting by global structure and texture propagation , 2007, ACM Multimedia.

[18]  Vitomir Struc,et al.  The Complete Gabor-Fisher Classifier for Robust Face Recognition , 2010, EURASIP J. Adv. Signal Process..

[19]  Shailendra Mishra,et al.  Review of Different Inpainting Algorithms , 2012 .

[20]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[21]  A. O'Toole,et al.  Recognizing people from dynamic and static faces and bodies: Dissecting identity with a fusion approach , 2010 .

[22]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[23]  Steven M. Drucker,et al.  Quality prediction for image completion , 2012, ACM Trans. Graph..

[24]  John P. Lewis,et al.  Face Inpainting with Local Linear Representations , 2004, BMVC.

[25]  N Iss.  Region Filling and Object Removal by Exemplar- Based Image Inpainting , 2012 .

[26]  Terrance E. Boult,et al.  MOON: A Mixed Objective Optimization Network for the Recognition of Facial Attributes , 2016, ECCV.

[27]  Seong-Whan Lee,et al.  Reconstruction of Partially Damaged Face Images Based on a Morphable Face Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Marios Savvides,et al.  A preliminary investigation on the sensitivity of COTS face recognition systems to forensic analyst-style face processing for occlusions , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[30]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[31]  William A. P. Smith,et al.  Statistical 3D face shape estimation from occluding contours , 2016, Comput. Vis. Image Underst..

[32]  Fei Yang,et al.  Expression flow for 3D-aware face component transfer , 2011, ACM Trans. Graph..

[33]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[34]  大西 仁,et al.  Pearl, J. (1988, second printing 1991). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan-Kaufmann. , 1994 .

[35]  Guillermo Sapiro,et al.  A Variational Framework for Exemplar-Based Image Inpainting , 2011, International Journal of Computer Vision.

[36]  Christine Guillemot,et al.  Image Inpainting : Overview and Recent Advances , 2014, IEEE Signal Processing Magazine.

[37]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[38]  Xin Zheng,et al.  Image Completion with Automatic Structure Propagation , 2012, 2012 Eighth International Conference on Computational Intelligence and Security.

[39]  Karl F. MacDorman,et al.  The Uncanny Valley [From the Field] , 2012, IEEE Robotics Autom. Mag..

[40]  Stephen Lin,et al.  A Closed-Form Solution to Retinex with Nonlocal Texture Constraints , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Wang-Q Lim,et al.  Image inpainting: theoretical analysis and comparison of algorithms , 2013, Optics & Photonics - Optical Engineering + Applications.

[42]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[43]  Jin-Jang Leou,et al.  Image Inpainting Using Structure-Guided Priority Belief Propagation and Label Transformations , 2010, 2010 20th International Conference on Pattern Recognition.

[44]  Vicent Caselles,et al.  Exemplar-Based Image Inpainting Using Multiscale Graph Cuts , 2013, IEEE Transactions on Image Processing.

[45]  Yi-Chun Liao,et al.  Exemplar-based Image Inpainting base on Structure Construction , 2008, J. Softw..

[46]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[47]  Tal Hassner,et al.  Effective face frontalization in unconstrained images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Zongben Xu,et al.  Image Inpainting by Patch Propagation Using Patch Sparsity , 2010, IEEE Transactions on Image Processing.

[49]  Christine Guillemot,et al.  Examplar-based inpainting based on local geometry , 2011, 2011 18th IEEE International Conference on Image Processing.

[50]  Shree K. Nayar,et al.  FaceTracer: A Search Engine for Large Collections of Images with Faces , 2008, ECCV.

[51]  Yueting Zhuang,et al.  Patch-guided facial image inpainting by shape propagation , 2009 .

[52]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[53]  Fei Yang,et al.  Expression flow for 3D-aware face component transfer , 2011, SIGGRAPH 2011.

[54]  William J. Christmas,et al.  Pose-invariant face recognition by matching on multi-resolution MRFs linked by supercoupling transform , 2011, Comput. Vis. Image Underst..

[55]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[56]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[57]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2008, Commun. ACM.

[58]  David Salesin,et al.  Interactive digital photomontage , 2004, SIGGRAPH 2004.

[59]  Wojciech Matusik,et al.  Video face replacement , 2011, ACM Trans. Graph..

[60]  Jiaya Jia,et al.  Image completion with structure propagation , 2005, SIGGRAPH 2005.

[61]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[62]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[63]  D. McNeill.  The Face: A Natural History , 1998 .

[64]  Harry Shum,et al.  Automatic eyeglasses removal from face images , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.