FM2u-Net: Face Morphological Multi-Branch Network for Makeup-Invariant Face Verification

It is challenging in learning a makeup-invariant face verification model, due to (1) insufficient makeup/non-makeup face training pairs, (2) the lack of diverse makeup faces, and (3) the significant appearance changes caused by cosmetics. To address these challenges, we propose a unified Face Morphological Multi-branch Network (FMMu-Net) for makeup-invariant face verification, which can simultaneously synthesize many diverse makeup faces through face morphology network (FM-Net) and effectively learn cosmetics-robust face representations using attention-based multi-branch learning network (AttM-Net). For challenges (1) and (2), FM-Net (two stacked auto-encoders) can synthesize realistic makeup face images by transferring specific regions of cosmetics via cycle consistent loss. For challenge (3), AttM-Net, consisting of one global and three local (task-driven on two eyes and mouth) branches, can effectively capture the complementary holistic and detailed information. Unlike DeepID2 which uses simple concatenation fusion, we introduce a heuristic method AttM-FM, attached to AttM-Net, to adaptively weight the features of different branches guided by the holistic information. We conduct extensive experiments on makeup face verification benchmarks (M-501, M-203, and FAM) and general face recognition datasets (LFW and IJB-A). Our framework FMMu-Net achieves state-of-the-art performances.

[1]  Gérard G. Medioni,et al.  Pose-Aware Face Recognition in the Wild , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Shiguang Shan,et al.  AttGAN: Facial Attribute Editing by Only Changing What You Want , 2017, IEEE Transactions on Image Processing.

[3]  Chuang Gan,et al.  Recurrent Topic-Transition GAN for Visual Paragraph Generation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Ioannis A. Kakadiaris,et al.  Patch-based Face Recognition using a Hierarchical Multi-label Matcher , 2018, Image Vis. Comput..

[5]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[6]  Shuicheng Yan,et al.  "Wow! you are so beautiful today!" , 2013, MM '13.

[7]  Dong Guo,et al.  Digital face makeup by example , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Yun Fu,et al.  Rule-Based Facial Makeup Recommendation System , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[9]  Tao Xiang,et al.  Multi-scale Deep Learning Architectures for Person Re-identification , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Xiaogang Wang,et al.  AdaCos: Adaptively Scaling Cosine Logits for Effectively Learning Deep Face Representations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[12]  Chi-Ho Chan,et al.  Robust face recognition by an albedo based 3D morphable model , 2014, IEEE International Joint Conference on Biometrics.

[13]  Tal Hassner,et al.  Do We Really Need to Collect Millions of Faces for Effective Face Recognition? , 2016, ECCV.

[14]  Michael S. Brown,et al.  Example-Based Cosmetic Transfer , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[15]  Martial Hebert,et al.  Image Deformation Meta-Networks for One-Shot Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[17]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[18]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[20]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Stefanos Zafeiriou,et al.  UV-GAN: Adversarial Facial UV Map Completion for Pose-Invariant Face Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Liang Lin,et al.  BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network , 2018, ACM Multimedia.

[23]  Fang Zhao,et al.  Dual-Agent GANs for Photorealistic and Identity Preserving Profile Face Synthesis , 2017, NIPS.

[24]  Hans-Peter Seidel,et al.  Computer‐Suggested Facial Makeup , 2011, Comput. Graph. Forum.

[25]  Kyung-Yong Chung,et al.  Effect of facial makeup style recommendation on visual sensibility , 2014, Multimedia Tools and Applications.

[26]  Yi Li,et al.  Anti-Makeup: Learning A Bi-Level Adversarial Network for Makeup-Invariant Face Verification , 2017, AAAI.

[27]  Kai Hu,et al.  MSDNN: Multi-Scale Deep Neural Network for Salient Object Detection , 2018, ArXiv.

[28]  G. Rhodes,et al.  Are Average Facial Configurations Attractive Only Because of Their Symmetry? , 1999 .

[29]  Martial Hebert,et al.  Low-Shot Learning from Imaginary Data , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Chin-Chuan Han,et al.  Fast face detection via morphology-based pre-processing , 2000, Pattern Recognit..

[31]  Yi Yang,et al.  Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in Vitro , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  William J. Christmas,et al.  When Face Recognition Meets with Deep Learning: An Evaluation of Convolutional Neural Networks for Face Recognition , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[33]  A F Ayoub,et al.  Towards building a photo-realistic virtual human face for craniomaxillofacial diagnosis and treatment planning. , 2007, International journal of oral and maxillofacial surgery.

[34]  M. Nachman,et al.  Morphological and population genomic evidence that human faces have evolved to signal individual identity , 2014, Nature Communications.

[35]  Xiangyang Xue,et al.  Multi-task Deep Neural Network for Joint Face Recognition and Facial Attribute Prediction , 2017, ICMR.

[36]  Yongxin Yang,et al.  Frankenstein: Learning Deep Face Representations Using Small Data , 2016, IEEE Transactions on Image Processing.

[37]  Federica Marcolin,et al.  Geometry-based 3D face morphology analysis: soft-tissue landmark formalization , 2012, Multimedia Tools and Applications.

[38]  Kai Hu,et al.  Multi-scale deep neural network for salient object detection , 2018, IET Image Process..

[39]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[40]  Chi-Keung Tang,et al.  LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[42]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[43]  Ting-Zhu Huang,et al.  Patch-Based Principal Component Analysis for Face Recognition , 2017, Comput. Intell. Neurosci..

[44]  Yun Fu,et al.  Examples-Rules Guided Deep Neural Network for Makeup Recommendation , 2017, AAAI.

[45]  Wen Gao,et al.  Hierarchical Ensemble of Global and Local Classifiers for Face Recognition , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[46]  Jean-Marie Morvan,et al.  Improving Shadow Suppression for Illumination Robust Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Xiaogang Wang,et al.  Deep Learning Face Representation by Joint Identification-Verification , 2014, NIPS.

[48]  Xiaoming Liu,et al.  Disentangled Representation Learning GAN for Pose-Invariant Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Yongxin Yang,et al.  Attribute-Enhanced Face Recognition with Neural Tensor Fusion Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[50]  Xiaochun Cao,et al.  Makeup Like a Superstar: Deep Localized Makeup Transfer Network , 2016, IJCAI.

[51]  Anil K. Jain,et al.  Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[53]  Omkar M. Parkhi,et al.  VGGFace2: A Dataset for Recognising Faces across Pose and Age , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[54]  Tao Xiang,et al.  Pose-Normalized Image Generation for Person Re-identification , 2017, ECCV.

[55]  Shuicheng Yan,et al.  Face Authentication With Makeup Changes , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[56]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Li Bai,et al.  Cosine Similarity Metric Learning for Face Verification , 2010, ACCV.

[58]  Adam Finkelstein,et al.  PairedCycleGAN: Asymmetric Style Transfer for Applying and Removing Makeup , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[59]  Tal Hassner,et al.  Extreme 3D Face Reconstruction: Seeing Through Occlusions , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Hong-Han Shuai,et al.  BeautyGlow: On-Demand Makeup Transfer Framework With Reversible Generative Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Bin Liu,et al.  A weakly supervised method for makeup-invariant face verification , 2017, Pattern Recognit..

[62]  Tieniu Tan,et al.  A Light CNN for Deep Face Representation With Noisy Labels , 2015, IEEE Transactions on Information Forensics and Security.