Automatic Facial Attractiveness Prediction by Deep Multi-Task Learning

Facial Attractiveness Prediction (FAP) is a useful yet challenging problem in the domain of computer vision. In this paper, we propose a deep learning based approach. Different from the existing deep methods, the proposed one models both the texture and shape clues within a multi-task learning framework consisting of attractiveness score prediction and fiducial landmark localization, thus highlighting both of their roles in assessing attractiveness of faces. Considering that the training data are not extensive, a lightweight CNN is designed to jointly learn the facial representation, landmark location, and facial attractiveness score. The proposed method is evaluated on the SCUT-FBP database, and a prediction correlation 0.92, is delivered, which shows the effectiveness of our method. Furthermore, two additional experiments in terms of comparison between facial images before and after make-up or beautification are conducted. The results also prove the advantage of the proposed method.

[1]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[2]  Jian Sun,et al.  Joint Cascade Face Detection and Alignment , 2014, ECCV.

[3]  Bo Li,et al.  Facial attractiveness computation by label distribution learning with deep CNN and geometric features , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[4]  Daniel Cohen-Or,et al.  A Humanlike Predictor of Facial Attractiveness , 2006, NIPS.

[5]  Hatice Gunes,et al.  Automatic analysis of facial attractiveness from video , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[6]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[8]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[9]  Jintu Fan,et al.  Prediction of facial attractiveness from facial proportions , 2012, Pattern Recognit..

[10]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[11]  David Zhang,et al.  Quantitative analysis of human facial beauty using geometric features , 2011, Pattern Recognit..

[12]  D. Perrett,et al.  Facial Skin Coloration Affects Perceived Health of Human Faces , 2009, International Journal of Primatology.

[13]  Bo Li,et al.  Label Distribution-Based Facial Attractiveness Computation by Deep Residual Learning , 2016, IEEE Transactions on Multimedia.

[14]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Lianwen Jin,et al.  Automatic classification of Chinese female facial beauty using Support Vector Machine , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[16]  Ming Shao,et al.  Attractive or Not?: Beauty Prediction with Attractiveness-Aware Encoders and Robust Late Fusion , 2014, ACM Multimedia.

[17]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Haibin Yan,et al.  Cost-sensitive ordinal regression for fully automatic facial beauty assessment , 2014, Neurocomputing.

[20]  Anita P. Barbee,et al.  "Their ideas of beauty are, on the whole, the same as ours": Consistency and variability in the cross-cultural perception of female physical attractiveness. , 1995 .

[21]  Yinhua Liu,et al.  Deep self-taught learning for facial beauty prediction , 2014, Neurocomputing.

[22]  Jie Xu,et al.  SCUT-FBP: A Benchmark Dataset for Facial Beauty Perception , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[23]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[24]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[25]  Jie Xu,et al.  A new humanlike facial attractiveness predictor with cascaded fine-tuning deep learning model , 2015, ArXiv.

[26]  Jie Xu,et al.  Facial attractiveness prediction using psychologically inspired convolutional neural network (PI-CNN) , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Wen-Chung Chiang,et al.  The cluster assessment of facial attractiveness using fuzzy neural network classifier based on 3D Moiré features , 2014, Pattern Recognit..

[29]  Carlos D. Castillo,et al.  An All-In-One Convolutional Neural Network for Face Analysis , 2016, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[30]  David Zhang,et al.  Facial beauty analysis based on geometric feature: Toward attractiveness assessment application , 2017, Expert Syst. Appl..

[31]  Serge J. Belongie,et al.  Relative ranking of facial attractiveness , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[32]  Bingbing Ni,et al.  Sense beauty via face, dressing, and/or voice , 2012, ACM Multimedia.

[33]  Lianwen Jin,et al.  A New Face Relighting Method Based on Edge-Preserving Filter , 2013, IEICE Trans. Inf. Syst..