2M BeautyNet: Facial Beauty Prediction Based on Multi-Task Transfer Learning

Facial beauty prediction (FBP) has become an emerging area in the field of artificial intelligence. However, the lacks of data and accurate face representation hinder the development of FBP. Multi-task transfer learning can effectively avoid over-fitting, and utilize auxiliary information of related tasks to optimize the main task. In this paper, we present a network named Multi-input Multi-task Beauty Network (2M BeautyNet) and use transfer learning to predict facial beauty. In the experiment, beauty prediction is the main task, and gender recognition is the auxiliary. For multi-task training, we employ multi-task loss weights automatic learning strategy to improve the performance of FBP. Finally, we replace the softmax classifier with a random forest. We conduct experiments on the Large Scale Facial Beauty Database (LSFBD) and SCUT-FBP5500 database. Results show that our method has achieved good results on LSFBD, the accuracy of FBP is up to 68.23%. Our 2M BeautyNet structure is suitable for multiple inputs of different databases.

[1]  Bo Li,et al.  Facial attractiveness computation by label distribution learning with deep CNN and geometric features , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Xiaohui Yuan,et al.  Transferring Rich Deep Features for Facial Beauty Prediction , 2018, ArXiv.

[4]  Xin Guo,et al.  Smile Detection in the Wild Based on Transfer Learning , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[5]  Jie Xu,et al.  Facial attractiveness prediction using psychologically inspired convolutional neural network (PI-CNN) , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Yu Huang,et al.  Benchmark of a large scale database for facial beauty prediction , 2016, ICIIP.

[7]  Yun Fu,et al.  Rule-Based Facial Makeup Recommendation System , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[8]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Fei Gao,et al.  Improving Facial Attractiveness Prediction via Co-attention Learning , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Yunhong Wang,et al.  Automatic Facial Attractiveness Prediction by Deep Multi-Task Learning , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[11]  Tieniu Tan,et al.  A Light CNN for Deep Face Representation With Noisy Labels , 2015, IEEE Transactions on Information Forensics and Security.

[12]  J. Langlois,et al.  Attractive Faces Are Only Average , 1990 .

[13]  Hatice Gunes,et al.  Assessing facial beauty through proportion analysis by image processing and supervised learning , 2006, Int. J. Hum. Comput. Stud..

[14]  Eytan Ruppin,et al.  Facial Attractiveness: Beauty and the Machine , 2006, Neural Computation.

[15]  Trevor Cohn,et al.  Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser , 2015, ACL.

[16]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[17]  Yi Ren,et al.  Sense Beauty by Label Distribution Learning , 2017, IJCAI.

[18]  D. Perrett,et al.  Effects of sexual dimorphism on facial attractiveness , 1998, Nature.

[19]  Andrea Bottino,et al.  Computer analysis of face beauty: A survey , 2014, Comput. Vis. Image Underst..

[20]  Lu Xu,et al.  Hierarchical Multi-Task Network For Race, Gender and Facial Attractiveness Recognition , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[21]  Can Yang,et al.  Unsupervised Cross-Dataset Person Re-identification by Transfer Learning of Spatial-Temporal Patterns , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[23]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yihong Gong,et al.  Predicting Facial Beauty without Landmarks , 2010, ECCV.

[25]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Han Yu,et al.  Multi-Label Metric Transfer Learning Jointly Considering Instance Space and Label Space Distribution Divergence , 2019, IEEE Access.

[27]  Luc Van Gool,et al.  Some Like It Hot — Visual Guidance for Preference Prediction , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[29]  Yinhua Liu,et al.  Deep self-taught learning for facial beauty prediction , 2014, Neurocomputing.

[30]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Qiang Yang,et al.  Distant Domain Transfer Learning , 2017, AAAI.

[32]  Ran R. Hassin,et al.  Facing faces: studies on the cognitive aspects of physiognomy. , 2000, Journal of personality and social psychology.

[33]  Jonathan Baxter,et al.  A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling , 1997, Machine Learning.

[34]  I. Cuthill,et al.  Asymmetry and human facial attractiveness: symmetry may not always be beautiful , 1995, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[35]  Xiaohui Yuan,et al.  Multi-level structured hybrid forest for joint head detection and pose estimation , 2017, Neurocomputing.

[36]  Luc Van Gool,et al.  Fast Scene Understanding for Autonomous Driving , 2017, ArXiv.

[37]  He Cao,et al.  BeautyNet: Joint Multiscale CNN and Transfer Learning Method for Unconstrained Facial Beauty Prediction , 2019, Comput. Intell. Neurosci..

[38]  Jie Xu,et al.  A new humanlike facial attractiveness predictor with cascaded fine-tuning deep learning model , 2015, ArXiv.

[39]  Lianwen Jin,et al.  Attribute-Aware Convolutional Neural Networks for Facial Beauty Prediction , 2019, IJCAI.

[40]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[41]  Walid Barhoumi,et al.  An image-based segmentation recommender using crowdsourcing and transfer learning for skin lesion extraction , 2019, Expert Syst. Appl..

[42]  Feng Liu,et al.  Unsupervised Heterogeneous Domain Adaptation via Shared Fuzzy Equivalence Relations , 2018, IEEE Transactions on Fuzzy Systems.

[43]  Lianwen Jin,et al.  SCUT-FBP5500: A Diverse Benchmark Dataset for Multi-Paradigm Facial Beauty Prediction , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[44]  Xuelong Li,et al.  Facial Skin Beautification Using Adaptive Region-Aware Masks , 2014, IEEE Transactions on Cybernetics.

[45]  Vladlen Koltun,et al.  Multi-Task Learning as Multi-Objective Optimization , 2018, NeurIPS.

[46]  Li Fei-Fei,et al.  Dynamic Task Prioritization for Multitask Learning , 2018, ECCV.

[47]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[48]  Lianwen Jin,et al.  Regression Guided by Relative Ranking Using Convolutional Neural Network (R$^3$3CNN) for Facial Beauty Prediction , 2019, IEEE Trans. Affect. Comput..

[49]  Tao Li,et al.  Understanding Beauty via Deep Facial Features , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[50]  Iasonas Kokkinos,et al.  UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).