Regression Guided by Relative Ranking Using Convolutional Neural Network (R$^3$3CNN) for Facial Beauty Prediction

Facial beauty prediction (FBP) aims to assess facial attractiveness with judgements based on human perception. Most of previous methods formulate FBP as a classification, regression or ranking problem of machine learning. However, humans not only represent facial attractiveness as a score, but also perceive the relative aesthetics of faces. Inspired by this observation, we formulate FBP as a specific regression problem guided by ranking information. Specifically, we propose a general CNN architecture, called R $^3$ CNN, to integrate the relative ranking of faces in terms of aesthetics to improve performance of facial beauty prediction. As R $^3$ CNN consists of both regression and ranking components, it is challenging to train and fine-tune it by existing techniques. To tackle this problem, we propose the following learning schemes for R $^3$ CNN: 1) hard pair sampling that generates challenging-to-predicted image pairs and pseudo ranking labels from true rating scores; 2) an assemble loss function that combines regression loss and pairwise ranking loss (PR-Loss); 3) a cascaded fine-tuning method that further improves prediction. Moreover, we build a benchmark dataset, called SCUT-FBP5500, containing 5,500 images of faces with diverse properties and labels. Experiments were performed on both the SCUT-FBP and the SCUT-FBP5500 benchmark datasets, where our method achieved state-of-the-art performance.