Face analysis through semantic face segmentation

Abstract Automatic face analysis, including head pose estimation, gender recognition, and expression classification, strongly benefits from an accurate segmentation of the human face. In this paper we present a multi-feature framework which first segments a face image into six parts, and then performs classification tasks on head pose, gender, and expression. Segmentation is achieved by training a discriminative model on a manually labeled face database, namely FASSEG , which we extend from previous versions, and which we publicly share. Three kinds of features accounting for location, shape, and color are extracted from uniformly sampled square image patches. Facial images are then pixel-wise segmented into six semantic classes – hair, skin, nose, eyes, mouth, and background, – using a Random Forest classifier (RF). Then a linear Support Vector Machine (SVM) is trained for each face analysis task i.e., head pose estimation, gender recognition, and expression classification by using the probability maps obtained during the segmentation step. Performance of the proposed framework is evaluated on four face databases, namely Pointing’04, FEI, FERET, and MPI, with results which outperform the current state-of-the-art.

[1]  Trevor Darrell,et al.  PANDA: Pose Aligned Networks for Deep Attribute Modeling , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Ji Zheng,et al.  A support vector machine classifier with automatic confidence and its application to gender classification , 2011, Neurocomputing.

[3]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[4]  Jean-Marc Odobez,et al.  Joint Adaptive Colour Modelling and Skin, Hair and Clothes Segmentation using Coherent Probabilistic Index Maps , 2011, BMVC.

[5]  H. Ellis,et al.  Perceiving and remembering faces , 1983 .

[6]  Honglak Lee,et al.  Augmenting CRFs with Boltzmann Machine Shape Priors for Image Labeling , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[8]  D. Enlow.  Handbook of facial growth , 1975 .

[9]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[10]  Qingshan Liu,et al.  Recognizing expressions from face and body gesture by temporal normalized motion and appearance features , 2013, Image Vis. Comput..

[11]  Massimo Mauro,et al.  Data for: Face analysis through semantic face segmentation , 2019 .

[12]  Harry Wechsler,et al.  The FERET database and evaluation procedure for face-recognition algorithms , 1998, Image Vis. Comput..

[13]  Riccardo Leonardi,et al.  Hair detection, segmentation, and hairstyle classification in the wild , 2018, Image Vis. Comput..

[14]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[15]  Emmanuel Dellandréa,et al.  Affective Video Content Analysis: A Multidisciplinary Insight , 2018, IEEE Transactions on Affective Computing.

[16]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[17]  Adrian Nestor,et al.  The segmental structure of faces and its use in gender recognition. , 2008, Journal of vision.

[18]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Angelo Cangelosi,et al.  Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods , 2017, Pattern Recognit..

[20]  Pawan Sinha,et al.  Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About , 2006, Proceedings of the IEEE.

[21]  Rama Chellappa,et al.  Growing Regression Forests by Classification: Applications to Object Pose Estimation , 2013, ECCV.

[22]  José Miguel Buenaposada,et al.  Revisiting Linear Discriminant Techniques in Gender Recognition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Larry S. Davis,et al.  Detection and analysis of hair , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Riccardo Leonardi,et al.  Shot scale distribution in art films , 2016, Multimedia Tools and Applications.

[25]  James L. Crowley,et al.  Head Pose Estimation on Low Resolution Images , 2006, CLEAR.

[26]  Carlos E. Thomaz,et al.  A Priori-Driven PCA , 2012, ACCV Workshops.

[27]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[28]  Christopher K. I. Williams,et al.  The Shape Boltzmann Machine: A Strong Model of Object Shape , 2012, International Journal of Computer Vision.

[29]  Shiguang Shan,et al.  CovGa: A novel descriptor based on symmetry of regions for head pose estimation , 2014, Neurocomputing.

[30]  Massimo Mauro,et al.  Head pose estimation through multi-class face segmentation , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[31]  Erik G. Learned-Miller,et al.  Towards unconstrained face recognition , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[32]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  James L. Crowley,et al.  Head Pose Estimation Using Multi-scale Gaussian Derivatives , 2013, SCIA.

[34]  Pritee Khanna,et al.  An illumination, expression, and noise invariant gender classifier using two-directional 2DPCA on real Gabor space , 2015, J. Comput. Lang..

[35]  Yuxiao Hu,et al.  Evaluation of Head Pose Estimation for Studio Data , 2006, CLEAR.

[36]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[37]  Roope Raisamo,et al.  An experimental comparison of gender classification methods , 2008, Pattern Recognit. Lett..

[38]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[39]  Aurobinda Routray,et al.  Automatic facial expression recognition using features of salient facial patches , 2015, IEEE Transactions on Affective Computing.

[40]  Bingpeng Ma,et al.  VoD: A novel image representation for head yaw estimation , 2015, Neurocomputing.

[41]  Dario Maio,et al.  A multi-classifier approach to face image segmentation for travel documents , 2012, Expert Syst. Appl..

[42]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Takeshi Saitoh,et al.  Head Pose Estimation Using Convolutional Neural Network , 2018 .

[44]  Luiz Eduardo Soares de Oliveira,et al.  Fusion of feature sets and classifiers for facial expression recognition , 2013, Expert Syst. Appl..

[45]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.