Real-time pose invariant spontaneous smile detection using conditional random regression forests

Abstract Detecting spontaneous smile in unconstrained environment is a challenging problem mainly due to the large intra-class variations caused by head poses. This paper presents a real-time smile detection method based on conditional random regression forests. Since the relation between image patches and smile intensity is modelled conditional to head pose, the proposed smile detection method is not sensitive to head poses. To achieve high smile detection performance, techniques including regression forest, multiple-label dataset augmentation and non-informative patch removement are employed. Experimental results show that the proposed method achieves competitive performance to state-of-the-art deep neural network based methods on two challenging real-world datasets, although using hand-crafted features. A dynamical forest ensemble scheme is also presented to make a trade-off between smile detection performance and processing speed. In contrast to deep neural networks, the proposed method can run in real-time on general hardware without GPU.

[1]  Tae-Kyun Kim,et al.  Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Junjun Jiang,et al.  Locality Preserving Matching , 2018, International Journal of Computer Vision.

[3]  Bir Bhanu,et al.  Efficient smile detection by Extreme Learning Machine , 2015, Neurocomputing.

[4]  Hong Liu,et al.  Smile detection in unconstrained scenarios using self-similarity of gradients features , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[5]  Kun Zhang,et al.  Robust head pose estimation using Dirichlet-tree distribution enhanced random forests , 2016, Neurocomputing.

[6]  Tal Hassner,et al.  Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns , 2015, ICMI.

[7]  Yuanyuan Liu,et al.  Spontaneous Smile Recognition for Interest Detection , 2016, CCPR.

[8]  Jingying Chen,et al.  A low‐cost real‐time face tracking system for ITSs and SDASs , 2017, Softw. Pract. Exp..

[9]  Min Sun,et al.  Conditional regression forests for human pose estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Yizong Cheng,et al.  Mean Shift, Mode Seeking, and Clustering , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Caifeng Shan,et al.  Smile detection by boosting pixel differences , 2012, IEEE Transactions on Image Processing.

[12]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[13]  Luc Van Gool,et al.  Real time head pose estimation with random regression forests , 2011, CVPR 2011.

[14]  Hong Liu,et al.  A new descriptor of gradients Self-Similarity for smile detection in unconstrained scenarios , 2016, Neurocomputing.

[15]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Zheru Chi,et al.  Smile detection in the wild with deep convolutional neural networks , 2017, Machine Vision and Applications.

[17]  Mohammad H. Mahoor,et al.  Going deeper in facial expression recognition using deep neural networks , 2015, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[18]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[19]  Wenming Zheng,et al.  Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression , 2014, IEEE Transactions on Affective Computing.

[20]  Ji Zhao,et al.  Non-rigid visible and infrared face registration via regularized Gaussian fields criterion , 2015, Pattern Recognit..

[21]  Shuicheng Yan,et al.  Conditional Convolutional Neural Network for Modality-Aware Face Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Kun Zhang,et al.  A hybrid intelligence-aided approach to affect-sensitive e-learning , 2014, Computing.

[23]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[24]  Vincent Lepetit,et al.  Pose-specific non-linear mappings in feature space towards multiview facial expression recognition , 2017, Image Vis. Comput..

[25]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Matti Pietikäinen,et al.  Local Binary Patterns , 2010, Scholarpedia.

[27]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[28]  Rama Chellappa,et al.  FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition , 2016, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[29]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[30]  Xiaohui Yuan,et al.  Conditional convolution neural network enhanced random forest for facial expression recognition , 2018, Pattern Recognit..

[31]  Guang-Bin Huang,et al.  Smile detection using Pair-wise Distance Vector and Extreme Learning Machine , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[32]  Peter Kontschieder,et al.  Deep Neural Decision Forests , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[34]  Xin Geng,et al.  Head Pose Estimation Based on Multivariate Label Distribution , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Fernando De la Torre,et al.  Estimating smile intensity: A better way , 2015, Pattern Recognit. Lett..

[36]  Luc Van Gool,et al.  Real-time facial feature detection using conditional regression forests , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Jingying Chen,et al.  Deep peak-neutral difference feature for facial expression recognition , 2018, Multimedia Tools and Applications.

[38]  Zhuowen Tu,et al.  Robust Point Matching via Vector Field Consensus , 2014, IEEE Transactions on Image Processing.

[39]  Guang-Bin Huang,et al.  ELM based smile detection using Distance Vector , 2018, Pattern Recognit..

[40]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[41]  Alan L. Yuille,et al.  Semi-Supervised Sparse Representation Based Classification for Face Recognition With Insufficient Labeled Samples , 2016, IEEE Transactions on Image Processing.

[42]  Tong Zhang,et al.  A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition , 2016, IEEE Transactions on Multimedia.

[43]  Rana El Kaliouby,et al.  Smile or smirk? Automatic detection of spontaneous asymmetric smiles to understand viewer experience , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).