Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns

3D facial expression recognition has been greatly promoted for overcoming the inherent drawbacks of 2D facial expression recognition and has achieved superior recognition accuracy to the 2D. In this paper, a novel holistic, full-automatic approach for 3D facial expression recognition is proposed. First, 3D face models are represented in 2D-image-like structure which makes it possible to take advantage of the wealth of 2D methods to analyze 3D models. Then an enhanced facial representation, namely polytypic multi-block local binary patterns (P-MLBP), is proposed. The P-MLBP involves both the feature-based irregular divisions to depict the facial expressions accurately and the fusion of depth and texture information of 3D models to enhance the facial feature. Based on the BU-3DFE database, three kinds of classifiers are employed to conduct 3D facial expression recognition for evaluation. Their experimental results outperform the state of the art and show the effectiveness of P-MLBP for 3D facial expression recognition. Therefore, the proposed strategy is validated for 3D facial expression recognition; and its simplicity opens a promising direction for fully automatic 3D facial expression recognition. 2D-image-like structure is proposed to represent 3D models.2D-image-like structure builds the bridge between 3D models and 2D methods.Irregular divisions and data fusion are utilized in P-MLBP to enhance facial feature.Proposed P-MLBP represented facial feature achieves preferable recognition results.The proposed approach is effective to automatic 3D facial expression recognition.

[1]  Patrick J. Flynn,et al.  A Region Ensemble for 3-D Face Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[2]  Wei Zeng,et al.  An automatic 3D expression recognition framework based on sparse representation of conformal images , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[3]  Ruchir Srivastava,et al.  3D facial expression recognition using residues , 2009, TENCON 2009 - 2009 IEEE Region 10 Conference.

[4]  Hasan Demirel,et al.  Optimal feature selection for 3D facial expression recognition using coarse-to-fine classification , 2010 .

[5]  Ioannis A. Kakadiaris,et al.  3D/4D facial expression analysis: An advanced annotated face model approach , 2012, Image Vis. Comput..

[6]  Andrea Cavallaro,et al.  3-D Face Detection, Landmark Localization, and Registration Using a Point Distribution Model , 2009, IEEE Transactions on Multimedia.

[7]  Hasan Demirel,et al.  Facial Expression Recognition Using 3D Facial Feature Distances , 2007, ICIAR.

[8]  Michael G. Strintzis,et al.  Bilinear Models for 3-D Face and Facial Expression Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[9]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[10]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Liming Chen,et al.  Fully automatic 3D facial expression recognition using a region-based approach , 2011, J-HGBU '11.

[12]  Ioan Marius Bilasco,et al.  Intelligent pixels of interest selection with application to facial expression recognition using multilayer perceptron , 2013, Signal Process..

[13]  Stefano Berretti,et al.  Shape analysis of local facial patches for 3D facial expression recognition , 2011, Pattern Recognit..

[14]  Frédéric Jurie,et al.  Face Recognition using Local Quantized Patterns , 2012, BMVC.

[15]  Dacheng Tao,et al.  Common Feature Discriminant Analysis for Matching Infrared Face Images to Optical Face Images , 2014, IEEE Transactions on Image Processing.

[16]  Bill Triggs,et al.  Visual Recognition Using Local Quantized Patterns , 2012, ECCV.

[17]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Subramanian Ramanathan,et al.  Human Facial Expression Recognition using a 3D Morphable Model , 2006, 2006 International Conference on Image Processing.

[19]  Qiuqi Ruan,et al.  3D Facial expression recognition based on basic geometric features , 2010, IEEE 10th INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING PROCEEDINGS.

[20]  Yanfeng Sun,et al.  3D face recognition using local binary patterns , 2013, Signal Process..

[21]  Liming Chen,et al.  Author manuscript, published in "Workshop 3D Face Biometrics, IEEE Automatic Facial and Gesture Recognition, Shanghai: China (2013)" Fully Automatic 3D Facial Expression Recognition using Differential Mean Curvature Maps and Histograms of Oriented Gradien , 2013 .

[22]  Ioannis A. Kakadiaris,et al.  Using Facial Symmetry to Handle Pose Variations in Real-World 3D Face Recognition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[24]  H. Demirel,et al.  3D facial expression recognition with geometrically localized facial features , 2008, 2008 23rd International Symposium on Computer and Information Sciences.

[25]  Hasan Demirel,et al.  Application of NSGA-II to feature selection for facial expression recognition , 2011, Comput. Electr. Eng..

[26]  Thomas S. Huang,et al.  3D facial expression recognition based on automatically selected features , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[27]  Paul Suetens,et al.  A Comparative Study of 3-D Face Recognition Under Expression Variations , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[28]  Yichen Wei,et al.  Face alignment by Explicit Shape Regression , 2012, CVPR.

[29]  Lijun Yin,et al.  Static and dynamic 3D facial expression recognition: A comprehensive survey , 2012, Image Vis. Comput..

[30]  Mohammed Bennamoun,et al.  An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[32]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[33]  Alberto Del Bimbo,et al.  A Set of Selected SIFT Features for 3D Facial Expression Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[34]  Chun Chen,et al.  Three-Dimensional Face Reconstruction From a Single Image by a Coupled RBF Network , 2012, IEEE Transactions on Image Processing.

[35]  Yuxiao Hu,et al.  Spontaneous Emotional Facial Expression Detection , 2006, J. Multim..

[36]  M. Iqbal Saripan,et al.  3D facial expression recognition using maximum relevance minimum redundancy geometrical features , 2012, EURASIP Journal on Advances in Signal Processing.

[37]  Ashraf A. Kassim,et al.  Resampling Approach to Facial Expression Recognition Using 3D Meshes , 2010, 2010 20th International Conference on Pattern Recognition.

[38]  Xinge You,et al.  Robust face recognition via occlusion dictionary learning , 2014, Pattern Recognit..

[39]  Emmanuel Dellandréa,et al.  Automatic 3D Facial Expression Recognition Based on a Bayesian Belief Net and a Statistical Facial Feature Model , 2010, 2010 20th International Conference on Pattern Recognition.

[40]  Baochang Zhang,et al.  Local Derivative Pattern Versus Local Binary Pattern: Face Recognition With High-Order Local Pattern Descriptor , 2010, IEEE Transactions on Image Processing.

[41]  Di Huang,et al.  Local Binary Patterns and Its Application to Facial Image Analysis: A Survey , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[42]  Ioannis A. Kakadiaris,et al.  3D facial expression recognition: A perspective on promises and challenges , 2011, Face and Gesture 2011.

[43]  Maurício Pamplona Segundo,et al.  Automatic 3D facial segmentation and landmark detection , 2007, 14th International Conference on Image Analysis and Processing (ICIAP 2007).

[44]  Chun Chen,et al.  Feature level analysis for 3D facial expression recognition , 2011, Neurocomputing.

[45]  Ashraf A. Kassim,et al.  A novel approach to classification of facial expressions from 3D-mesh datasets using modified PCA , 2009, Pattern Recognit. Lett..

[46]  Timo Ahonen,et al.  Recognition of blurred faces using Local Phase Quantization , 2008, 2008 19th International Conference on Pattern Recognition.

[47]  Hasan Demirel,et al.  Feature selection for person-independent 3D facial expression recognition using NSGA-II , 2009, 2009 24th International Symposium on Computer and Information Sciences.

[48]  Yoichi Sato,et al.  Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates , 2007, ACCV.

[49]  Alberto Del Bimbo,et al.  3D facial expression recognition using SIFT descriptors of automatically detected keypoints , 2011, The Visual Computer.

[50]  Xiaoou Tang,et al.  Automatic facial expression recognition on a single 3D face by exploring shape deformation , 2009, ACM Multimedia.

[51]  Thomas S. Huang,et al.  3D facial expression recognition based on properties of line segments connecting facial feature points , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.