On the use of Action Units and fuzzy explanatory models for facial expression recognition

Facial expression recognition is related to the automatic identification of affective states of a subject by computational means. Facial expression recognition is used for many applications, such as security, human-computer interaction, driver safety, and health care. Although many works aim to tackle the problem of facial expression recognition, and the discriminative power may be acceptable, current solutions have limited explicative power, which is insufficient for certain applications, such as facial rehabilitation. Our aim is to alleviate the current limited explicative power by exploiting explainable fuzzy models over sequences of frontal face images. The proposed model uses appearance features to describe facial expressions in terms of facial movements, giving a detailed explanation of what movements are in the face, and why the model is making a decision. The model architecture was selected to keep the semantic meaning of the found facial movements. The proposed model can discriminate between the seven basic facial expressions, obtaining an average accuracy of 90.8±14%, with a maximum value of 92.9±28%.

[1]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[2]  Hui Zhao,et al.  Facial Complex Expression Recognition Based on Fuzzy Kernel Clustering and Support Vector Machines , 2007, Third International Conference on Natural Computation (ICNC 2007).

[3]  David J. Reinkensmeyer,et al.  Gesture Therapy: A Vision-Based System for Arm Rehabilitation after Stroke , 2008, BIOSTEC.

[4]  Marta Mejail,et al.  Facial expression recognition: A comparison between static and dynamic approaches , 2016, ICPR 2016.

[5]  Mahadev Satyanarayanan,et al.  OpenFace: A general-purpose face recognition library with mobile applications , 2016 .

[6]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[8]  Jean Ponce,et al.  A Theoretical Analysis of Feature Pooling in Visual Recognition , 2010, ICML.

[9]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[10]  Witold Pedrycz,et al.  Granular fuzzy models: Analysis, design, and evaluation , 2015, Int. J. Approx. Reason..

[11]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[12]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[13]  Tae-Sun Choi,et al.  Boosted NNE collections for multicultural facial expression recognition , 2016, Pattern Recognit..

[14]  M. Mohammadi,et al.  Fuzzy local binary patterns: A comparison between Min-Max and Dot-Sum operators in the application of facial expression recognition , 2013, 2013 8th Iranian Conference on Machine Vision and Image Processing (MVIP).

[15]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  J. Rice,et al.  Topical minoxidil for male pattern baldness. , 1986, The American journal of the medical sciences.

[17]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Sridha Sridharan,et al.  Automatically detecting pain using facial actions , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[20]  Maryam Ahmady,et al.  Facial expression recognition using facial effective areas and Fuzzy logic , 2014, 2014 Iranian Conference on Intelligent Systems (ICIS).

[21]  Patrick J Byrne,et al.  Importance of facial expression in facial nerve rehabilitation , 2004, Current opinion in otolaryngology & head and neck surgery.

[22]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[23]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[24]  Hongxing Li,et al.  Fuzzy Neural Intelligent Systems , 2000 .

[25]  Manuel Franco-Martín,et al.  The Efficacy of Emotion Recognition Rehabilitation for People with Alzheimer's Disease. , 2017, Journal of Alzheimer's disease : JAD.

[26]  A. Khanam,et al.  Fuzzy Based Facial Expression Recognition , 2008, 2008 Congress on Image and Signal Processing.

[27]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[28]  R. A. Rahmat,et al.  GENERATION OF FUZZY RULES WITH SUBTRACTIVE CLUSTERING , 2005 .

[29]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[30]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[31]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[32]  Rodrigo Ferreira Berriel,et al.  Cross-Database Facial Expression Recognition Based on Fine-Tuned Deep Convolutional Network , 2017, 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).

[33]  Brian C. Lovell,et al.  Landmark manifold: Revisiting the Riemannian manifold approach for facial emotion recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[34]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.