iFER: facial expression recognition using automatically selected geometric eye and eyebrow features

Abstract. Facial expressions have an important role in interpersonal communications and estimation of emotional states or intentions. Automatic recognition of facial expressions has led to many practical applications and became one of the important topics in computer vision. We present a facial expression recognition system that relies on geometry-based features extracted from eye and eyebrow regions of the face. The proposed system detects keypoints on frontal face images and forms a feature set using geometric relationships among groups of detected keypoints. Obtained feature set is refined and reduced using the sequential forward selection (SFS) algorithm and fed to a support vector machine classifier to recognize five facial expression classes. The proposed system, iFER (eye–eyebrow only facial expression recognition), is robust to lower face occlusions that may be caused by beards, mustaches, scarves, etc. and lower face motion during speech production. Preliminary experiments on benchmark datasets produced promising results outperforming previous facial expression recognition studies using partial face features, and comparable results to studies using whole face information, only slightly lower by   ∼  2.5  %   compared to the best whole face facial recognition system while using only   ∼  1  /  3 of the facial region.

[1]  Tae-Sun Choi,et al.  Boosted NNE collections for multicultural facial expression recognition , 2016, Pattern Recognit..

[2]  Abolghasem A. Raie,et al.  Suitable models for face geometry normalization in facial expression recognition , 2015, J. Electronic Imaging.

[3]  Y. V. Venkatesh,et al.  Facial expression recognition using radial encoding of local Gabor features and classifier synthesis , 2012, Pattern Recognit..

[4]  Alaa Eleyan,et al.  Facial expression recognition using Support Vector Machines , 2015, 2015 23nd Signal Processing and Communications Applications Conference (SIU).

[5]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[6]  Kamlesh Mistry,et al.  Intelligent Appearance and shape based facial emotion recognition for a humanoid robot , 2014, The 8th International Conference on Software, Knowledge, Information Management and Applications (SKIMA 2014).

[7]  Ninad Thakoor,et al.  Vision and Attention Theory Based Sampling for Continuous Facial Emotion Recognition , 2014, IEEE Transactions on Affective Computing.

[8]  Martin D. Levine,et al.  Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers , 2014, IEEE Transactions on Affective Computing.

[9]  C. Reddy.  Improving the Classification Accuracy of Emotion Recognition using Facial Expressions , 2016 .

[10]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[11]  Jean Meunier,et al.  Emotion recognition using dynamic grid-based HoG features , 2011, Face and Gesture 2011.

[12]  Javier R. Movellan,et al.  Design and early evaluation of the RUBI-5 sociable robots , 2012, 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL).

[13]  Yi Zhang,et al.  Facial expression feature extraction using hybrid PCA and LBP , 2013 .

[14]  Sang Hyun Park,et al.  Facial expression recognition based on local region specific features and support vector machines , 2016, Multimedia Tools and Applications.

[15]  Toon De Pessemier,et al.  Enhancing Recommender Systems for TV by Face Recognition , 2016, WEBIST.

[16]  Hamidreza Rashidy Kanan,et al.  Local weighted Pseudo Zernike Moments and fuzzy classification for facial expression recognition , 2013, 2013 13th Iranian Conference on Fuzzy Systems (IFSC).

[17]  Stephan K. Chalup,et al.  Scene perception using pareidolia of faces and expressions of emotion , 2013, 2013 IEEE Symposium on Computational Intelligence for Creativity and Affective Computing (CICAC).

[18]  Hela Mahersia,et al.  Using multiple steerable filters and Bayesian regularization for facial expression recognition , 2015, Eng. Appl. Artif. Intell..

[19]  Muneeswaran Karuppiah,et al.  Recognize the facial emotion in video sequences using eye and mouth temporal Gabor features , 2016, Multimedia Tools and Applications.

[20]  A. Mehrabian.  Some referents and measures of nonverbal behavior , 1968 .

[21]  Ren C. Luo,et al.  Multimodal information fusion for human-robot interaction , 2015, 2015 IEEE 10th Jubilee International Symposium on Applied Computational Intelligence and Informatics.

[22]  Yongzhao Zhan,et al.  Multi-pose facial expression recognition based on SURF boosting , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[23]  Yan Zhang,et al.  Driver fatigue recognition based on facial expression analysis using local binary patterns , 2015 .

[24]  Anna Hilsmann,et al.  Real-time avatar animation with dynamic face texturing , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[25]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Qing Liu,et al.  Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning , 2016, J. Electronic Imaging.

[27]  Ashley Smith,et al.  An end-to-end vechicle classification pipeline using vibrometry data , 2014, Defense + Security Symposium.

[28]  Marco La Cascia,et al.  Using Hankel matrices for dynamics-based facial emotion recognition and pain detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Anthony Choi,et al.  Facial Emotion Recognition Using Fuzzy Systems , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[30]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[31]  A. Wayne Whitney,et al.  A Direct Method of Nonparametric Measurement Selection , 1971, IEEE Transactions on Computers.

[32]  Shuzhi Sam Ge,et al.  Active affective facial analysis for human-robot interaction , 2008, RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication.

[33]  Chokri Ben Amar,et al.  Vigilance measurement system through analysis of visual and emotional driver's signs using wavelet networks , 2015, 2015 15th International Conference on Intelligent Systems Design and Applications (ISDA).

[34]  Kandarpa Kumar Sarma,et al.  Different face regions detection based facial expression recognition , 2015, 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN).

[35]  Svetlana Yanushkevich,et al.  Effects of training set selection on pain recognition via facial expressions , 2016, International Workshop on Pattern Recognition.

[36]  Bin Jiang,et al.  Robust facial expression recognition algorithm based on local metric learning , 2016, J. Electronic Imaging.

[37]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[38]  Bo Sun,et al.  Facial expression recognition in the wild based on multimodal texture features , 2016, J. Electronic Imaging.

[39]  Matti Pietikäinen,et al.  Towards a dynamic expression recognition system under facial occlusion , 2012, Pattern Recognit. Lett..

[40]  Zahir M. Hussain,et al.  Feature extraction for facial expression recognition based on hybrid face regions , 2009 .

[41]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[42]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..

[43]  Myung Jin Chung,et al.  Cognitive face analysis system for future interactive TV , 2009, IEEE Transactions on Consumer Electronics.

[44]  Zhaohui Wu,et al.  Facial expression recognition based on meta probability codes , 2013, Pattern Analysis and Applications.

[45]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[46]  Takeo Kanade,et al.  Facial Expression Recognition , 2011, Handbook of Face Recognition.

[47]  Héctor M. Pérez Meana,et al.  Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach , 2016, Knowl. Based Syst..

[48]  Arundhati Das,et al.  Facial components extraction and expression recognition in static images , 2015, 2015 International Conference on Green Computing and Internet of Things (ICGCIoT).

[49]  Takehisa Onisawa,et al.  Human-agent communication in poker game , 2010, 2010 IEEE International Conference on Systems, Man and Cybernetics.

[50]  Prem Kalra,et al.  Face to virtual face , 1998, Proc. IEEE.

[51]  Ashim Saha,et al.  A Human Facial Expression Recognition Model Based on Eigen Face Approach , 2015 .

[52]  Hélio Pedrini,et al.  Effects of cultural characteristics on building an emotion classifier through facial expression analysis , 2015, J. Electronic Imaging.

[53]  Youping Deng,et al.  Feature Selection and Classification of MAQC-II Breast Cancer and Multiple Myeloma Microarray Gene Expression Data , 2009, PloS one.

[54]  Pojala Chiranjeevi,et al.  Neutral Face Classification Using Personalized Appearance Models for Fast and Robust Emotion Detection , 2015, IEEE Transactions on Image Processing.

[55]  Mehdi Ghayoumi,et al.  An integrated approach for efficient analysis of facial expressions , 2014, 2014 International Conference on Signal Processing and Multimedia Applications (SIGMAP).

[56]  Kamlesh Mistry,et al.  Adaptive facial point detection and emotion recognition for a humanoid robot , 2015, Comput. Vis. Image Underst..

[57]  Pierluigi Carcagnì,et al.  Automatic Emotion Recognition in Robot-Children Interaction for ASD Treatment , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[58]  G. Ram Mohana Reddy,et al.  A real time facial emotion recognition using depth sensor and interfacing with Second Life based Virtual 3D avatar , 2014, International Conference on Recent Advances and Innovations in Engineering (ICRAIE-2014).

[59]  Ninad Thakoor,et al.  Facial emotion recognition in continuous video , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[60]  Constantine Kotropoulos,et al.  Sequential forward feature selection with low computational cost , 2005, 2005 13th European Signal Processing Conference.