What happens in Face during a facial expression? Using data mining techniques to analyze facial expression motion vectors

One of the most common problems encountered in human-computer interaction is automatic facial expression recognition. Although it is easy for human observer to recognize facial expressions, automatic recognition remains difficult for machines. One of the methods that machines can recognize facial expression is analyzing the changes in face during facial expression presentation. In this paper, optical flow algorithm was used to extract deformation or motion vectors created in the face because of facial expressions. Then, these extracted motion vectors are used to be analyzed. Their positions and directions were exploited for automatic facial expression recognition using different data mining techniques. It means that by employing motion vector features used as our data, facial expressions were recognized. Some of the most state-of-the-art classification algorithms such as C5.0, CRT, QUEST, CHAID, Deep Learning (DL), SVM and Discriminant algorithms were used to classify the extracted motion vectors. Using 10-fold cross validation, their performances were calculated. To compare their performance more precisely, the test was repeated 50 times. Meanwhile, the deformation of face was also analyzed in this research. For example, what exactly happened in each part of face when a person showed fear? Experimental results on Extended Cohen-Kanade (CK+) facial expression dataset demonstrated that the best methods were DL, SVM and C5.0, with the accuracy of 95.3%, 92.8% and 90.2% respectively.

[1]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Hanêne Ben-Abdallah,et al.  Data Mining-based Facial Expressions Recognition System , 2013, SCAI.

[3]  Maja Pantic,et al.  A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Kin-Man Lam,et al.  Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study , 2018, J. Vis. Commun. Image Represent..

[5]  H. Hannan.  The Sage Encyclopedia of Theory in Psychology , 2017 .

[6]  Jafar Habibi,et al.  A data mining approach for diagnosis of coronary artery disease , 2013, Comput. Methods Programs Biomed..

[7]  Haesun Park,et al.  A comparison of generalized linear discriminant analysis algorithms , 2008, Pattern Recognit..

[8]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[9]  Lauri Nummenmaa,et al.  Perceptual and affective mechanisms in facial expression recognition: An integrative review , 2016, Cognition & emotion.

[10]  Yan Zhao,et al.  WGAN-Based Robust Occluded Facial Expression Recognition , 2019, IEEE Access.

[11]  P C Vasanth,et al.  Facial Expression Recognition Using SVM Classifier , 2015 .

[12]  Mohamed Hammami,et al.  Automatic Facial Expression Recognition System , 2013, 2013 ACS International Conference on Computer Systems and Applications (AICCSA).

[13]  Subhi R. M. Zeebaree,et al.  An Analytical Appraisal for Supervised Classifiers’ Performance on Facial Expression Recognition Based on Relief-F Feature Selection , 2021 .

[14]  J. Shanbehzadeh,et al.  Automatic Adaptive Center of Pupil Detection Using Face Detection and CDF Analysis , 2010 .

[15]  Ke Wang,et al.  Unsupervised feature selection based on spectral regression from manifold learning for facial expression recognition , 2015, IET Comput. Vis..

[16]  Rajendran Parthiban,et al.  Joint facial expression recognition and intensity estimation based on weighted votes of image sequences , 2017, Pattern Recognit. Lett..

[17]  Jules R. Tapamo,et al.  Improved gradient local ternary patterns for facial expression recognition , 2017, EURASIP J. Image Video Process..

[18]  Juan Manuel Górriz,et al.  Early diagnosis of Alzheimer's disease based on Partial Least Squares and Support Vector Machine , 2013, Expert Syst. Appl..

[19]  Aggelos K. Katsaggelos,et al.  Automatic facial expression recognition using facial animation parameters and multistream HMMs , 2006, IEEE Transactions on Information Forensics and Security.

[20]  Seok-Pil Lee,et al.  Human emotion recognition based on the weighted integration method using image sequences and acoustic features , 2020 .

[21]  Haitao Zhao,et al.  An efficient unconstrained facial expression recognition algorithm based on Stack Binarized Auto-encoders and Binarized Neural Networks , 2017, Neurocomputing.

[22]  Qingshan Liu,et al.  Spatio-temporal convolutional features with nested LSTM for facial expression recognition , 2018, Neurocomputing.

[23]  José Marques Soares,et al.  Systematic review of 3D facial expression recognition methods , 2020, Pattern Recognit..

[24]  Shervin Minaee,et al.  Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network , 2019, Sensors.

[25]  Alaa Eleyan,et al.  Facial expression recognition using Support Vector Machines , 2015, 2015 23nd Signal Processing and Communications Applications Conference (SIU).

[26]  Satishkumar S. Chavan,et al.  Analysis of PCA and LDA Features for Facial Expression Recognition Using SVM and HMM Classifiers , 2020 .

[27]  Mayur Rahul,et al.  Facial expression recognition using local multidirectional score pattern descriptor and modified hidden Markov model , 2021, Int. J. Adv. Intell. Paradigms.

[28]  Kartika Candra Kirana,et al.  Facial Emotion Recognition Based on Viola-Jones Algorithm in the Learning Environment , 2018, 2018 International Seminar on Application for Technology of Information and Communication.

[29]  Saeid Nahavandi,et al.  Deep Learning for Neuroimaging-based Diagnosis and Rehabilitation of Autism Spectrum Disorder: A Review , 2020, Comput. Biol. Medicine.

[30]  Jean-Pierre Seifert,et al.  A new CRT-RSA algorithm secure against bellcore attacks , 2003, CCS '03.

[31]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[32]  Asit Barman,et al.  Facial expression recognition using distance and texture signature relevant features , 2019, Appl. Soft Comput..

[33]  Aurobinda Routray,et al.  Robust facial expression classification using shape and appearance features , 2015, 2015 Eighth International Conference on Advances in Pattern Recognition (ICAPR).

[34]  Sanjay Singh,et al.  Facial Expression Recognition Using Improved Adaptive Local Ternary Pattern , 2018, CVIP.

[35]  Saeid Nahavandi,et al.  Applications of Deep Learning Techniques for Automated Multiple Sclerosis Detection Using Magnetic Resonance Imaging: A Review , 2021, Comput. Biol. Medicine.

[36]  Jörgen Ahlberg,et al.  Fast facial expression recognition using local binary features and shallow neural networks , 2018, The Visual Computer.

[37]  Changqin Quan,et al.  Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients , 2015, 2015 IEEE International Conference on Mechatronics and Automation (ICMA).

[38]  Mohammed Yeasin,et al.  Recognition of facial expressions and measurement of levels of interest from video , 2006, IEEE Transactions on Multimedia.

[39]  Mohammad H. Mahoor,et al.  Spatio-Temporal Facial Expression Recognition Using Convolutional Neural Networks and Conditional Random Fields , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[40]  Chien-Liang Lin,et al.  Evaluation of CART, CHAID, and QUEST algorithms: a case study of construction defects in Taiwan , 2019 .

[41]  Fabian J. Theis,et al.  Denoising using local projective subspace methods , 2006, Neurocomputing.

[42]  Arman Savran,et al.  Regression-based intensity estimation of facial action units , 2012, Image Vis. Comput..

[43]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[44]  Nyein Aye,et al.  Automatic Facial expression Recognition System using Orientation Histogram and Neural Network , 2013 .

[45]  Milan Sonka,et al.  Image Processing, Analysis and Machine Vision , 1993, Springer US.

[46]  Vikash Yadav,et al.  Gabor Filter and ICA-Based Facial Expression Recognition Using Two-Layered Hidden Markov Model , 2021 .

[47]  Khalid Satori,et al.  Clustering and Jarque-Bera Normality Test to Face Recognition , 2018 .

[48]  Sheng Yang,et al.  A fast mouth detection algorithm based on face organs , 2009, 2009 2nd International Conference on Power Electronics and Intelligent Transportation System (PEITS).

[49]  Wael Mohammad Alenazy,et al.  Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition , 2020 .

[50]  Kesari Verma,et al.  Facial Expression Recognition Using Data Mining Algorithm , 2013 .

[51]  Jens Myrup Pedersen,et al.  A method for classification of network traffic based on C5.0 Machine Learning Algorithm , 2012, 2012 International Conference on Computing, Networking and Communications (ICNC).

[52]  Ahmad Y. Javaid,et al.  Recognition of Emotion Intensities Using Machine Learning Algorithms: A Comparative Study , 2019, Sensors.

[53]  Adnan Mohsin Abdulazeez,et al.  Facial Expression Recognition Based on Deep Learning Convolution Neural Network: A Review , 2021 .

[54]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[55]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[56]  Haitao Zhao,et al.  A visual attention based ROI detection method for facial expression recognition , 2018, Neurocomputing.

[57]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[58]  R. K. Kulkarni,et al.  Face detection and facial expression recognition system , 2014, 2014 International Conference on Electronics and Communication Systems (ICECS).

[59]  Shiqing Zhang,et al.  A Review on Facial Expression Recognition: Feature Extraction and Classification , 2016 .

[60]  Maja Pantic,et al.  Meta-Analysis of the First Facial Expression Recognition Challenge , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[61]  Marc M. Van Hulle,et al.  A phase-based approach to the estimation of the optical flow field using spatial filtering , 2002, IEEE Trans. Neural Networks.

[62]  Pham The Bao,et al.  A New Approach to Mouth Detection Using Neural Network , 2009, 2009 IITA International Conference on Control, Automation and Systems Engineering (case 2009).

[63]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[64]  Wang Li,et al.  Mouth Detection Based on Interest Point , 2006, 2007 Chinese Control Conference.

[65]  Soumaya Yacout,et al.  Intelligent Data Mining For Automatic Face Recognition , 2013 .

[66]  Hsi-Chieh Lee,et al.  Facial Expression Recognition Using Image Processing Techniques and Neural Networks , 2013 .

[67]  Ahmad R. Naghsh-Nilchi,et al.  An Efficient Algorithm for Motion Detection Based Facial Expression Recognition using Optical Flow , 2008 .

[68]  Ing Ren Tsang,et al.  FERAtt: Facial Expression Recognition With Attention Net , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[69]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[70]  Guang Liang,et al.  Identity- and Pose-Robust Facial Expression Recognition through Adversarial Feature Learning , 2019, ACM Multimedia.

[71]  S. Demleitner.  [Communication without words]. , 1997, Pflege aktuell.

[72]  Qiang Ji,et al.  Measuring the intensity of spontaneous facial action units with dynamic Bayesian network , 2015, Pattern Recognit..

[73]  A. Durmuşoğlu,et al.  Face Expression Recognition Using a Combination of Local Binary Patterns and Local Phase Quantization , 2021, 2021 International Conference on Communication, Control and Information Sciences (ICCISc).

[74]  Dewi Yanti Liliana,et al.  Fuzzy emotion: a natural approach to automatic facial expression recognition from psychological perspective using fuzzy system , 2019, Cognitive Processing.

[75]  I. Bar-Itzhack.  REQUEST: A Recursive QUEST Algorithm for Sequential Attitude Determination , 1996 .