A fuzzy logic approach to reliable real-time recognition of facial emotions

This paper represents our newly developed software for emotion recognition from facial expressions. Besides allowing emotion recognition from image files and recorded video files, it uses webcam data to provide real-time, continuous, and unobtrusive facial emotional expressions. It uses FURIA algorithm for unordered fuzzy rule induction to offer timely and appropriate feedback based on learners’ facial expressions. The main objective of this study was first to validate the use of webcam data for a real-time and accurate analysis of facial expressions in e-learning environments. Second, transform these facial expressions to detected emotional states using the FURIA algorithm. We measured the performance of the software with ten participants, provided them with the same computer-based tasks, requested them a hundred times to mimic specific facial expressions, and recorded all sessions on video. We used the recorded video files to feed our newly developed software. We then used two experts’ opinions to annotate and rate participants’ recorded behaviours and to validate the software’s results. The software provides accurate and reliable results with the overall accuracy of 83.2%, which is comparable to the recognition by humans. This study will help to increase the quality of e-learning.

[1]  Toby Sharp,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR.

[2]  W. Marston,et al.  Systolic blood pressure symptoms of deception. , 1917 .

[3]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[4]  Peter Brusilovsky,et al.  Student model centered architecture for intelligent learning environments , 1994 .

[5]  Y. V. Venkatesh,et al.  A novel application of self-organizing network for facial expression recognition from radial encoded contours , 2009, Soft Comput..

[6]  Yuan Luo,et al.  Facial expression recognition based on fusion feature of PCA and LBP with SVM , 2013 .

[7]  Omar AlZoubi,et al.  Affect Detection from Multichannel Physiology during Learning Sessions with AutoTutor , 2011, AIED.

[8]  Vinay Bettadapura,et al.  Face Expression Recognition and Analysis: The State of the Art , 2012, ArXiv.

[9]  P. Hager,et al.  Recovering Informal Learning: Wisdom, Judgement and Community , 2007 .

[10]  Kiavash Bahreini,et al.  SDISSASA: A Multiagent-Based Web Mining via Semantic Access to Web Resources in Enterprise Architecture , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[11]  Wim Westera,et al.  RAGE Architecture for Reusable Serious Gaming Technology Components , 2016, Int. J. Comput. Games Technol..

[12]  Zhengyou Zhang,et al.  Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments with A Multilayer Perceptron , 1999, Int. J. Pattern Recognit. Artif. Intell..

[13]  Shyamal Patel,et al.  A review of wearable sensors and systems with application in rehabilitation , 2012, Journal of NeuroEngineering and Rehabilitation.

[14]  Kiavash Bahreini,et al.  Towards multimodal emotion recognition in e-learning environments , 2016, Interact. Learn. Environ..

[15]  Fuji Ren,et al.  The study on text emotional orientation based on a three-dimensional emotion space model , 2009, 2009 International Conference on Natural Language Processing and Knowledge Engineering.

[16]  Q. Ruan,et al.  Facial expression recognition based orthogonal local fisher discriminant analysis , 2010, IEEE 10th INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING PROCEEDINGS.

[17]  Abdolhossein Sarrafzadeh,et al.  "How do you know that I don't understand?" A look at the future of intelligent tutoring systems , 2008, Comput. Hum. Behav..

[18]  K. Scherer,et al.  Bodily expression of emotion , 2009 .

[19]  L. Porter,et al.  Communication in Organizations. , 1972 .

[20]  Henk T. van der Molen,et al.  Communication in Organizations: Basic Skills and Conversation Models , 2005 .

[21]  Sazali Yaacob,et al.  Facial emotion recognition using empirical mode decomposition , 2015, Expert Syst. Appl..

[22]  Oliver Lüdtke,et al.  Characteristics of teaching and students’ emotions in the classroom: Investigating differences across domains , 2013 .

[23]  R. S. Jadon,et al.  Effectiveness of Eigenspaces for Facial Expressions Recognition , 2009 .

[24]  Bir Bhanu,et al.  Facial expression recognition using emotion avatar image , 2011, Face and Gesture 2011.

[25]  W. Westera,et al.  Communication skills training exploiting multimodal emotion recognition , 2017, Interact. Learn. Environ..

[26]  R. Pekrun.  The impact of emotions on learning and achievement : towards a theory of cognitive/motivational mediators , 1992 .

[27]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[28]  Eero Väyrynen,et al.  Emotion recognition from speech using prosodic features , 2014 .

[29]  Mahmoud Neji,et al.  The Affective Tutoring System , 2010, Expert Syst. Appl..

[30]  Hosub Lee,et al.  Towards unobtrusive emotion recognition for affective social communication , 2012, 2012 IEEE Consumer Communications and Networking Conference (CCNC).

[31]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[32]  Nicu Sebe,et al.  Multimodal approaches for emotion recognition: a survey , 2005, IS&T/SPIE Electronic Imaging.

[33]  S.V. Dudul,et al.  Emotion recognition from facial expression using neural networks , 2008, 2008 Conference on Human System Interactions.

[34]  Alessandro De Gloria,et al.  Serious Games for education and training , 2014, Int. J. Serious Games.

[35]  Kiavash Bahreini,et al.  Towards real-time speech emotion recognition for affective e-learning , 2015, Education and Information Technologies.

[36]  P. Ekman,et al.  Facial action coding system , 2019 .

[37]  Emiel Krahmer,et al.  Audiovisual Expression of Emotions in Communication , 2010 .

[38]  P. L. Brusilovskiy,et al.  The construction and application of student models in intelligent tutoring systems , 1994 .

[39]  J.A. Ortega,et al.  A low-cost Webcam & personal computer opens doors , 2005, IEEE Aerospace and Electronic Systems Magazine.

[40]  Kiavash Bahreini,et al.  A New Software Architecture for J2EE Enterprise Environments via Semantic Access to Web Sources for Web Mining by Distributed Intelligent Software Agents , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[41]  Arunkumar P Chavan,et al.  A 1.5-V, 10-bit, 200-MS/s CMOS Pipeline Analog-to- Digital Converter , 2014 .

[42]  Nicu Sebe,et al.  Multimodal interfaces: Challenges and perspectives , 2009, J. Ambient Intell. Smart Environ..

[43]  Hamed Shah-Hosseini,et al.  A novel fuzzy facial expression recognition system based on facial feature extraction from color face images , 2012, Eng. Appl. Artif. Intell..

[44]  Chung-Hsien Wu,et al.  Emotion recognition from text using semantic labels and separable mixture models , 2006, TALIP.

[45]  Rob Nadolski,et al.  Serious games for higher education: a framework for reducing design complexity , 2008, J. Comput. Assist. Learn..

[46]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[47]  Sean Davis,et al.  Statistical Genomics. Methods and Protocols. , 2016, Anticancer research.

[48]  Bernd Kleinjohann,et al.  Real-Time Facial Expression Recognition Using a Fuzzy Emotion Model , 2007, 2007 IEEE International Fuzzy Systems Conference.

[49]  Eyke Hüllermeier,et al.  FURIA: an algorithm for unordered fuzzy rule induction , 2009, Data Mining and Knowledge Discovery.

[50]  Rita Francese,et al.  Wiimote and Kinect: gestural user interfaces add a natural third dimension to HCI , 2012, AVI.

[51]  Ursula Hess,et al.  Empathic reactions of younger and older adults: No age related decline in affective responding ☆ , 2014 .

[52]  Martin Magdin,et al.  Evaluating the Emotional State of a User Using a Webcam , 2016 .

[53]  Edgar Erdfelder,et al.  GPOWER: A general power analysis program , 1996 .

[54]  Thanasis Daradoumis,et al.  Emotion Measurement in Intelligent Tutoring Systems: What, When and How to Measure , 2011, 2011 Third International Conference on Intelligent Networking and Collaborative Systems.

[55]  Jean-Claude Martin,et al.  Gesture and emotion: Can basic gestural form features discriminate emotions? , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[56]  Hazem M. Hajj,et al.  Emotion Recognition from Text Based on Automatically Generated Rules , 2014, 2014 IEEE International Conference on Data Mining Workshop.

[57]  Siva Prasad Nandyala,et al.  Automatic Speech Emotion and Speaker Recognition Based on Hybrid GMM and FFBNN , 2014 .

[58]  T. Chau,et al.  Comparison of blood volume pulse and skin conductance responses to mental and affective stimuli at different anatomical sites , 2011, Physiological measurement.

[59]  Hatice Gunes,et al.  Automatic, Dimensional and Continuous Emotion Recognition , 2010, Int. J. Synth. Emot..

[60]  Nicu Sebe,et al.  Affective multimodal human-computer interaction , 2005, ACM Multimedia.

[61]  Astrid Paeschke,et al.  A database of German emotional speech , 2005, INTERSPEECH.

[62]  Nikos Papamarkos,et al.  A Dynamic Gesture and Posture Recognition System , 2014, J. Intell. Robotic Syst..

[63]  Yongzhao Zhan,et al.  A neural-AdaBoost based facial expression recognition system , 2014, Expert Syst. Appl..

[64]  Ken W. White,et al.  The Online Teaching Guide : A Handbook of Attitudes, Strategies, and Techniques for the Virtual Classroom , 1999 .

[65]  Kasia Muldner,et al.  Emotion Sensors Go To School , 2009, AIED.

[66]  Minjuan Wang,et al.  Affective e-Learning: Using "Emotional" Data to Improve Learning in Pervasive Learning Environment , 2009, J. Educ. Technol. Soc..

[67]  Francesca Odone,et al.  Real-time Automatic Emotion Recognition from Body Gestures , 2014, ArXiv.

[68]  D. J. Kim,et al.  Facial expression recognition using ASM-based post-processing technique , 2016, Pattern Recognition and Image Analysis.

[69]  Marco Anisetti,et al.  Emotional State Inference Using Face Related Features , 2009, KES IIMSS.

[70]  M. Arfan Jaffar,et al.  Wavelets-based facial expression recognition using a bank of support vector machines , 2012, Soft Comput..

[71]  Arthur C. Graesser,et al.  AutoTutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back , 2012, TIIS.

[72]  Horace Ho-Shing Ip,et al.  Enhance emotional and social adaptation skills for children with autism spectrum disorder: A virtual reality enabled approach , 2018, Comput. Educ..

[73]  Mitsuru Ishizuka,et al.  A chat system based on emotion estimation from text and embodied conversational messengers , 2005, Proceedings of the 2005 International Conference on Active Media Technology, 2005. (AMT 2005)..

[74]  Kiavash Bahreini,et al.  Data Fusion for Real-time Multimodal Emotion Recognition through Webcams and Microphones in E-Learning , 2016, Int. J. Hum. Comput. Interact..

[75]  Tieniu Tan,et al.  Affective Computing: A Review , 2005, ACII.

[76]  J. Russell.  A circumplex model of affect. , 1980 .

[77]  Wim Westera,et al.  RAGE Reusable Game Software Components and Their Integration into Serious Game Engines , 2016, ICSR.

[78]  Khanna Preeti.  MULTIMODAL EMOTION RECOGNITION FOR ENHANCING HUMAN COMPUTER INTERACTION , 2012 .

[79]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[80]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[81]  Anil K. Jain,et al.  Handbook of Face Recognition, 2nd Edition , 2011 .