Towards multimodal emotion recognition in e-learning environments

This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner's facial expressions and verbalizations. FILTWAM's facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners' behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner's emotional states.

[1]  Martin Ebner,et al.  E-Learning 2.0 = e-Learning 1.0 + Web 2.0? , 2007, The Second International Conference on Availability, Reliability and Security (ARES'07).

[2]  R. S. Jadon,et al.  Effectiveness of Eigenspaces for Facial Expressions Recognition , 2009 .

[3]  Emiel Krahmer,et al.  Audiovisual Expression of Emotions in Communication , 2010 .

[4]  Kiavash Bahreini,et al.  FILTWAM - A Framework for Online Affective Computing in Serious Games , 2012, VS-GAMES.

[5]  Thanasis Daradoumis,et al.  Emotion Measurement in Intelligent Tutoring Systems: What, When and How to Measure , 2011, 2011 Third International Conference on Intelligent Networking and Collaborative Systems.

[6]  Nicu Sebe,et al.  Affective multimodal human-computer interaction , 2005, ACM Multimedia.

[7]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[8]  Mahmoud Neji,et al.  The Affective Tutoring System , 2010, Expert Syst. Appl..

[9]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Tieniu Tan,et al.  Affective Computing: A Review , 2005, ACII.

[11]  Henk T. van der Molen,et al.  Communication in Organizations: Basic Skills and Conversation Models , 2005 .

[12]  R. Pekrun.  The impact of emotions on learning and achievement : towards a theory of cognitive/motivational mediators , 1992 .

[13]  Nicu Sebe,et al.  Multimodal interfaces: Challenges and perspectives , 2009, J. Ambient Intell. Smart Environ..

[14]  P. Hager,et al.  Recovering Informal Learning: Wisdom, Judgement and Community , 2007 .

[15]  Zhengyou Zhang,et al.  Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments with A Multilayer Perceptron , 1999, Int. J. Pattern Recognit. Artif. Intell..

[16]  Kiavash Bahreini,et al.  FILTWAM - A Framework for Online Game-based Communication Skills Training , 2012 .

[17]  Maria Virvou,et al.  Automatic generation of emotions in tutoring agents for affective e-learning in medical education , 2011, Expert Syst. Appl..

[18]  Zixue Cheng,et al.  A personalized ubiquitous education support environment by comparing learning instructional requirement with learner's behavior , 2005, 19th International Conference on Advanced Information Networking and Applications (AINA'05) Volume 1 (AINA papers).

[19]  Lawrence S. Chen,et al.  Joint processing of audio-visual information for the recognition of emotional expressions in human-c , 2000 .

[20]  Hendrik Drachsler,et al.  Translating Learning into Numbers: A Generic Framework for Learning Analytics , 2012, J. Educ. Technol. Soc..

[21]  Ursula Hess,et al.  Empathic reactions of younger and older adults: No age related decline in affective responding ☆ , 2014 .

[22]  C. Chibelushi,et al.  Facial Expression Recognition : A Brief Tutorial Overview , 2022 .

[23]  Vinay Bettadapura,et al.  Face Expression Recognition and Analysis: The State of the Art , 2012, ArXiv.

[24]  Firouz B. Anaraki,et al.  Developing An Effective And Efficient eLearning Platform Using Open Source Software , 2006 .

[25]  Simon Lucey,et al.  Deformable Model Fitting by Regularized Landmark Mean-Shift , 2010, International Journal of Computer Vision.

[26]  Abdolhossein Sarrafzadeh,et al.  "How do you know that I don't understand?" A look at the future of intelligent tutoring systems , 2008, Comput. Hum. Behav..