Using Facial Expression to Detect Emotion in E-learning System: A Deep Learning Method

E-learning system is becoming more and more popular among students nowadays. However, the emotion of students is usually neglected in e-learning system. This study is mainly concerned about using facial expression to detect emotion in e-learning system. A deep learning method called convolutional neural network (CNN) is used in our research. Firstly, CNN is introduced to detect emotion in e-learning system based on using facial expression in this paper. Secondly, the training process and testing process of CNN are described. To learn about the accuracy of CNN in emotion detection, three databases (CK+, JAFFE and NVIE) are chosen to train and test the model. 10-fold cross validation method is used to calculate the accuracy. Thirdly, we introduce how to apply the trained CNN to e-learning system, and the design of e-learning system with emotion detection module is given. At last, we propose the design of an experiment to evaluate the performance of this method in real e-learning system.

[1]  Junmo Kim,et al.  Development of deep learning-based facial expression recognition system , 2015, 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV).

[2]  M. Taner Eskil,et al.  Facial expression recognition based on anatomy , 2014, Comput. Vis. Image Underst..

[3]  Anastasios A. Economides,et al.  Affective Learning: Empathetic Agents with Emotional Facial and Tone of Voice Expressions , 2012, IEEE Transactions on Affective Computing.

[4]  Beverly Park Woolf,et al.  Building Intelligent Interactive Tutors: Student-centered Strategies for Revolutionizing E-learning , 2008 .

[5]  Qinghua Zheng,et al.  A Study of Learner-Oriented Negative Emotion Compensation in E-learning , 2014, J. Educ. Technol. Soc..

[6]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[7]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[8]  Abdellah Madani,et al.  Facial Expression Recognition Using Decision Trees , 2016, 2016 13th International Conference on Computer Graphics, Imaging and Visualization (CGiV).

[9]  Yong Man Ro,et al.  Intra-Class Variation Reduction Using Training Expression Images for Sparse Representation Based Facial Expression Recognition , 2014, IEEE Transactions on Affective Computing.

[10]  Rosa M. Carro,et al.  Sentiment analysis in Facebook and its application to e-learning , 2014, Comput. Hum. Behav..

[11]  Samuel W. K. Chan,et al.  Sentiment analysis in financial texts , 2017, Decis. Support Syst..

[12]  Mohammad Soleymani,et al.  Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection , 2016, IEEE Transactions on Affective Computing.

[13]  Xiaogang Wang,et al.  DeepID3: Face Recognition with Very Deep Neural Networks , 2015, ArXiv.

[14]  M. den Uyl,et al.  The FaceReader: Online facial expression recognition , 2006 .

[15]  Wioleta Szwoch,et al.  Comparison of selected off-the-shelf solutions for emotion recognition based on facial expressions , 2016, 2016 9th International Conference on Human System Interactions (HSI).

[16]  Shiguang Shan,et al.  AU-inspired Deep Networks for Facial Expression Feature Learning , 2015, Neurocomputing.

[17]  G. Ram Mohana Reddy,et al.  An E-Learning System with Multifacial Emotion Recognition Using Supervised Machine Learning , 2015, 2015 IEEE Seventh International Conference on Technology for Education (T4E).

[18]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[19]  Chih-Hsiang Ko,et al.  Applying FaceReader to Recognize Consumer Emotions in Graphic Styles , 2017 .

[20]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[21]  J.-M. Sun,et al.  Facial emotion recognition in modern distant education system using SVM , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[22]  Carlos Busso,et al.  Emotion recognition using a hierarchical binary decision tree approach , 2011, Speech Commun..

[23]  Richard M. Schwartz,et al.  Fast and Robust Neural Network Joint Models for Statistical Machine Translation , 2014, ACL.

[24]  Zhiyong Feng,et al.  Facial expression recognition via deep learning , 2014, 2014 International Conference on Smart Computing.

[25]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[26]  Dong Yu,et al.  Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[27]  K. O’Regan.  EMOTION AND E-LEARNING , 2019, Online Learning.