Emotion Recognition of Students Based on Facial Expressions in Online Education Based on the Perspective of Computer Simulation

Online education has developed rapidly due to its irreplaceable convenience. Under the severe circumstances caused by COVID-19 recently, many schools around the world have delayed opening and adopted online education as one of the main teaching methods. However, the efficiency of online classes has long been questioned. Compared with traditional face-to-face classes, there is a lack of direct, timely, and effective communication and feedback between teachers and students in the online courses. Previous studies have shown that there is a close and stable relationship between a person’s facial expressions and emotions generally. From the perspective of computer simulation, a framework combining a face expression recognition (FER) algorithm with online courses platforms is proposed in this work. The cameras in the devices are used to collect students’ face images, and the facial expressions are analyzed and classified into 8 kinds of emotions by the FER algorithm. An online course containing 27 students conducted on Tencent Meeting is used to test the proposed method, and the result proved that this method performs robustly in different environments. This framework can also be applied to other similar scenarios such as online meetings.

[1]  Oladimeji Farri,et al.  Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification , 2019, Artif. Intell. Medicine.

[2]  Xiaolong Zhang,et al.  Speech Emotion Recognition Based on Decision Tree and Improved SVM Mixed Model , 2017 .

[3]  Michel Valstar,et al.  Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition , 2016 .

[4]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Gabriel Hermosilla,et al.  Reduced isothermal feature set for long wave infrared (LWIR) face recognition , 2017 .

[6]  Young Bin Kim,et al.  Efficiently detecting outlying behavior in video-game players , 2015, PeerJ.

[7]  Namita Mittal,et al.  Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy , 2019, The Visual Computer.

[8]  Dragan Ilic,et al.  Investigating the efficacy of practical skill teaching: a pilot-study comparing three educational methods , 2013, Advances in health sciences education : theory and practice.

[9]  Paul Ekman,et al.  The universality of a contempt expression: A replication , 1988 .

[10]  David Matsumoto,et al.  More evidence for the universality of a contempt expression , 1992 .

[11]  P. Ekman,et al.  A new pan-cultural facial expression of emotion , 1986 .

[12]  P. Ekman,et al.  Measuring facial movement , 1976 .

[13]  Naoyuki Kubota,et al.  Attention mechanism-based CNN for facial expression recognition , 2020, Neurocomputing.

[14]  Jie Shao,et al.  Three convolutional neural network models for facial expression recognition in the wild , 2019, Neurocomputing.

[15]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Eimear Dolan,et al.  An evaluation of online learning to teach practical competencies in undergraduate health science students , 2015, Internet High. Educ..

[17]  Yoshua Bengio,et al.  Challenges in Representation Learning: A Report on Three Machine Learning Contests , 2013, ICONIP.

[18]  Markus Flierl,et al.  Graph-Preserving Sparse Nonnegative Matrix Factorization With Application to Facial Expression Recognition , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  Maode Ma,et al.  Intelligent Image Recognition System for Marine Fouling Using Softmax Transfer Learning and Deep Convolutional Neural Networks , 2017, Complex..

[20]  Tianxu Zhang,et al.  Blind spectrum reconstruction algorithm with L0-sparse representation , 2015 .

[21]  Kamilia Kamardin,et al.  Offline Signature Verification using Deep Learning Convolutional Neural Network (CNN) Architectures GoogLeNet Inception-v1 and Inception-v3 , 2019, Procedia Computer Science.

[22]  Güray Tonguç,et al.  Automatic recognition of student emotions from facial expressions during a lecture , 2020, Comput. Educ..

[23]  Hao Zhang,et al.  FBRDLR: Fast blind reconstruction approach with dictionary learning regularization for infrared microscopy spectra , 2018 .

[24]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Robertas Damasevicius,et al.  A neuro-heuristic approach for recognition of lung diseases from X-ray images , 2019, Expert Syst. Appl..

[26]  Björn W. Schuller,et al.  Categorical and dimensional affect analysis in continuous input: Current trends and future directions , 2013, Image Vis. Comput..

[27]  Ishan Bhardwaj,et al.  A spoof resistant multibiometric system based on the physiological and behavioral characteristics of fingerprint , 2017, Pattern Recognit..

[28]  Carolyn L Cason,et al.  Performance outcomes of an online first aid and CPR course for laypersons , 2011 .

[29]  Juan Song,et al.  Simultaneous enhancement and noise reduction of a single low-light image , 2016, IET Image Process..

[30]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[31]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[32]  Bin Huang,et al.  Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture , 2020, Pattern Recognit. Lett..

[33]  Bin Zhou,et al.  A regional adaptive variational PDE model for computed tomography image reconstruction , 2019, Pattern Recognit..

[34]  Yangjie Wei,et al.  Multi-feature fusion for thermal face recognition , 2016 .

[35]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Zhengtao Li,et al.  Optical remote sensing image enhancement with weak structure preservation via spatially adaptive gamma correction , 2018, Infrared Physics & Technology.

[37]  Shuicheng Yan,et al.  Peak-Piloted Deep Network for Facial Expression Recognition , 2016, ECCV.

[38]  P. Ekman,et al.  Who knows what about contempt: A reply to Izard and Haynes , 1988 .

[39]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..