Java Tutoring System with Facial and Text Emotion Recognition

This paper presents the design and implementation of an intelligent tutoring system (ITS) for teaching JAVA, which can recognize the user's emotional state through facial expressions and textual dialogues. For facial emotion recognition we implemented a neural network with WEKA library and a facial feature extractor with OPENCV library. The ITS applies a semantic algorithm (ASEM) to extract textual emotions through dialogues, which has shown a degree of assertiveness of 80% in tests for graduate students. In addition, the tutor uses a set of fuzzy rules to determine the complexity of the next exercise, considering the program implementation time, program executions and compilations, and current difficulty level.

[1]  Diana Inkpen,et al.  Using a Heterogeneous Dataset for Emotion Analysis in Text , 2011, Canadian Conference on AI.

[2]  Arthur C. Graesser,et al.  AutoTutor: an intelligent tutoring system with mixed-initiative dialogue , 2005, IEEE Transactions on Education.

[3]  K. Scherer,et al.  Emotion recognition from expressions in face, voice, and body: the Multimodal Emotion Recognition Test (MERT). , 2009, Emotion.

[4]  Arthur C. Graesser,et al.  Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features , 2010, User Modeling and User-Adapted Interaction.

[5]  Nicu Sebe,et al.  MULTIMODAL EMOTION RECOGNITION , 2005 .

[6]  Chung-Hsien Wu,et al.  Emotion recognition from text using semantic labels and separable mixture models , 2006, TALIP.

[7]  Roman V. Belavkin,et al.  THE ROLE OF EMOTION IN PROBLEM SOLVING , 2001 .

[8]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[9]  Luis Enrique Sucar,et al.  Building an Affective Model for Intelligent Tutoring Systems with Base on Teachers' Expertise , 2008, MICAI.

[10]  Vidyasagar Potdar,et al.  Computational approaches for emotion detection in text , 2010, 4th IEEE International Conference on Digital Ecosystems and Technologies.

[11]  Arthur C. Graesser,et al.  AutoTutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back , 2012, TIIS.