Detecting & interpreting self-manipulating hand movements for student’s affect prediction

BackgroundIn this paper, we report on development of a non-intrusive student mental state prediction system from his (her) unintentional hand-touch-head (face) movements.MethodsHand-touch-head (face) movement is a typical case of occlusion of otherwise easily detectable image features due to similar skin color and texture, however, in our proposed scheme, i.e., the Sobel-operated local binary pattern (SLBP) method using force field features. We code six different gestures of more than 100 human subjects, and use these codes as manual input to a three-layered Bayesian network (BN). The first layer holds mental state to gesture relationships obtained in an earlier study while the second layer embeds gesture and SLBP generated binary codes.ResultsWe find it very successful in separating hand (s) from face region in varying illuminating conditions. The proposed scheme when evaluated on a novel data set is found promising resulting with an accuracy of about 85%.ConclusionThe framework will be utilized for developing intelligent tutoring system.

[1]  Anne Miller,et al.  Video-Cued Recall: Its use in a Work Domain Analysis , 2004 .

[2]  P. Ekman.  The argument and evidence about universals in facial expressions of emotion. , 1989 .

[3]  Jochen Triesch,et al.  Classification of hand postures against complex backgrounds using elastic graph matching , 2002, Image Vis. Comput..

[4]  Cristina Conati,et al.  Probabilistic assessment of user's emotions in educational games , 2002, Appl. Artif. Intell..

[5]  Baochang Zhang,et al.  Sobel-LBP , 2008, 2008 15th IEEE International Conference on Image Processing.

[6]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[7]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  K. Wentzel.  Student motivation in middle school: The role of perceived pedagogical caring. , 1997 .

[9]  Scotty D. Craig,et al.  AutoTutor Detects and Responds to Learners Affective and Cognitive States , 2008 .

[10]  Roddy.  EMOTION-ORIENTED COMPUTING: STATE of the ART and KEY CHALLENGES , 2006 .

[11]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[12]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  A. Rogier.  [Communication without words]. , 1971, Tijdschrift voor ziekenverpleging.

[14]  Mubarak Shah,et al.  Resolving Hand over Face Occlusion , 2005, ICCV-HCI.

[15]  Abdolhossein Sarrafzadeh,et al.  Making sense of student use of nonverbal cues for intelligent tutoring systems , 2005, OZCHI.

[16]  Rana El Kaliouby,et al.  Viewing Student Affect and Learning through Classroom Observation and Physical Sensors , 2008, Intelligent Tutoring Systems.

[17]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[18]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Yoshiaki Shirai,et al.  Hand shape estimation under complex backgrounds for sign language recognition , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[20]  Takeaki Uno,et al.  Student mental state inference from unintentional body gestures using dynamic Bayesian networks , 2010, Journal on Multimodal User Interfaces.

[21]  H. Wallbott.  Bodily expression of emotion , 1998 .

[22]  Rosalind W. Picard.  Affective Computing , 1997 .

[23]  Amy Price.  What Are Confidence Intervals? Can They Be Made Simple , 2013 .

[24]  Thomas S. Huang,et al.  Tracking articulated hand motion with eigen dynamics analysis , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[25]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[26]  Mann Oo. Hay.  Emotion recognition in human-computer interaction , 2012 .

[27]  A. Pease,et al.  The Definitive Book of Body Language , 2004 .

[28]  M. D. Meijer.  The contribution of general features of body movement to the attribution of emotions , 1989 .

[29]  Rana El Kaliouby,et al.  Towards Communicative Face Occlusions: Machine Detection of Hand-over-Face Gestures , 2009, ICIAR.

[30]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Sidney K. D'Mello,et al.  What Are You Feeling? Investigating Student Affective States During Expert Human Tutoring Sessions , 2008, Intelligent Tutoring Systems.

[32]  Lynne M Connelly.  What are confidence intervals? , 2013, Medsurg nursing : official journal of the Academy of Medical-Surgical Nurses.

[33]  Mubarak Shah,et al.  Recognizing Hand Gestures , 1994, ECCV.