Using word-level pitch features to better predict student emotions during spoken tutoring dialogues

Abstract In this paper, we advocate for the usage of word-level pitch features for detecting user emotional states during spoken tutoring dialogues. Prior research has primarily focused on the use of turn-level features as predictors. We compute pitchfeatures at the word level and resolve the problem of combining multiple features per turn using a word-levelemotion model . Even under a very simple word-level emotionmodel, our results show an improvement in prediction using word-level features over using turn-level features. We find that the advantage of word-level features lies in a betterprediction of longer turns. 1. Introduction We investigate the utility of using pitch features applied at the word level for the task of predicting student emotions in twocorpora of spoken tutoring dialogues. Motivation for thiswork comes from the performance gap between human tutorsand current machine tutors; typically students tutored byhuman tutors learn more than students tutored by computer tutors. One of the methods currently being explored as a wayof closing this gap is to incorporate affective reasoning intocurrent computer tutoring systems, including dialogue-based tutoring systems, e.g. [1, 2].Previous spoken dialogue research in other domains hasshown that turn-level prosodic, lexical, dialogue, and otherfeatures can be used to predict user emotional states [3-5]. Tobetter approximate the prosodic information [6] uses word-level features and successfully applies them to a different emotion detection task. To our knowledge, there is noprevious work that directly compares the impact of usingfeatures at the sub-turn rather than the turn level for emotionprediction. In this paper we are performing a first comparisonof the two levels for the task of detecting student emotional states.There are many choices for sub-turn units (breath groups, intonational phrases, syntactic chunks, words, syllables). Wewill use words as our sub-turn units because it is straightforward to do the segmentation and because theseunits have been used successfully by other researchers for similar tasks [6]. Moreover, in a real-time dialogue system,the segmentation is available as a byproduct of the automatic speech recognition. To simplify our word versus turn-level featurecomparison, we will focus