Improving machine-learned detection of miscommunications in human-machine dialogues through informed data splitting

In this paper we study two types of machine learning techniques, rule-induction and memorybased learning, for error detection in spoken dialogue systems. The learners are trained and tested on two tasks: predicting whether the current user utterance will cause problems, and identifying whether the previous user utterance has caused a problem in the ongoing dialogue. We focus on a variety of features readily available in the majority of spoken dialogue systems: dialogue history, recognized words, and prosodic characteristics of the user input. We find that the learners gain relatively little from the inclusion of prosodic features, even though at first sight the general prosodic trends in our corpus are in agreement with earlier observations from the literature. A closer inspection of the data reveals that the prosodic feature values are highly dependent on the problem’s context, represented by the most recently asked system question type. As a consequence, when separate classifiers are trained on subsets of the data that are split by system question type, the learners profit much more from prosodic information. It is shown that such an informed splitting is beneficial for our other feature sets as well. The consequences of this approach for error detection are discussed.