Robust Speech Understanding for Multi-Modal Human-Robot Communication

In order to model complex human robot interaction researchers not only have to consider different tasks but also to handle the complex interplay of different modules of one single robot system. In our context we constructed a robot assistant integrated in a home or office environment. We allow for a fairly natural communication style, which means that the users communicate using speech but are also allowed to use gestures and moreover to use contextual scene knowledge. Against this background, this paper presents a robust speech understanding component for situated human-robot communication. It serves as interface between speech recognition and dialog management. To increase robustness of speech processing it rates the speech recognition output by means of semantic coherence. Even if the recognized word-stream is not grammatically correct the speech understanding component provides semantic interpretations in context of multi-modal input for dialog management. For the understanding process, we designed special semantic concepts grounded to the domain of situated communication. They also provide additional information about the dialog act. A processing mechanism uses these concept units to generate the most likely semantic interpretation of the utterances