Are You Looking at Me, Are You Talking with Me: Multimodal Classification of the Focus of Attention

Automatic dialogue systems get easily confused if speech is recognized which is not directed to the system Besides noise or other people's conversation, even the user's utterance can cause difficulties when he is talking to someone else or to himself (“Off-Talk”) In this paper the automatic classification of the user's focus of attention is investigated In the German SmartWeb project, a mobile device is used to get access to the semantic web In this scenario, two modalities are provided – speech and video signal This makes it possible to classify whether a spoken request is addressed to the system or not: with the camera of the mobile device, the user's gaze direction is detected; in the speech signal, prosodic features are analyzed Encouraging recognition rates of up to 93 % are achieved in the speech-only condition Further improvement is expected from the fusion of the two information sources.

[1]  Norbert Reithinger,et al.  A look under the hood: design and development of the first SmartWeb system demonstrator , 2005, ICMI '05.

[2]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[3]  Wolfgang Wahlster,et al.  SmartWeb: Mobile Applications of the Semantic Web , 2004, GI Jahrestagung.

[4]  Shumeet Baluja,et al.  Efficient face orientation discrimination , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[5]  Nicole Beringer,et al.  Off-talk - a problem for human-machine-interaction? , 2001, INTERSPEECH.

[6]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[7]  Tanja Schultz,et al.  Identifying the addressee in human-human-robot interactions based on head pose and speech , 2004, ICMI '04.

[8]  Elmar Nöth,et al.  Prosodic Classification of Offtalk: First Experiments , 2002, TSD.

[9]  Joachim Denzler,et al.  A Comparative Evaluation of Template and Histogram Based 2D Tracking Algorithms , 2005, DAGM-Symposium.