An Agent-Based Multimodal Interface for Sketch Interpretation

We present a multimodal interface for sketch interpretation that relies on a multi-agent architecture. The design of the interpretation engine and the different agents are based on a user-centered approach where efficiency measure is defined as user satisfaction. So far, several graphical agents have been implemented for recognizing basic graphical objects (e.g. lines, circles, etc.) as well as more complex (e.g., hatches, stairs, captions, etc) in architectural design. Besides, vocal agents have been developed for recognizing spoken annotations (e.g. dimensions) and interface commands. Realistic evaluations with professional users have demonstrated the potential interest of the proposed system

[1]  Sharon L. Oviatt,et al.  Multimodal Integration - A Statistical View , 1999, IEEE Trans. Multim..

[2]  M. Eskenazi,et al.  The French language database: Defining, planning, and recording a large database , 1984, ICASSP.

[3]  Christophe Ris,et al.  Use of acoustic prior information for confidence measure in ASR applications , 2001, INTERSPEECH.

[4]  Steve Renals,et al.  Confidence measures from local posterior probability estimates , 1999, Comput. Speech Lang..

[5]  Christine Alvarado,et al.  Dynamically constructed Bayes nets for multi-domain sketch understanding , 2005, IJCAI.

[6]  Alex Acero,et al.  Spoken Language Processing: A Guide to Theory, Algorithm and System Development , 2001 .

[7]  Maxine Eskénazi,et al.  BREF, a large vocabulary spoken corpus for French , 1991, EUROSPEECH.

[8]  Nicholas E. Matsakis.  Recognition of Handwritten Mathematical Expressions , 1999 .

[9]  Michele Risi,et al.  A Parsing Technique for Sketch Recognition Systems , 2004, 2004 IEEE Symposium on Visual Languages - Human Centric Computing.

[10]  Erkki Oja,et al.  On-line adaptation in recognition of handwritten alphanumeric characters , 1999, Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cat. No.PR00318).

[11]  Pierre Leclercq,et al.  A freehand-sketch environment for architectural design supported by a multi-agent system , 2005, Comput. Graph..

[12]  Stéphane Safin,et al.  Portable tool for finalizing freehand drawings: activity analysis and design requirements , 2005 .

[13]  Hervé Bourlard,et al.  Connectionist Speech Recognition: A Hybrid Approach , 1993 .

[14]  Randall Davis,et al.  Speech and sketching for multimodal design , 2004, IUI '04.