Software Implications for Multimodal User Interfaces

This paper discusses software considerations for multimodal user interfaces, that is, systems able to support human-to-human modalities of communication (such as gesture and natural language). A definition and a classification of multimodal systems is proposed and the distinction between multimodal and multimedia systems is clarified. Then multiagent models and techniques used in graphical user interfaces are compared with concepts and approaches developed in computer vision and natural language processing. This analysis leads the way to the definition of a general framework aimed at the integration of multiple modalities.

[1]  Leonard J. Bass,et al.  Developing software for the user interface , 1991, The SEI series in software engineering / Software Engineering Institute.

[2]  Ralph D. Hill,et al.  Supporting concurrency, communication, and synchronization in human-computer interaction—the Sassafras UIMS , 1986, TOGS.

[3]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[4]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[5]  J. Nanard.  La Manipulation directe en interface homme-machine , 1990 .

[6]  Roger B. Dannenberg,et al.  Garnet: comprehensive support for graphical, highly interactive user interfaces , 1990, Computer.

[7]  William W. Gaver.  Auditory Icons: Using Sound in Computer Interfaces , 1986, Hum. Comput. Interact..

[8]  K WallaceGregory.  The JPEG still picture compression standard , 1991 .

[9]  Mark S. Ackerman,et al.  A construction set for multimedia applications , 1989, IEEE Software.

[10]  Paul R. Calder,et al.  Composing user interfaces with InterViews , 1989, Computer.

[11]  Adele Goldberg,et al.  Smalltalk-80 - the interactive programming environment , 1984 .

[12]  Mark S. Ackerman,et al.  Augmenting a window system with speech input , 1990, Computer.

[13]  Jeff Conklin,et al.  Hypertext: An Introduction and Survey , 1987, Computer.

[14]  Geoffrey E. Hinton,et al.  Building adaptive interfaces with neural networks: The glove-talk pilot study , 1990, INTERACT.

[15]  Mark W. Salisbury,et al.  Talk and draw: bundling speech and graphics , 1990, Computer.

[16]  James D. Hollan,et al.  Direct Manipulation Interfaces , 1985, Hum. Comput. Interact..

[17]  Roger B. Dannenberg,et al.  Garnet: comprehensive support for graphical, highly interactive user interfaces , 1995 .