MODALITY THEORY IN SUPPORT OF MULTIMODAL INTERFACE DESIGN

Modality Theory is a comparatively new field of investigation which addresses the following general problem of mapping task domain information into interactive multimodal interfaces: given any particular set of information which needs to be exchanged between user and system during task performance in context, identify the input/ output modalities which constitute an optimal solution to the representation and exchange of that information. This paper proposes a research agenda for Modality Theory and presents two steps towards its implementation. The first is a generative taxonomy of output modalities covering the media of graphics, sound and touch. The second is a methodology for carrying out information-mapping in design practice. As it matures, Modality Theory promises to provide useful support to contemporary designers of interactive human-computer interfaces who have begun to use a rapidly increasing number of different, and often alternative input/output modalities for the expression and exchange of information between systems and their users.

[1]  Jon Oberlander,et al.  Reasoning with Words, Pictures and Calculi: Computation Versus Justification , 1991 .

[2]  B. Marx.  The Visual Display of Quantitative Information , 1985 .

[3]  Phil Barnard,et al.  Bridging between basic theories and the artifacts of human-computer interaction , 1991 .

[4]  D. Rumelhart,et al.  Philosophy and Connectionist Theory , 1991 .

[5]  John M. Carroll,et al.  Designing Interaction: Psychology at the Human-Computer Interface , 1991 .

[6]  Edward R. Tufte,et al.  Envisioning Information , 1990 .

[7]  S Buckingham Shum,et al.  Developing the Design Space with Design Space Analysis , 1993 .

[8]  Michael Twyman,et al.  A Schema for the Study of Graphic Language (Tutorial Paper) , 1979 .

[9]  G. Duncan,et al.  Speaking with computers: a multimodal approach , 1993, EUROSPEECH.

[10]  Mary Beth Rosson,et al.  Getting around the task-artifact cycle: how to make claims and design by scenario , 1992, TOIS.

[11]  Niels Ole Bernsen.  Structuring design spaces , 1993, CHI '93.

[12]  Mark T. Maybury,et al.  Intelligent multimedia interfaces , 1994, CHI Conference Companion.

[13]  Joëlle Coutaz,et al.  A design space for multimodal systems: concurrent processing and data fusion , 1993, INTERCHI.

[14]  Joëlle Coutaz,et al.  The MSM Framework: A Design Space for Multi-Sensori-Motor Systems , 1993, EWHCI.

[15]  John J. Bertin,et al.  The semiology of graphics , 1983 .

[16]  Niels Ole Bernsen,et al.  COSITUE: Towards a Methodology for Constructing Scenarios , 1993 .

[17]  Neff Walker,et al.  Classifying graphical information , 1991 .

[18]  Niels Ole Bernsen.  A RESEARCH AGENDA FOR MODALITY THEORY , 1993 .

[19]  John M. Carroll,et al.  Creating a Design Science of Human-Computer Interaction , 1992, Interact. Comput..