Ecological Interfaces: Extending the Pointing Paradigm by Visual Context

Following the ecological approach to visual perception, this paper presents an innovative framework for the design of multimodal systems. The proposal emphasises the role of the visual context on gestural communication. It is aimed at extending the concept of affordances to explain referring gesture variability. The validity of the approach is confirmed by results of a simulation experiment. A discussion of practical implications of our findings for software architecture design is presented.

[1]  Antonella De Angeli,et al.  Acting on a visual world : the role of perception in multimodal HCI , 1998, AAAI 1998.

[2]  Antonella De Angeli,et al.  Visual display, pointing, and natural language: the power of multimodal interaction , 1998, AVI '98.

[3]  M. Wertheimer.  Untersuchungen zur Lehre von der Gestalt. II , 1923 .

[4]  Sharon L. Oviatt,et al.  Unification-based Multimodal Integration , 1997, ACL.

[5]  Michael Streit.  Active and Passive Gestures - Problems with the Resolution of Deictic and Elliptic Expressions in a Multimodal System , 1997 .

[6]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[7]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[8]  Willem J. M. Levelt,et al.  Pointing and voicing in deictic expressions , 1985 .

[9]  Max Wertheimer,et al.  Untersuchungen zur Lehre von der Gestalt , .

[10]  Frédéric Wolff.  Analyse contextuelle des gestes de désignation en dialogue homme-machine , 1999 .

[11]  Sharon L. Oviatt,et al.  A rapid semi-automatic simulation technique for investigating interactive speech and handwriting , 1992, ICSLP.

[12]  Joëlle Coutaz,et al.  A design space for multimodal systems: concurrent processing and data fusion , 1993, INTERCHI.

[13]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[14]  Nigel Gilbert,et al.  Simulating speech systems , 1991 .

[15]  Mark A. McDaniel,et al.  Mental models, pictures, and text: Integration of spatial and verbal information , 1992, Memory & cognition.

[16]  Kristinn R. Thórisson,et al.  Simulated Perceptual Grouping: An Application to Human-Computer Interaction , 2019, Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society.

[17]  M. S. Mayzner,et al.  Cognition And Reality , 1976 .

[18]  Sharon Oviatt,et al.  Multimodal interactive maps: designing for human performance , 1997 .

[19]  P. Feyereisen.  La compréhension des gestes référentiels , 1997 .