Referring in Multimodal Systems: The Importance of User Expertise and System Features

This paper empirically investigates how humans use reference in space when interacting with a multimodal system able to understand written natural language and pointing with the mouse. We verified that user expertise plays an important role in the use of multimodal systems: experienced users performed 84% multimodal inputs while inexpert only 30%. Moreover experienced are able to efficiently use multimodality shortening the written input and transferring part of the reference meaning on the pointing. Results showed also the importance of the system layout: when very short labels (one character) are available users strongly adopt a redundant reference strategy, i.e. they referred to the object in a linguistic way and use pointing too. Starting from these facts some guidelines for future multimodal systems are suggested.

[1]  Willem J. M. Levelt,et al.  Pointing and voicing in deictic expressions , 1985 .

[2]  Arne Jönsson,et al.  Talking to a Computer Is Not like Talking to Your Best Friend , 1988, SCAI.

[3]  Sharon L. Oviatt,et al.  Integration themes in multimodal human-computer interaction , 1994, ICSLP.

[4]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[5]  Nigel Gilbert,et al.  Simulating speech systems , 1991 .

[6]  Sharon L. Oviatt,et al.  A rapid semi-automatic simulation technique for investigating interactive speech and handwriting , 1992, ICSLP.

[7]  Antonella De Angeli,et al.  VALUTARE I SISTEMI FLESSIBILI: UN APPROCCIO GLOBALE ALLA HCI , 1997 .

[8]  B. Buxton.  The “Natural” Language Of Interaction: A Perspective On Non-Verbal Dialogues , 1989 .

[9]  Ronan G. Reilly,et al.  Discourse Theory and Interface Design: The Case of Pointing with the Mouse , 1990, Int. J. Man Mach. Stud..

[10]  James D. Hollan,et al.  Direct Manipulation Interfaces , 1985, Hum. Comput. Interact..

[11]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[12]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[13]  Sharon L. Oviatt,et al.  Multimodal interfaces for dynamic interactive maps , 1996, CHI.

[14]  Jakob Nielsen,et al.  The Anti-Mac interface , 1996, CACM.

[15]  Sharon L. Oviatt,et al.  Predicting spoken disfluencies during human-computer interaction , 1995, Comput. Speech Lang..

[16]  Michael Böttner,et al.  Natural Language , 1997, Relational Methods in Computer Science.

[17]  Arne Jönsson,et al.  Empirical Studies Of Discourse Representations For Natural Language Interfaces , 1989, EACL.

[18]  Mark A. McDaniel,et al.  Mental models, pictures, and text: Integration of spatial and verbal information , 1992, Memory & cognition.

[19]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[20]  木村 和夫.  Pragmatics , 1997, Language Teaching.