Eucalyptus: Integrating Natural Language Input with a Graphical User Interface

Abstract : This report describes Eucalyptus, a natural language (NL) interface that has been integrated with the graphical user interface of the KOALAS Test Planning Tool, a simulated Naval air combat command system. The multimodal, multimedia interface handles both imperative commands and database queries (either typed or spoken into a microphone) while still allowing full use of the original graphical interface. In this way the precision and consistency of direct manipulation is balanced and augmented by the descriptive power and reduced redundancy of NL. The two input media used together yield such powerful interaction techniques as deixis (simultaneous speech and pointing) and the ability to use mouse clicks and verbal referring expressions interchangeably. Finally, the system's discourse handling capability allows abbreviated NL follow-ups (anaphora and ellipsis) to receive full interpretations based on the prior interaction context, whether verbal or graphical. Natural language processing, Human-computer interface, Speech recognition.

[1]  Stuart C. Shapiro,et al.  Intelligent Multi-Media Interface Technology , 1988, SGCH.

[2]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[3]  Ralph Grishman,et al.  Computational linguistics : an introduction , 1986 .

[4]  Elaine Marsh,et al.  Focus of Attention in Decision Support Systems. , 1995 .

[5]  Jerry R. Hobbs.  Resolving pronoun references , 1986 .

[6]  Jeannette G. Neal,et al.  Multi-Modal References in Human-Computer Dialogue , 1988, AAAI.

[7]  Stuart C. Shapiro,et al.  Intelligent Multi-Media Integrated Interface Project , 1990 .

[8]  Stephanie S. Everett,et al.  Talking to InterFIS: Adding Speech Input to a Natural Language Interface , 1992 .

[9]  Barbara J. Grosz,et al.  The Representation and Use of Focus in a System for Understanding Dialogs , 1977, IJCAI.

[10]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[11]  Wolfgang Wahlster,et al.  Combining Deictic Gestures and Natural Language for Referent Identification , 1986, COLING.

[12]  Jeannette G. Neal,et al.  An intelligent multi-media human-computer dialogue system , 1988 .

[13]  Craig W. Thompson,et al.  A toolkit for building “menu-based natural language” interfaces , 1985, ACM '85.

[14]  C. L. Barrett,et al.  Real time expert advisory systems : considerations and imperatives , 1990 .

[15]  Philip R. Cohen.  The role of natural language in a multimodal interface , 1992, UIST '92.

[16]  Norbert Reithinger,et al.  Bidirectional Use of Knowledge in the Multi-modal NL Access System XTRA , 1989, IJCAI.

[17]  Richard Montague,et al.  The Proper Treatment of Quantification in Ordinary English , 1973 .

[18]  Christopher Williamson,et al.  Dynamic queries: database searching by direct manipulation , 1992, CHI.

[19]  Dennis Perzanowski,et al.  InterFIS: A Natural Language Interface to the Fault Isolation Shell , 1990 .

[20]  Philip J. Hayes,et al.  Steps towards Integrating Natural Language and Graphical Interaction for Knowledge-Based Systems , 1986, European Conference on Artificial Intelligence.

[21]  Rachel Reichman-Adar,et al.  Extended Person-Machine Interface , 1984, Artif. Intell..

[22]  Elaine Marsh,et al.  Towards Friendlier User Interfaces for Expert Systems , 1995 .

[23]  Ralph Grishman,et al.  PROTEUS Parser Reference Manual. , 1986 .

[24]  Wolfgang Wahlster,et al.  User and discourse models for multimodal communication , 1991 .

[25]  Philip R. Cohen,et al.  Synergistic use of direct manipulation and natural language , 1989, CHI '89.

[26]  Arne Jönsson,et al.  Wizard of Oz studies -- why and how , 1993, Knowl. Based Syst..

[27]  M. Brady,et al.  Focusing in the Comprehension of Definite Anaphora , 1983 .

[28]  C. Y. Thielman,et al.  Natural Language with Integrated Deictic and Graphic Gestures , 1989, HLT.

[29]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[30]  Kenneth Wauchope.  A Tandem Semantic Interpreter for Incremental Parse Selection , 1990 .

[31]  David C. Brown,et al.  An experimental graphics system with natural language input , 1979, Comput. Graph..

[32]  Norbert Reithinger,et al.  XTRA: A Natural-Language Access System to Expert Systems , 1989, Int. J. Man Mach. Stud..

[33]  Susann LuperFoy,et al.  The Representation of Multimodal User Interface Dialogues Using Discourse Pegs , 1992, ACL.

[34]  Stuart C. Shapiro,et al.  Intelligent multi-media interface technology , 1991 .

[35]  Kenneth Wauchope,et al.  Eucalyptus: An Integrated Spoken Language/Graphical Interface for Human-Computer Dialogue. , 1991 .

[36]  James F. Allen.  Natural language understanding , 1987, Bejnamin/Cummings series in computer science.