From visual perception to multimodal communication: Incremental route descriptions

In the last few years in cognitive science there has been a growing interest in the connection between visual perception and natual language. The question of interest is: How ca we discuss what we see? With this question in mind in this article we will look at the area of incremental route descriptions. Here, a speaker step-by-step presents the relevant rout information in a 3D-environment. The speaker must adjust his/her descriptions to the currently visible objects. Two major questions arise in this context: 1. How is visually obtained informaiton used in natural language generation? and 2. How are the modalities coordniated? We will present a computational framework for the interaction of visual perception and natural language descriptions which integrates several processes and representations. Especially discussed is the interaction between the spatial representation and the presentation representation used for natural language descriptions. We have implemented a prototypical version of the proposed model, called MOSES.

[1]  Benjamin Kuipers,et al.  Representing Knowledge of Large-scale Space , 1977 .

[2]  Gordon McCalla,et al.  The Execution of Plans in an Independent Dynamic Microworld , 1979, IJCAI.

[3]  Lynn A. Streeter,et al.  How to Tell People Where to Go: Comparing Navigational Aids , 1985, Int. J. Man Mach. Stud..

[4]  Jörg R. J. Schirra.  Einige Überlegungen zu Bildvorstellungen in kognitiven Systemen , 1990, Repräsentation und Verarbeitung räumlichen Wissens.

[5]  Henry Kautz,et al.  A model of naive temporal reasoning , 1985 .

[6]  Barbara Hayes-Roth,et al.  Differences in spatial knowledge acquired from maps and navigation , 1982, Cognitive Psychology.

[7]  Aravind K. Joshi.  Factoring Recursion and Dependencies: an Aspect of Tree Adjoining Grammars (Tag) and a Comparison of Some Formal Properties of Tags, GPSGs, Plgs, and LPGS , 1983, ACL.

[8]  Gerd Herzog,et al.  VITRA GUIDE : Utilisation du Langage Naturel et de Représentations Graphiques pour la Description d'Itinéraires , 1993 .

[9]  Walter Schneider,et al.  Controlled and Automatic Human Information Processing: 1. Detection, Search, and Attention. , 1977 .

[10]  M. S. Mayzner,et al.  Cognition And Reality , 1976 .

[11]  Michael E. Lesk,et al.  Route Finding in Street Maps by Computers and People , 1982, AAAI.

[12]  Barbara Hayes-Roth,et al.  A Cognitive Model of Planning , 1979, Cogn. Sci..

[13]  B. Landau,et al.  “What” and “where” in spatial language and spatial cognition , 1993 .

[14]  Tadasu Oyama,et al.  Visual Space Perception , 1962 .

[15]  Klaus-Peter Gapp.  Basic Meanings of Spatial Relations: Computation and Evaluation in 3D Space , 1994, AAAI.

[16]  H. Barlow.  Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[17]  R. Klatzky,et al.  Navigator: A psychologically based model of environmental learning through navigation , 1989 .

[18]  J. Piaget,et al.  Child's Conception Of Geometry , 1960 .

[19]  Richard E. Korf,et al.  Real-Time Heuristic Search , 1990, Artif. Intell..

[20]  G. Miller,et al.  Cognitive science. , 1981, Science.

[21]  Tommy Gärling,et al.  The role of cognitive maps in spatial decisions , 1989 .

[22]  Wai-Kiang Yeap.  Towards a Computational Theory of Cognitive Maps , 1988, Artif. Intell..

[23]  I. Altman,et al.  Handbook of environmental psychology , 1987 .

[24]  Christopher Habel,et al.  Prozedurale Aspekte der Wegplanung und Wegbeschreibung , 1987, LILOG-Report.

[25]  B. Kuipers.  Modelling spatial knowledge , 1977, IJCAI 1977.

[26]  M. Gluck.  Making Sense of Human Wayfinding: Review of Cognitive and Linguistic Knowledge for Personal Navigation with a New Research Direction , 1991 .

[27]  S. Kosslyn.  Seeing and imagining in the cerebral hemispheres: a computational approach. , 1987, Psychological review.

[28]  E. Tolman.  Cognitive maps in rats and men. , 1948, Psychological review.

[29]  W. Klein.  Local deixis in route directions , 1982 .

[30]  David Lewis,et al.  OBSERVATIONS ON ROUTE FINDING AND SPATIAL ORIENTATION AMONG THE ABORIGINAL PEOPLES OF THE WESTERN DESERT REGION OF CENTRAL AUSTRALIA , 1976 .

[31]  Dieter Wunderlich,et al.  How to get there from here , 1982 .

[32]  R. Jackendoff.  On beyond Zebra: The relation of linguistic and visual information , 1987, Cognition.

[33]  Barbara J. Grosz,et al.  Focusing and Description in Natural Language Dialogues , 1979 .

[34]  Wolfgang Wahlster,et al.  WIP: Integrating text and graphics design for adaptive information presentation , 1992 .

[35]  Janice I. Glasgow,et al.  THE IMAGERY DEBATE REVISITED: A COMPUTATIONAL PERSPECTIVE , 1993 .

[36]  P. Thorndyke,et al.  Spatial learning and reasoning skill , 1981 .

[37]  Wolfgang Maass.  A Cognitive Model for the Process of Multimodal, Incremental Route Descriptions , 1993, COSIT.