Processing instructions

Instructions play an important role in everyday communication, e.g. in task-oriented dialogs. Based on a (psycho-)linguistic theoretical background, which classifies instructions as requests, we conducted experiments using a cross-modal experimental design in combination with a reaction time paradigm in order to get insights in human instruction processing. We concentrated on the interpretation of basic single sentence instructions. Here, we especially examined the effects of the specificity of verbs, object names, and prepositions in interaction with factors of the visual object context regarding an adequate reference resolution. We were able to show that linguistic semantic and syntactic factors as well as visual context information context influence the interpretation of instructions. Especially the context information proves to be very important. Above and beyond the relevance for basic research, these results are also important for the design of human-computer interfaces capable of understanding natural language. Thus, following the experimentalsimulative approach, we also pursued the processing of instructions from the perspective of computer science. Here, a natural language processing interface created for a virtual reality environment served as basis for the simulation of the empirical findings. The comparison of human vs. virtual system performance using a local performance measure for instruction understanding based on fuzzy constraint satisfaction led to further insights concerning the complexity of instruction processing in humans and artificial systems. Using selected examples, we were able to show that the visual context has a comparable influence on the performance of both systems, whereas this approach is limited when it comes to explaining some effects due to variations of the linguistic structure. In order to get deeper insights into the timing and interaction of the sub-processes relevant for instruction understanding and to model these effects in the computer simulation, more specific data on human performance are necessary, e.g. by using eye-tracking techniques. In the long run, such an approach will result in the development of a more natural and cognitively adequate human-computer interface.

[1]  Marc Erich Latoschik,et al.  Resolving object references in multimodal dialogues for immersive virtual environments , 2004, IEEE Virtual Reality 2004.

[2]  Bernhard Jung,et al.  Task-Level Assembly Modeling in Virtual Environments , 2003, ICCSA.

[3]  Marc Erich Latoschik.  Designing transition networks for multimodal VR-interactions using a markup language , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[4]  M. Tanenhaus,et al.  Circumscribing Referential Domains during Real-Time Language Comprehension , 2002 .

[5]  O. Wilhelm,et al.  Effects of directionality in deductive reasoning: I. The comprehension of single relational premises. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[6]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[7]  G. Rickheit,et al.  Empirische Untersuchungen zur Rezeption von Handlungsanweisungen : der Einfluss semantischer und kontextueller Faktoren , 1999 .

[8]  Julie C. Sedivy,et al.  Integration of visuospatial and linguistic information: language comprehension in real time and real space , 1998 .

[9]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[10]  M. A. Britt,et al.  The Interaction of Referential Ambiguity and Argument Structure in the Parsing of Prepositional Phrases , 1994 .

[11]  Janet R. Meyer.  Fluency in the Production of Requests: Effects of Degree of Imposition, Schematicity and Instruction Set , 1992 .

[12]  G. Miller,et al.  Semantic networks of english , 1991, Cognition.

[13]  G. Kasper,et al.  Cross-Cultural Pragmatics: Requests and Apologies , 1991 .

[14]  P. Winterhoff-Spurk,et al.  Your Smile is My Command: Interaction between Verbal and Nonverbal Components of Requesting Specific to Situational Characteristics , 1988 .

[15]  S. Blum-Kulka.  Indirectness and politeness in requests: Same or different? , 1987 .

[16]  C. Clifton,et al.  The independence of syntactic processing , 1986 .

[17]  T. Herrmann,et al.  Speech and Situation: A Psychological Conception of Situated Speaking , 1983 .

[18]  L. Harris,et al.  Spatial direction and grammatical form of instructions affect the solution of spatial problems , 1975, Memory & cognition.

[19]  William A. Woods,et al.  Computational Linguistics Transition Network Grammars for Natural Language Analysis , 2022 .

[20]  J. Huttenlocher,et al.  Comprehension and a statement's relation to the situation it describes , 1968 .

[21]  Petra Weiß,et al.  Das Benennen von Objekten , 2003 .

[22]  P. Weiß.  „Schraub’ in“ oder „schraub’ an“ — Präpositionen bei der Verarbeitung von Handlungsanweisungen , 2001 .

[23]  Evelyn C. Ferstl,et al.  Das Lesen von Wörtern und Sätzen , 1999 .

[24]  G. Rickheit,et al.  Verb-, Objekt- und Kontextinformation bei der Rezeption von Handlungsanweisungen , 1999 .

[25]  G. Miller,et al.  A Semantic Network of English Verbs , 1998 .

[26]  Eckard Rolf.  Illokutionäre Kräfte : Grundbegriffe der Illokutionslogik , 1997 .

[27]  R. Mangold,et al.  Bunt gemeint, doch farblos gesagt. Wann wird die Farbe eines Objektes nicht benannt? , 1997 .

[28]  M. González.  Politeness: some universals in language usage , 1995 .

[29]  M. Gernsbacher.  Cognitive processes and mechanisms in language comprehension : the structure building framework , 1991 .

[30]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .

[31]  Mark Steedman,et al.  On not being led up the garden path : The use of context by the psychological syntax processor , 1985 .

[32]  T. Herrmann,et al.  Speech and Situation: A General Model for the Process of Speech Production , 1985 .

[33]  Götz Hindelang,et al.  Auffordern : die Untertypen des Aufforderns und ihre sprachlichen Realisierungsformen , 1978 .

[34]  E. Goffman.  Interaction Ritual: Essays on Face-To-Face Behavior , 1967 .