Conversations with graphics: implications for the design of natural language/graphics interfaces

The design of interfaces which support a user's natural cognitive processes and structures depends on an understanding of communicational codes as well as task structures etc. Research in human-computer interaction has, however, tended to neglect the former in favour of the latter. This paper seeks to redress this imbalance by reporting in detail the results of an empirical enquiry into how people use two communicational codes?natural language and drawing?to achieve a shared understanding of a problem (the redesign of a kitchen) and its solution. This enquiry clearly indicates the complex interdependency of these forms of communication when used in combination. While a graphical depiction may provide a context for linguistic interpretation, especially in respect of the disambiguation of spatial expressions, graphical expressions (pictures and drawings) themselves require a context-dependent interpretation which, itself, can derive from an accompanying natural language expression. Often, however, neither form of expression can be independently interpreted. Rather the meaning of the situation is dependent on the synergistic combination of both forms of expression and is heavily dependent on the common background knowledge of participants in the interaction. While natural language expressions may be explicitly linked to graphical depictions through pointing actions, such actions are not mandatory for effective communication. The implications of these observations for the design of natural language/graphics interfaces are discussed. Among the questions raised by the paper are: how to characterize the difference between representation or modelling and communication in graphics; how to apply current object-oriented theories of knowledge representation to the highly fluid yet knowledge-rich use of pictures that was observed in our study; and finally what differences might emerge between dialogues of this type in different domains.

[1]  W. Kintsch,et al.  Strategies of discourse comprehension , 1983 .

[2]  David Frohlich,et al.  Applying the Technology of Conversation to the Technology for Conversation , 1990 .

[3]  B. Chandrasekaran,et al.  Design considerations for Picture Production in a Natural Language graphics system , 1981, COMG.

[4]  Lawrence M. Fagan,et al.  Graphical specification of procedural knowledge for an expert system , 1987 .

[5]  Steve Whittaker,et al.  Co-ordinating activity: an analysis of interaction in computer-supported co-operative work , 1991, International Conference on Human Factors in Computing Systems.

[6]  Anita Kollerbaur,et al.  Visual Languages and Human Computer Interaction , 1987, BCS HCI.

[7]  Herbert A. Simon,et al.  The Structure of Ill Structured Problems , 1973, Artif. Intell..

[8]  J. Deręgowski.  Real space and represented space: Cross-cultural perspectives , 1989, Behavioral and Brain Sciences.

[9]  Henk Zeevat,et al.  Integrating natural language and graphics in dialogue , 1990, INTERACT.

[10]  Thomas R. G. Green,et al.  When do diagrams make good computer languages , 1979 .

[11]  G. C. Bes,et al.  Natural Language and Graphics Interface: Results and Perspectives from the ACORD Project , 1992 .

[12]  Wolfgang Wahlster,et al.  Combining Deictic Gestures and Natural Language for Referent Identification , 1986, COLING.

[13]  Jeannette G. Neal,et al.  An intelligent multi-media human-computer dialogue system , 1988 .

[14]  L. Carmichael,et al.  An experimental study of the effect of language on the reproduction of visually perceived form. , 1932 .

[15]  Nicholas Negroponte.  An iconoclastic view beyond the desktop metaphor , 1989, Int. J. Hum. Comput. Interact..

[16]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[17]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[18]  Jeannette G. Neal,et al.  Multi-Modal References in Human-Computer Dialogue , 1988, AAAI.

[19]  W. Winn.  7 The Design and Use of Instructional Graphics , 1989 .

[20]  Steven K. Feiner,et al.  Generating coordinated multimedia explanations , 1990, Sixth Conference on Artificial Intelligence for Applications.

[21]  Terry Winograd,et al.  Understanding computers and cognition - a new foundation for design , 1987 .

[22]  Aaron Sloman,et al.  Interactions Between Philosophy and Artificial Intelligence: The Role of Intuition and Non-Logical Reasoning in Intelligence , 1971, IJCAI.

[23]  Ewan Klein,et al.  Semantics and graphical information , 1990, INTERACT.

[24]  Bryan Lawson,et al.  How Designers Think: The Design Process Demystified , 1990 .

[25]  J. Bruner,et al.  Assimilation in the immediate reproduction of visually perceived figures. , 1952, Journal of experimental psychology.

[26]  Philip J. Hayes,et al.  Steps towards Integrating Natural Language and Graphical Interaction for Knowledge-Based Systems , 1986, European Conference on Artificial Intelligence.

[27]  R. Nickerson.  Some Characteristics of Conversations , 1981 .

[28]  Bob Kemp,et al.  Knowledge-Based Graphical Dialogue: A Strategy and Architecture , 1989 .

[29]  M. Just,et al.  Understanding Machines from Text and Diagrams. , 1989 .

[30]  Fausto Giunchiglia,et al.  From Descriptions to Images: What Reasoning in between? , 1984, ECAI.

[31]  Luis Alberto Pineda Cortes.  Graflog: a theory of semantics for graphics with applications to human-computer interaction and cad systems , 1989 .

[32]  Heinz Mandl,et al.  Knowledge Acquisition from Text and Pictures , 1989 .

[33]  G. Bower,et al.  Comprehension and memory for pictures , 1975, Memory & cognition.

[34]  M. McDaniel,et al.  Illustrations as adjuncts to prose: a text-appropriate processing approach , 1988 .

[35]  Marcia K. Johnson,et al.  Contextual prerequisites for understanding: Some investigations of comprehension and recall , 1972 .

[36]  R. Montague.  Formal philosophy; selected papers of Richard Montague , 1974 .

[37]  Hans Hörmann,et al.  The influence of general and specific verbal labels on the recognition of labeled and unlabeled parts of pictures , 1978 .

[38]  E. Rosch,et al.  Cognition and Categorization , 1980 .

[39]  Maryanne M. Gobble,et al.  Design Thinking , 2010, The Palgrave Encyclopedia of the Possible.

[40]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[41]  W. Howard Levie,et al.  Effects of text illustrations: A review of research , 1982 .

[42]  J. Feldman.  Four frames suffice: A provisional model of vision and space , 1985, Behavioral and Brain Sciences.

[43]  Brad A. Myers,et al.  Taxonomies of visual programming and program visualization , 1990, J. Vis. Lang. Comput..

[44]  Ewan Klein,et al.  GRAFLOG: Understanding Drawings through Natural Language , 1988, Comput. Graph. Forum.