Eye-tracking evidence for multimodal language-graphics comprehension: The role of integrated conceptual representations

In this paper, we propose a computational architecture for multimodal comprehension of text and graphics. A theoretical account of the integrated conceptual structures induced by linguistic and graphical entities is presented. We exemplify these structures with the analysis of an excerpt from a report published by Point Reyes Bird Observatory (PRBO). Experimental evidence, based on the analyses of subject’s eye movement recordings was evaluated under the framework of the architecture.

[1]  Richard Mayer,et al.  Multimedia Learning , 2001, Visible Learning Guide to Student Achievement.

[2]  Priti Shah,et al.  The Comprehension of Quantitative Information in Graphical Displays. , 2005 .

[3]  M. Tanenhaus,et al.  Introduction to the special issue on language–vision interactions , 2007 .

[4]  Wilfried Brauer,et al.  Spatial Cognition III , 2003, Lecture Notes in Computer Science.

[5]  R. Jackendoff.  Linguistics in Cognitive Science: The state of the art , 2007 .

[6]  Tina Seufert.  Supporting Coherence Formation in Learning from Multiple Representations , 2003 .

[7]  Adrian Staub,et al.  Eye movements and on-line comprehension processes , 2007 .

[8]  R. Jackendoff.  The architecture of the linguistic-spatial interface , 1996 .

[9]  Lars Kulik,et al.  Lexical Specifications of Paths , 2000, Spatial Cognition.

[10]  Steven Pinker,et al.  A theory of graph comprehension. , 1990 .

[11]  S. Kosslyn.  Understanding charts and graphs , 1989 .

[12]  Jana Holsanova.  Discourse, vision, and cognition , 2008 .

[13]  S. Ullman.  Visual routines , 1984, Cognition.

[14]  J. Wagemans,et al.  Segmentation of object outlines into parts: A large-scale integrative study , 2006, Cognition.

[15]  S. Levinson,et al.  LANGUAGE AND SPACE , 1996 .

[16]  Christopher Habel,et al.  On Reciprocal Improvement in Multimodal Generation : Co-reference by Text and Information Graphics * , 2007 .

[17]  Wolfgang Schnotz,et al.  The Cambridge Handbook of Multimedia Learning: An Integrated Model of Text and Picture Comprehension , 2005 .

[18]  Lars Kulik,et al.  A Geometric Agent Following Route Instructions , 2003, Spatial Cognition.