Inspecting Pictures for Information to Verify a Sentence: Eye Movements in General Encoding and in Focused Search

When we see combinations of text and graphics, such as photographs and their captions in printed media, how do we compare the information in the two components? Two experiments used a sentence–picture verification task in which statements about photographs of natural scenes were read in order to make a true/false decision about the validity of the sentence, and in which eye movements were recorded. In Experiment 1 the sentence and the picture were presented concurrently, and objects and words could be inspected in any order. In Experiment 2 the two components were presented one after the other, either picture first or sentence first. Fixation durations on pictures were characteristically longer than those on sentences in both experiments, and fixations on sentences varied according to whether they were being encoded as abstract propositions or as coreferents of objects depicted in a previously inspected picture. The decision time data present a difficulty for existing models of sentence verification tasks, with an inconsistent pattern of differences between true and false trials.

[1]  P. Goolkasian,et al.  Pictures, Words, and Sounds: From Which Format Are We Best Able to Reason? , 2000, The Journal of general psychology.

[2]  P Wenderoth,et al.  Effects of Pattern Orientation and Number of Symmetry Axes on the Detection of Mirror Symmetry in Dot and Solid Patterns , 1998, Perception.

[3]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987 .

[4]  Mary Hegarty,et al.  The Mechanics of Comprehension and Comprehension of Mechanics , 1992 .

[5]  W. Epstein,et al.  Priming Spatial Layout of Scenes , 1997 .

[6]  J. Henderson,et al.  The effects of semantic consistency on eye movements during complex scene viewing , 1999 .

[7]  Herbert H. Clark,et al.  On the process of comparing sentences against pictures , 1972 .

[8]  P. Wason,et al.  NEGATIVES: DENOTATION AND CONNOTATION. , 1963, British journal of psychology.

[9]  P. Carroll,et al.  Visual Analysis of Cartoons: A View from the Far Side , 1992 .

[10]  M. Hegarty.  Mental animation: inferring motion from static displays of mechanical systems. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[11]  G. Underwood.  Eye guidance in reading and scene perception , 1998 .

[12]  K. Rayner.  Eye movements and visual cognition : scene perception and reading , 1992 .

[13]  Marcel Adam Just,et al.  Comprehension of negation with quantification , 1971 .

[14]  K. Rayner,et al.  Contextual effects on word perception and eye movements during reading , 1981 .

[15]  David Crundall,et al.  VISUAL SEARCH WHILE DRIVING: SKILL AND AWARENESS DURING INSPECTION OF THE SCENE , 2002 .

[16]  Andrew J. Stewart,et al.  Integrating text and pictorial information: eye movements when looking at print advertisements. , 2001, Journal of experimental psychology. Applied.

[17]  K. Rayner,et al.  Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity , 1986, Memory & cognition.

[18]  David Crundall,et al.  Effects of experience and processing demands on visual information acquisition in drivers , 1998 .

[19]  P. Goolkasian,et al.  Picture-word differences in a sentence verification task , 1996, Memory & cognition.

[20]  Marcel Adam Just,et al.  Sentence comprehension: A psycholinguistic processing model of verification. , 1975 .

[21]  N. Mackworth,et al.  Cognitive determinants of fixation location during picture viewing. , 1978, Journal of experimental psychology. Human perception and performance.

[22]  Joël Pynte,et al.  Reading as a Perceptual Process , 2000 .

[23]  M. Ross Quillian,et al.  Retrieval time from semantic memory , 1969 .

[24]  Philip B. Gough,et al.  Grammatical transformations and speed of understanding , 1965 .

[25]  P. Chapman,et al.  Visual Search of Driving Situations: Danger and Experience , 1998, Perception.

[26]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[27]  P. C. Wason,et al.  The Processing of Positive and Negative Information , 1959 .

[28]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[29]  John M. Findlay,et al.  How People Extract Information from Graphs: Evidence from a Sentence-Graph Verification Paradigm , 2000, Diagrams.

[30]  R. L. Dominowski.  Anagram solving as a function of letter moves , 1966 .