An Eye-Tracking Study of Exploitations of Spatial Constraints in Diagrammatic Reasoning

Semantic studies on diagrammatic notations (Barwise & Etchemendy, 1990; Shimojima, 1995; Stenning & Lemon, 2001) have revealed that the "non-deductive," "emergent," or "perceptual" effects of diagrams (Chandrasekaran, Kurup, Banerjee, Josephson, & Winkler, 2004; Kulpa, 2003; Larkin & Simon, 1987; Lindsay, 1988) are all rooted in the exploitation of spatial constraints on graphical structures. Thus, theoretically, this process is a key factor in inference with diagrams, explaining the frequently observed reduction of inferential load. The purpose of this study was to examine the empirical basis for this theoretical suggestion, focusing on the reality of the constraint-exploitation strategy in actual practices of diagrammatic reasoning. Eye movements were recorded while participants used simple position diagrams to solve three- or four-term transitive inference problems. Our experiments revealed that the participants could exploit spatial constraints on graphical structures even when (a) they were not in the position of actually manipulating diagrams, (b) the semantic rule for the provided diagrams did not match their preferences, and (c) the constraint-exploitation strategy invited a partly adverse effect. These findings indicate that the hypothesized process is in fact robust, with the potential to broadly account for the inferential advantage of diagrams.

[1]  Hilda M. Fehd,et al.  Eye movements during multiple object tracking: Where do participants look? , 2008, Cognition.

[2]  D. Kahneman,et al.  The reviewing of object files: Object-specific integration of information , 1992, Cognitive Psychology.

[3]  A. Clark,et al.  The Extended Mind , 1998, Analysis.

[4]  Daniel C. Richardson,et al.  Representation, space and Hollywood Squares: looking at things that aren't there anymore , 2000, Cognition.

[5]  Philip N. Johnson-Laird,et al.  Models and heterogeneous reasoning , 2006, J. Exp. Theor. Artif. Intell..

[6]  B. Chandrasekaran,et al.  A Constraint Satisfaction Framework for Executing Perceptions and Actions in Diagrammatic Reasoning , 2010, J. Artif. Intell. Res..

[7]  M J Farah,et al.  Mechanisms of imagery-perception interaction. , 1989, Journal of experimental psychology. Human perception and performance.

[8]  D. E. Irwin,et al.  Eye movements and the integration of visual memory and visual perception , 2005, Perception & psychophysics.

[9]  Susan Bell Trickett,et al.  Extracting Explicit and Implict Information from Complex Visualizations , 2002, Diagrams.

[10]  Z. Pylyshyn The imagery debate: Analogue media versus tacit knowledge. , 1981 .

[11]  M. J. Intons-Peterson,et al.  Imagery paradigms: how vulnerable are they to experimenters' expectations? , 1983, Journal of experimental psychology. Human perception and performance.

[12]  Daniel C. Richardson,et al.  When facts go down the rabbit hole: Contrasting features and objecthood as indexes to memory , 2008, Cognition.

[13]  Jon Oberlander,et al.  A Cognitive Theory of Graphical and Linguistic Reasoning: Logic and Implementation , 1995, Cogn. Sci..

[14]  Roger Johansson,et al.  Pictures and Spoken Descriptions Elicit Similar Eye Movements During Mental Imagery, Both in Light and in Complete Darkness , 2006, Cogn. Sci..

[15]  S. Kosslyn,et al.  The case for mental imagery , 2006 .

[16]  Daniel A. Gajewski,et al.  Minimal use of working memory in a scene comparison task , 2005 .

[17]  Daniel L. Schwartz,et al.  Reasoning about the referent of a picture versus reasoning about the picture as the referent: An effect of visual realism , 1995, Memory & cognition.

[18]  N. Hari Narayanan,et al.  Diagrammatic Reasoning: Cognitive and Computational Perspectives , 1995 .

[19]  E. Gordon,et al.  Defining the temporal threshold for ocular fixation in free-viewing visuocognitive tasks , 2003, Journal of Neuroscience Methods.

[20]  G. Zelinsky,et al.  An eye movement analysis of multiple object tracking in a realistic environment , 2008 .

[21]  L. Stark,et al.  Spontaneous Eye Movements During Visual Imagery Reflect the Content of the Visual Scene , 1997, Journal of Cognitive Neuroscience.

[22]  Bruno Laeng,et al.  Eye scanpaths during visual imagery reenact those of perception of the same visual scene , 2002, Cogn. Sci..

[23]  Atsushi Shimojima,et al.  Do We Really Reason about a Picture as the Referent , 2003 .

[24]  Barry H. Cohen,et al.  Evidence for image-scanning eye movements during transitive inference , 1998, Biological Psychology.

[25]  S M Kosslyn,et al.  Visual images preserve metric spatial information: evidence from studies of image scanning. , 1978, Journal of experimental psychology. Human perception and performance.

[26]  Oliver Lemon,et al.  Aligning Logical and Psychological Perspectives on Diagrammatic Reasoning , 2001, Artificial Intelligence Review.

[27]  P N Johnson-Laird,et al.  Reasoning about relations. , 2005, Psychological review.

[28]  J. Barwise,et al.  Visual information and valid reasoning , 1991 .

[29]  Z. Pylyshyn The role of location indexes in spatial perception: A sketch of the FINST spatial-index model , 1989, Cognition.

[30]  Vidal Annan,et al.  Can indexes be voluntarily assigned in multiple object tracking , 2010 .

[31]  R. Shepard,et al.  Functional representations common to visual perception and imagination. , 1978, Journal of experimental psychology. Human perception and performance.

[32]  Rajesh P. N. Rao,et al.  Embodiment is the foundation, not a level , 1996, Behavioral and Brain Sciences.

[33]  Susan Bell Trickett,et al.  A New Model of Graph and Visualization Usage , 2001 .

[34]  M. Knauff Visual and Spatial Representations in Relational Reasoning , 2000 .

[35]  P. Johnson-Laird,et al.  Visual imagery can impede reasoning , 2002, Memory & cognition.

[36]  Mary Hegarty,et al.  What Diagrams Reveal about Representations in Linear Reasoning, and How They Help , 2008, Diagrams.

[37]  Daniel C. Richardson,et al.  Thinking outside the brain: Spatial indices to visual and linguistic Information. , 2004 .

[38]  Michael J. Spivey,et al.  Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects , 2001, Psychological research.

[39]  Susan Bell Trickett,et al.  "What if...": The Use of Conceptual Simulations in Scientific Reasoning , 2007, Cogn. Sci..

[40]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[41]  Z W Pylyshyn,et al.  Tracking multiple independent targets: evidence for a parallel tracking mechanism. , 1988, Spatial vision.

[42]  J. O'Regan,et al.  Solving the "real" mysteries of visual perception: the world as an outside memory. , 1992, Canadian journal of psychology.

[43]  S. Kosslyn Scanning visual images:Some structural implications , 1973 .

[44]  B. Chandrasekaran,et al.  Augmenting Cognitive Architectures to Support Diagrammatic Imagination , 2011, Top. Cogn. Sci..

[45]  M. Knauff A Neuro-Cognitive Theory of Relational Reasoning with Mental Models and Visual Images , 2006 .

[46]  R. K. Lindsay Images and inference , 1988, Cognition.

[47]  James E. Hoffman,et al.  The role of eye fixations in concentration and amplification effects during multiple object tracking , 2009 .

[48]  S. Kosslyn Image and Brain: The Resolution of the Imagery Debate , 1994, Journal of Cognitive Neuroscience.

[49]  Mary Hegarty,et al.  Spatial Visualization in Physics Problem Solving , 2007, Cogn. Sci..

[50]  B. Chandrasekaran,et al.  A Spatial Search Framework for Executing Perceptions and Actions in Diagrammatic Reasoning , 2010, Diagrams.

[51]  Atsushi Shimojima,et al.  Operational constraints in diagrammatic reasoning , 1996 .

[52]  Aaron Sloman Interactions between philosophy and artificial intelligence: the role of intuition and non-logical reasoning in intelligence , 1971, IJCAI 1971.

[53]  P. Johnson-Laird,et al.  PSYCHOLOGICAL SCIENCE Research Article HOW DIAGRAMS CAN IMPROVE REASONING , 2022 .

[54]  B. Chandrasekaran,et al.  An Architecture for Problem Solving with Diagrams , 2004, Diagrams.

[55]  Susan Bell Trickett,et al.  Connecting Internal and External Representations: Spatial Transformations of Scientific Visualizations , 2005 .

[56]  Patrick Olivier,et al.  Diagrammatic Reasoning: An Artificial Intelligence Perspective , 2001, Artificial Intelligence Review.