Space and the Vision–Language Interface: A Model-Theoretic Approach

The relation between spatial vision and spatial language has always been a source of controversy. Three problems can be identified as in need of a solution. A first problem pertains to the nature of the minimal information units that make up spatial vision and language. A second problem pertains to the ‘dynamic’ aspects of vision and language, or what visual information to and similar adpositions correspond to. A third problem pertains to how these different types of information are related one another, and what is the status of this ‘interface’, especially within a broader theory of language and cognition. The solution proposed here consists in a formal (model-theoretic) treatment of visual and linguistic information, both static and dynamic, that is couched within (a simplified form of) Discourse Representation Theory. It is shown that this solution is consistent with general theories of cognition and may shed some (novel) light on the nature of the FLN/FLB distinction.

[1]  E. Spelke,et al.  Perceiving and reasoning about objects: Insights from infants , 1993 .

[2]  Donald Davidson,et al.  Thought and Talk , 2001 .

[3]  J. Fodor,et al.  Concepts: Where Cognitive Science Went Wrong , 1998 .

[4]  Robert Stalnaker,et al.  Presuppositions , 1973, J. Philos. Log..

[5]  Max J. Cresswell,et al.  A New Introduction to Modal Logic , 1998 .

[6]  Vivienne Fong,et al.  The order of things : what directional locatives denote , 1998 .

[7]  B. Scholl.  Object Persistence in Philosophy and Psychology , 2007 .

[8]  Rob A. von der Sandt.  Context and presupposition , 1988 .

[9]  E. Maguire,et al.  The Human Hippocampus and Spatial and Episodic Memory , 2002, Neuron.

[10]  E. Spelke,et al.  Perception and understanding of effects of gravity and inertia on object motion , 1999 .

[11]  S. Laurence,et al.  Concepts and Cognitive Science , 1999 .

[12]  J. O’Keefe,et al.  Neuronal computations underlying the firing of place cells and their role in navigation , 1996, Hippocampus.

[13]  Laura A. Carlson,et al.  Attention in Spatial Language: Bridging Geometry and Function , 2005, Functional Features in Language and Space.

[14]  D. Wunderlich.  How do prepositional phrases fit into compositional syntax and semantics? , 1991 .

[15]  E. Gibson.  Gibson, James J. , 2006 .

[16]  Denise Brandão de Oliveira e Britto,et al.  The faculty of language , 2007 .

[17]  Tomaso Poggio,et al.  A Note on Object Class Representation and Categorical Perception , 1999 .

[18]  Laura A. Carlson,et al.  Selecting a reference frame , 1999, Spatial Cogn. Comput..

[19]  J. Fodor.  The Mind Doesn't Work That Way : The Scope and Limits of Computational Psychology , 2000 .

[20]  M. Krifka.  The Origins of Telicity , 1998 .

[21]  Barbara Landau,et al.  Objects and places: Geometric and syntactic representations in early lexical learning , 1990 .

[22]  P. Blitvich.  Levinson, Stephen C. , 2012 .

[23]  Noam Chomsky,et al.  The faculty of language: what is it, who has it, and how did it evolve? , 2002, Science.

[24]  Jeffrey M. Zacks,et al.  Perceiving, remembering, and communicating structure in events. , 2001, Journal of experimental psychology. General.

[25]  Laura A. Carlson,et al.  Defining Spatial Relations: Reconciling Axis and Vector Representations , 2003 .

[26]  Barbara Landau,et al.  Parallels between spatial cognition and spatial language: Evidence from Williams syndrome , 2005 .

[27]  Rob A. van der Sandt,et al.  Presupposition Projection as Anaphora Resolution , 1992, J. Semant..

[28]  Simon J. Greenhill,et al.  Evolved structure of language shows lineage-specific trends in word-order universals , 2011, Nature.

[29]  E. Rosch,et al.  The Embodied Mind: Cognitive Science and Human Experience , 1993 .

[30]  Lynn Nadel,et al.  Encyclopedia of Cognitive Science , 2003 .

[31]  Emar Maier,et al.  Belief in Context: Towards a unified Semantis of de re and de se attitude reports , 2006 .

[32]  Fred Landman,et al.  Events and Plurality: The Jerusalem Lectures , 2001 .

[33]  P. Blackburn,et al.  Book Reviews: The Proper Treatment of Events, by Michiel van Lambalgen and Fritz Hamm , 2005, CL.

[34]  Angelika Kratzer,et al.  The Event Argument and the Semantics of Verbs , 2002 .

[35]  A. Kratzer.  An investigation of the lumps of thought , 1989 .

[36]  E. Spelke,et al.  Spatiotemporal integration and object perception in infancy , 1996 .

[37]  Kenny R. Coventry,et al.  Seeing, saying and acting: The psychological semantics of spatial prepositions , 2004 .

[38]  Jon Barwise,et al.  Information Flow: The Logic of Distributed Systems , 1997 .

[39]  John E. Hummel,et al.  Two Roles for Attention in Shape Perception: A Structural Description Model of Visual Scrutiny , 1998 .

[40]  E. Spelke,et al.  Object representation and predictive action in infancy , 2000 .

[41]  S. Crain,et al.  The Logic Instinct , 2010 .

[42]  R. Jackendoff.  Parts and boundaries , 1991, Cognition.

[43]  Barbara Tversky,et al.  The structure of experience. , 2008 .

[44]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[45]  Godehard Link.  The Logical Analysis of Plurals and Mass Terms: A Lattice‐theoretical Approach , 2008 .

[46]  D. McDermott.  LANGUAGE OF THOUGHT , 2012 .

[47]  G. Graffi.  200 Years of Syntax: A critical survey , 2001 .

[48]  H. Kamp.  Events, Instants and Temporal Reference , 1979 .

[49]  Joost Zwarts,et al.  A hierarchy of locations: evidence from the encoding of direction in adpositions and cases , 2010 .

[50]  Mark Steedman,et al.  The use of context by the psychological parser , 1981 .

[51]  F. Keil.  Concepts, Kinds, and Cognitive Development , 1989 .

[52]  A. Piazza.  History and Geography of Human Genes , 1994 .

[53]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[54]  J. Eijck.  Formal Concept Analysis and Prototypes , 2004 .

[55]  O. Van.  Word and Object , 1960 .

[56]  James E. Tomberlin,et al.  On the Plurality of Worlds. , 1989 .

[57]  Neil Burgess,et al.  Theta activity, virtual navigation and the human hippocampus , 1999, Trends in Cognitive Sciences.

[58]  B. Duchaine,et al.  Visual Agnosia, 2nd Ed. , 2005 .

[59]  N. Burgess.  Spatial Cognition and the Brain , 2008, Annals of the New York Academy of Sciences.

[60]  Annette Herskovits.  Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English , 2009 .

[61]  Richard Montague,et al.  The Proper Treatment of Quantification in Ordinary English , 1973 .

[62]  Susan Carey,et al.  Infants' knowledge of objects: beyond object files and object tracking , 2001, Cognition.

[63]  E. Tulving.  Elements of episodic memory , 1983 .

[64]  U. Bronfenbrenner.  Ecological systems theory. , 1992 .

[65]  Halldor Armann Sigurðsson,et al.  The syntax of Person, Tense, and speech features , 2004 .

[66]  Peter Svenonius,et al.  The Emergence of Axial Parts , 2006 .

[67]  Seungho Nam,et al.  The semantics of locative prepositional phrases in English , 1995 .

[68]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[69]  Terry Regier,et al.  An Attentional Constraint on Spatial Meaning , 2003 .

[70]  E. Rosch.  Cognitive Representations of Semantic Categories. , 1975 .

[71]  Z. Pylyshyn.  Some primitive mechanisms of spatial attention , 1994, Cognition.

[72]  G. Buzsáki.  Rhythms of the brain , 2006 .

[73]  Uwe Reyle,et al.  37. Discourse Representation Theory , 2011 .

[74]  Michiel van Lambalgen,et al.  A Logic of Vision , 2000 .

[75]  Tomaso Poggio,et al.  Models of object recognition , 2000, Nature Neuroscience.

[76]  Z. Pylyshyn.  Visual indexes, preconceptual objects, and situated vision , 2001, Cognition.

[77]  Peter M. Vishton,et al.  Predictive action in infancy: tracking and reaching for moving objects , 1998, Cognition.

[78]  S. Crain,et al.  Is Logic Innate? , 2007, Biolinguistics.

[79]  Alona Soschen,et al.  On the Nature of Syntax , 2008, Biolinguistics.

[80]  Catharine Abell,et al.  Seeing and Visualizing: It's Not What You Think , 2005 .

[81]  Frank Wolter,et al.  Handbook of Modal Logic , 2007, Studies in logic and practical reasoning.

[82]  E. Spelke,et al.  Early knowledge of object motion: continuity and inertia , 1994, Cognition.

[83]  Joost Zwarts,et al.  Prepositional Aspect and the Algebra of Paths , 2005 .

[84]  Bernhard Ganter,et al.  Formal Concept Analysis: Mathematical Foundations , 1998 .

[85]  Noam Chomsky,et al.  The evolution of the language faculty: Clarifications and implications , 2005, Cognition.

[86]  Colin Phillips,et al.  Order and structure , 1996 .

[87]  J O'Keefe,et al.  Vector grammar, places, and the functional role of the spatial prepositions in English , 2001 .

[88]  Neil Burgess,et al.  Neural representations in human spatial memory , 2003, Trends in Cognitive Sciences.

[89]  Thomas Serre,et al.  Object recognition with features inspired by visual cortex , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[90]  Jeffery. M. Zacks,et al.  Activation of human motion processing areas during event perception , 2003, Cognitive, affective & behavioral neuroscience.

[91]  D. Kahneman,et al.  The reviewing of object files: Object-specific integration of information , 1992, Cognitive Psychology.

[92]  H. Kamp.  A Theory of Truth and Semantic Representation , 2008 .

[93]  A. Beaton,et al.  Dyslexia, Reading and the Brain: A Sourcebook of Psychological and Biological Research , 2004 .

[94]  I. Biederman,et al.  Dynamic binding in a neural network for shape recognition. , 1992, Psychological review.

[95]  William Lopez,et al.  Attention Unites Form and Function in Spatial Language , 2006, Spatial Cogn. Comput..

[96]  Linda B. Smith,et al.  Object name Learning Provides On-the-Job Training for Attention , 2002, Psychological science.

[97]  Irene Heim,et al.  The semantics of definite and indefinite noun phrases : a dissertation , 1982 .

[98]  N. Cowan.  Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system. , 1988, Psychological bulletin.

[99]  Zenon W. Pylyshyn,et al.  Computation and Cognition: Toward a Foundation for Cognitive Science , 1984 .

[100]  S. Ullman.  The Interpretation of Visual Motion , 1979 .

[101]  James Pustejovsky,et al.  Word Meaning and Commonsense Metaphysics , 2005 .

[102]  E. Tulving,et al.  Episodic and semantic memory , 1972 .

[103]  Bridget Samuels,et al.  The Structure of Phonological Theory , 2009 .

[104]  D. Bickerton.  Adam's Tongue: How Humans Made Language, How Language Made Humans , 2009 .

[105]  R. Jackendoff,et al.  The capacity for music: What is it, and what’s special about it? , 2006, Cognition.

[106]  Linda B. Smith,et al.  Naming in young children: a dumb attentional mechanism? , 1996, Cognition.

[107]  Barbara Landau,et al.  Object Shape, Object Name, and Object Kind: Representation and Development , 1994 .

[108]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[109]  F. Landman.  Indefinites and the type of sets , 2004 .

[110]  E S Spelke,et al.  Infants' sensitivity to effects of gravity on visible object motion. , 1992, Journal of experimental psychology. Human perception and performance.

[111]  Charles Reiss,et al.  Modularity in the “Sound” Domain: Implications for the Purview of Universal Grammar , 2007 .

[112]  Robert Stalnaker.  Context and content : essays on intentionality in speech and thought , 1999 .

[113]  Charles Reiss,et al.  The Phonological Enterprise , 2008 .

[114]  J. Hummel,et al.  Categorical relations in shape perception. , 1996, Spatial vision.

[115]  Boban Arsenijević,et al.  From Spatial Cognition to Language , 2008, Biolinguistics.

[116]  E. Zee,et al.  Representing Direction in Language and Space , 2002 .

[117]  S. Levinson.  Space in language and cognition: Explorations in cognitive diversity , 2003 .

[118]  Z. Pylyshyn,et al.  Dynamics of target selection in multiple object tracking (MOT). , 2006, Spatial vision.

[119]  E. Spelke,et al.  Perception, ontology, and word meaning , 1992, Cognition.

[120]  Susan Carey,et al.  Language acquisition and conceptual development: Whorf versus continuity theorists: bringing data to bear on the debate , 2001 .

[121]  Koji Fujita,et al.  A Prospect for Evolutionary Adequacy: Merge and the Evolution and Development of Human Language , 2009, Biolinguistics.

[122]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[123]  G. Scott McGlashan,et al.  Towards a cognitive linguistic approach to language comprehension , 1992 .

[124]  T. Crow.  The nuclear symptoms of schizophrenia reveal the four quadrant structure of language and its deictic frame , 2010, Journal of Neurolinguistics.

[125]  Nelson Cowan,et al.  Working Memory Capacity , 2005 .

[126]  T. Poggio,et al.  A network that learns to recognize three-dimensional objects , 1990, Nature.

[127]  Jan van Eijck,et al.  Representing Discourse in Context , 1997, Handbook of Logic and Language.

[128]  Leonard Talmy,et al.  Figure and Ground in Complex Sentences , 1975 .

[129]  Nirmalangshu Mukherji,et al.  The Primacy of Grammar , 2010 .

[130]  John O'Keefe,et al.  The hippocampal cognitive map and navigational strategies. , 1991 .

[131]  M Coltheart,et al.  DRC: a dual route cascaded model of visual word recognition and reading aloud. , 2001, Psychological review.

[132]  E. Tulving.  [Episodic memory: from mind to brain]. , 2004, Revue neurologique.

[133]  Cristina Guardiano,et al.  The Biolinguistic Program and Historical Reconstruction , 2011 .

[134]  Terence Parsons,et al.  Events in the Semantics of English: A Study in Subatomic Semantics , 1990 .

[135]  Linda B. Smith,et al.  Object perception and object naming in early development , 1998, Trends in Cognitive Sciences.

[136]  Z. Harris,et al.  Foundations of language , 1941 .

[137]  Jeffrey M. Zacks,et al.  Using movement and intentions to understand simple events , 2004, Cogn. Sci..

[138]  J. O’Keefe,et al.  A computational theory of the hippocampal cognitive map. , 1990, Progress in brain research.

[139]  Hans Kamp,et al.  Some remarks on the logic of change, Part I , 1980 .

[140]  Ray Jackendoff,et al.  The Architecture of the Language Faculty , 1996 .

[141]  N. Burgess,et al.  Spatial memory: how egocentric and allocentric combine , 2006, Trends in Cognitive Sciences.

[142]  S. Carey.  Becoming a face expert. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[143]  Uwe Reyle,et al.  From discourse to logic , 1993 .

[144]  Angelika Kratzer,et al.  Situations in Natural Language Semantics , 2007 .

[145]  J. Gibson.  The Senses Considered As Perceptual Systems , 1967 .

[146]  Christoph Schwarze,et al.  Meaning, Use, and Interpretation of Language , 1983 .

[147]  Z. Pylyshyn.  Some puzzling findings in multiple object tracking: I. Tracking without keeping track of object identities , 2004 .

[148]  Barbara Landau,et al.  The effects of spatial language on spatial representation: Setting some boundaries , 2003 .

[149]  Francesco Orilia,et al.  Semantics and Cognition , 1991 .

[150]  P. Bloom.  How children learn the meanings of words , 2000 .

[151]  S. Levinson,et al.  'Natural Concepts' in the Spatial Topologial Domain--Adpositional Meanings in Crosslinguistic Perspective: An Exercise in Semantic Typology , 2003 .

[152]  David L. Davidson,et al.  The Logical Form of Action Sentences , 2001 .

[153]  M. Gazzaniga.  The new cognitive neurosciences, 2nd ed. , 2000 .

[154]  Z. Pylyshyn.  Some puzzling findings in multiple object tracking (MOT): II. Inhibition of moving nontargets , 2006 .

[155]  Z. Pylyshyn.  The role of location indexes in spatial perception: A sketch of the FINST spatial-index model , 1989, Cognition.

[156]  Linda B. Smith,et al.  Syntactic context and the shape bias in children's and adults' lexical learning , 1992 .

[157]  Maria Trofimova,et al.  Case assignment by prepositions in Russian aphasia , 2009 .

[158]  B. Landau,et al.  “What” and “where” in spatial language and spatial cognition , 1993 .

[159]  J. Fodor.  The Modularity of mind. An essay on faculty psychology , 1986 .

[160]  D. E. Irwin,et al.  Reference Frame Activation during Spatial Term Assignment , 1994 .

[161]  W. H. F. Barnes.  The Nature of Explanation , 1944, Nature.

[162]  N. Cowan.  Attention and Memory: An Integrated Framework , 1995 .

[163]  D. Marr,et al.  Representation and recognition of the spatial organization of three-dimensional shapes , 1978, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[164]  Peter Gärdenfors,et al.  Conceptual spaces - the geometry of thought , 2000 .

[165]  Tomaso Poggio,et al.  From the Retina to the Neocortex: Selected Papers of David Marr , 2012 .

[166]  Godehard Link.  Algebraic semantics in language and philosophy , 1997 .

[167]  Laura A. Carlson,et al.  Grounding spatial language in perception: an empirical and computational investigation. , 2001, Journal of experimental psychology. General.

[168]  Gillian Ramchand,et al.  Verb Meaning and the Lexicon: A First Phase Syntax , 2008 .

[169]  S. Pinker,et al.  The nature of the language faculty and its implications for evolution of language (Reply to Fitch, Hauser, and Chomsky) , 2005, Cognition.

[170]  L. M. Faltz,et al.  Boolean semantics for natural language , 1984 .

[171]  Jeffrey M. Zacks,et al.  A Computational Model of Event Segmentation From Perceptual Prediction , 2007, Cogn. Sci..

[172]  F. Landman.  Structures for semantics , 1991 .

[173]  P. Bloom,et al.  Language and space , 1998 .

[174]  Linda B. Smith,et al.  The Spatial Foundations of Language and Cognition , 2009, Explorations in language and space.

[175]  R. Weale.  Vision. A Computational Investigation Into the Human Representation and Processing of Visual Information. David Marr , 1983 .

[176]  Emmanuel Dupoux,et al.  Continuity, competence, and the object concept , 2001 .

[177]  S. Pinker,et al.  The faculty of language: what's special about it? , 2005, Cognition.

[178]  I. I. N. Kamp.  Combining Montague Semantics and Discourse Representation , 1996 .

[179]  Raymond Turner,et al.  Semantics and property theory , 1988 .

[180]  Yoad Winter,et al.  Vector Space Semantics: A Model-Theoretic Analysis of Locative Prepositions , 2000, J. Log. Lang. Inf..

[181]  E. Spelke,et al.  Straddling the perception–conception boundary , 2004 .

[182]  Kenny R. Coventry,et al.  Towards a Classification of Extra-geometric Influences on the Comprehension of Spatial Prepositions , 2005, Functional Features in Language and Space.

[183]  R. Jackendoff.  Foundations of Language: Brain, Meaning, Grammar, Evolution , 2002 .

[184]  Nicholas Asher,et al.  Lexical Meaning in Context - A Web of Words , 2011 .

[185]  Marcus Kracht,et al.  On the Semantics of Locatives , 2002 .

[186]  Kenny R. Coventry,et al.  Spatial Language and Dialogue , 2009, Explorations in language and space.

[187]  T. Poggio,et al.  Neural mechanisms of object recognition , 2002, Current Opinion in Neurobiology.

[188]  Jeffrey M. Zacks,et al.  Event perception: a mind-brain perspective. , 2007, Psychological bulletin.

[189]  Alex M. Andrew,et al.  Computation and Cognition: Towards A Foundation for Cognitive Science, by Zenon W. Pylyshyn, MIT Press, Cambridge, Mass., xxiii + 292 pp., £26.15 , 1985, Robotica.

[190]  B. Geurts.  Presuppositions and Pronouns , 1999 .

[191]  Y. Grodzinsky.  Theoretical perspectives on language deficits , 1990 .

[192]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[193]  P. Johnson-Laird.  Human and Machine Thinking , 1992 .

[194]  B. Scholl.  Objects and attention: the state of the art , 2001, Cognition.

[195]  Rocky Ross,et al.  Mental models , 2004, SIGA.