Modelling Language, Action, and Perception in Type Theory with Records

Formal models of natural language semantics using TTR Type Theory with Records attempt to relate natural language to perception, modelled as classification of objects and events by types which are available as resources to an agent. We argue that this is better suited for representing the meaning of spatial descriptions in the context of agent modelling than traditional formal semantic models which do not relate spatial concepts to perceptual apparatus. Spatial descriptions include perceptual, conceptual and discourse knowledge which we represent all in a single framework. Being a general framework for modelling both linguistic and non-linguistic cognition, TTR is more suitable for the modelling of situated conversational agents in robotics and virtual environments where interoperability between language, action and perception is required. The perceptual systems gain access to abstract conceptual meaning representations of language while the latter can be justified in action and perception.

[1]  G. Miller,et al.  Language and Perception , 1976 .

[2]  Massimo Poesio,et al.  Strudel: A Corpus-Based Semantic Model Based on Properties and Types , 2010, Cogn. Sci..

[3]  Stuart M. Shieber,et al.  An Introduction to Unification-Based Approaches to Grammar , 1986, CSLI Lecture Notes.

[4]  Guido Bugmann,et al.  Training Personal Robots Using Natural Language Instruction , 2001, IEEE Intell. Syst..

[5]  Robin Cooper,et al.  Records and Record Types in Semantic Theory , 2005, J. Log. Comput..

[6]  Nicholas Rescher,et al.  The Logic of Decision and Action , 1967 .

[7]  Ian Witten,et al.  Data Mining , 2000 .

[8]  Kenny R. Coventry,et al.  Spatial Language and Dialogue , 2009, Explorations in language and space.

[9]  Shalom Lappin,et al.  Linguistic Nativism and the Poverty of the Stimulus , 2011 .

[10]  Jeffrey Mark Siskind,et al.  Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic , 1999, J. Artif. Intell. Res..

[11]  Luke S. Zettlemoyer,et al.  Learning to Parse Natural Language Commands to a Robot Control System , 2012, ISER.

[12]  Terry Winograd,et al.  Understanding natural language , 1974 .

[13]  Simon Dobnik.  Teaching mobile robots to use spatial words , 2009 .

[14]  Laura A. Carlson,et al.  Grounding spatial language in perception: an empirical and computational investigation. , 2001, Journal of experimental psychology. General.

[15]  Jonathan Ginzburg,et al.  The interactive stance : meaning for conversation , 2012 .

[16]  Hugh F. Durrant-Whyte,et al.  A solution to the simultaneous localization and map building (SLAM) problem , 2001, IEEE Trans. Robotics Autom..

[17]  S. Levinson,et al.  LANGUAGE AND SPACE , 1996 .

[18]  Staffan Larsson,et al.  Towards a Formal View of Corrective Feedback , 2009 .

[19]  Annette Herskovits,et al.  Language and spatial cognition , 1986 .

[20]  Leonard Talmy,et al.  Toward a cognitive semantics, Vol. 1: Concept structuring systems. , 2000 .

[21]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[22]  Yiannis Aloimonos,et al.  A Language for Human Action , 2007, Computer.

[23]  Annette Herskovits.  Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English , 2009 .

[24]  Klaus-Peter Gapp.  Basic Meanings of Spatial Relations: Computation and Evaluation in 3D Space , 1994, AAAI.

[25]  Tim Fernando,et al.  A Finite-state Approach to Events in Natural Language Semantics , 2004, J. Log. Comput..

[26]  David L. Davidson,et al.  The Logical Form of Action Sentences , 2001 .

[27]  Deb Roy,et al.  Semiotic schemas: A framework for grounding language in action and perception , 2005, Artif. Intell..

[28]  Francis Jeffry Pelletier,et al.  Representation and Inference for Natural Language: A First Course in Computational Semantics , 2005, Computational Linguistics.

[29]  Gordon D. Logan,et al.  A computational analysis of the apprehension of spatial relations , 1996 .

[30]  Charles Kemp,et al.  How to Grow a Mind: Statistics, Structure, and Abstraction , 2011, Science.

[31]  R. Cooper.  Type Theory and Semantics in Flux , 2012 .

[32]  K. Jon Barwise,et al.  The situation in logic , 1989, CSLI lecture notes series.

[33]  Kenny R. Coventry,et al.  The interplay between geometry and function in the comprehension of''over , 2001 .

[34]  S. Levinson.  Space in language and cognition: Explorations in cognitive diversity , 2003 .

[35]  David A. Forsyth,et al.  Describing objects by their attributes , 2009, CVPR.

[36]  Robin Cooper,et al.  Austinian Truth, Attitudes and Type Theory , 2005 .

[37]  Luc Steels,et al.  Perspective alignment in spatial language , 2006, Spatial Language and Dialogue.

[38]  Maria Liakata,et al.  Learning domain theories , 2003, RANLP.