Bayesian validation of grammar productions for the language of thought

Probabilistic proposals of Language of Thoughts (LoTs) can explain learning across different domains as statistical inference over a compositionally structured hypothesis space. While frameworks may differ on how a LoT may be implemented computationally, they all share the property that they are built from a set of atomic symbols and rules by which these symbols can be combined. In this work we propose an extra validation step for the set of atomic productions defined by the experimenter. It starts by expanding the defined LoT grammar for the cognitive domain with a broader set of arbitrary productions and then uses Bayesian inference to prune the productions from the experimental data. The result allows the researcher to validate that the resulting grammar still matches the intuitive grammar chosen for the domain. We then test this method in the language of geometry, a specific LoT model for geometrical sequence learning. Finally, despite the fact of the geometrical LoT not being a universal (i.e. Turing-complete) language, we show an empirical relation between a sequence’s probability and its complexity consistent with the theoretical relationship for universal languages described by Levin’s Coding Theorem.

[1]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[2]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[3]  Thomas L. Griffiths,et al.  A Rational Analysis of Rule-Based Concept Learning , 2008, Cogn. Sci..

[4]  John Goldsmith,et al.  Probabilistic Models of Grammar: Phonology as Information Minimization , 2002 .

[5]  R. Jacobs,et al.  Learning abstract visual concepts via probabilistic program induction in a Language of Thought , 2017, Cognition.

[6]  Thomas L. Griffiths,et al.  Bayesian Inference for PCFGs via Markov Chain Monte Carlo , 2007, NAACL.

[7]  Robert A. Jacobs,et al.  Four Problems Solved by the Probabilistic Language of Thought , 2016 .

[8]  R. Nosofsky.  Attention, similarity, and the identification-categorization relationship. , 1986, Journal of experimental psychology. General.

[9]  Armando Solar-Lezama,et al.  Unsupervised Learning by Program Synthesis , 2015, NIPS.

[10]  S. Dehaene,et al.  Bayesian selection of grammar productions for the language of thought , 2017, bioRxiv.

[11]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Charles Kemp,et al.  How to Grow a Mind: Statistics, Structure, and Abstraction , 2011, Science.

[13]  Noah D. Goodman,et al.  The logical primitives of thought: Empirical foundations for compositional cognitive models. , 2016, Psychological review.

[14]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[15]  Barry Loewer,et al.  Meaning in mind : Fodor and his critics , 1993 .

[16]  G. Boole.  An Investigation of the Laws of Thought: On which are founded the mathematical theories of logic and probabilities , 2007 .

[17]  E. Rosch,et al.  Structural bases of typicality effects. , 1976 .

[18]  Elizabeth S Spelke,et al.  Core foundations of abstract geometry , 2013, Proceedings of the National Academy of Sciences.

[19]  Mariano Sigman,et al.  The language of geometry: Fast comprehension of geometrical primitives and rules in human adults and preschoolers , 2017, PLoS Comput. Biol..

[20]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[21]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[22]  Noah D. Goodman,et al.  Theory learning as stochastic search in the language of thought , 2012 .

[23]  John A. Goldsmith,et al.  Unsupervised Learning of the Morphology of a Natural Language , 2001, CL.

[24]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[25]  Noah D. Goodman,et al.  Bootstrapping in a language of thought: A formal model of numerical concept learning , 2012, Cognition.

[26]  J. Fodor,et al.  The Language of Thought , 1980 .

[27]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[28]  Elizabeth M. Brannon,et al.  Space, Time and Number in the Brain , 2011 .

[29]  Mariano Sigman,et al.  LT^2C^2: A language of thought with Turing-computable Kolmogorov complexity , 2013, ArXiv.

[30]  Eleanor Rosch,et al.  Principles of Categorization , 1978 .

[31]  J. Gomez,et al.  Production and perception rules underlying visual patterns: effects of symmetry and hierarchy , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[32]  Pierre Pica,et al.  Core knowledge of geometry in an Amazonian indigene group. , 2006, Science.

[33]  Murat Aydede,et al.  Language of Thought: The Connectionist Contribution , 1997, Minds and Machines.

[34]  Dedre Gentner,et al.  Structure-Mapping: A Theoretical Framework for Analogy , 1983, Cogn. Sci..

[35]  Pierre Pica,et al.  Geometry as a Universal mental Construction , 2011 .

[36]  Jonathan Knowles.  The language of thought and natural language understanding , 1998 .

[37]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[38]  Sang Ah Lee,et al.  Navigation as a source of geometric knowledge: Young children’s use of length, angle, distance, and direction in a reorientation task , 2012, Cognition.

[39]  Drew H. Abney,et al.  Journal of Experimental Psychology : Human Perception and Performance Influence of Musical Groove on Postural Sway , 2015 .

[40]  Jorge Luis Borges.  Ficciones (1935-1944) , 1944 .

[41]  Cristian S. Calude,et al.  Computing a Glimpse of Randomness , 2002, Exp. Math..

[42]  Robert A Jacobs,et al.  Learning multisensory representations for auditory-visual transfer of sequence category knowledge: a probabilistic language of thought approach , 2014, Psychonomic bulletin & review.

[43]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[44]  H Gleitman,et al.  Spatial knowledge and geometric representation in a child blind from birth. , 1981, Science.

[45]  Johan Wagemans,et al.  The role of vertical mirror symmetry in visual shape detection. , 2009, Journal of vision.