A Generative Symbolic Model for More General Natural Language Understanding and Reasoning

We present a new fully-symbolic Bayesian model of semantic parsing and reasoning which we hope to be the first step in a research program toward more domainand task-general NLU and AI. Humans create internal mental models of their observations which greatly aid in their ability to understand and reason about a large variety of problems. We aim to capture this in our model, which is fully interpretable and Bayesian, designed specifically with generality in mind, and therefore provides a clearer path for future research to expand its capabilities. We derive and implement an inference algorithm, and evaluate it on an out-of-domain PROOFWRITER questionanswering/reasoning task, achieving zeroshot accuracies of 100% and 93.43%, depending on the experimental setting, thereby demonstrating its value as a proof-ofconcept.

[1]  Ailsa H. Land,et al.  An Automatic Method of Solving Discrete Programming Problems , 1960 .

[2]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[3]  Chuang Gan,et al.  CLEVRER: CoLlision Events for Video REpresentation and Reasoning , 2020, ICLR.

[4]  Ryan P. Adams,et al.  SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models , 2020, ICLR.

[5]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[6]  Pedro M. Domingos,et al.  Learning and Inference in Tractable Probabilistic Knowledge Bases , 2015, UAI.

[7]  Tom Mitchell,et al.  A Probabilistic Generative Grammar for Semantic Parsing , 2016, CoNLL.

[8]  D. Aldous.  Exchangeability and related topics , 1985 .

[9]  William W. Cohen,et al.  Faithful Embeddings for Knowledge Base Queries , 2020, NeurIPS.

[10]  Stuart J. Russell,et al.  Artificial Intelligence - A Modern Approach, Third International Edition , 2010 .

[11]  G. Gentzen.  Untersuchungen über das logische Schließen. I , 1935 .

[12]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[13]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[14]  Howard Gregory.  Language and Logics: An Introduction to the Logical Foundations of Language , 2015 .

[15]  Allen Newell,et al.  SOAR: An Architecture for General Intelligence , 1987, Artif. Intell..

[16]  Tim Rocktäschel,et al.  End-to-end Differentiable Proving , 2017, NIPS.

[17]  Dan Klein,et al.  Type-Based MCMC , 2010, HLT-NAACL.

[18]  Steffen Staab,et al.  Knowledge graphs , 2021, Commun. ACM.

[19]  Pasquale Minervini,et al.  Complex Query Answering with Neural Link Predictors , 2021, ICLR.

[20]  P. Tighe,et al.  Bias and ethical considerations in machine learning and the automation of perioperative risk assessment , 2020, British Journal of Anaesthesia.

[21]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[22]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[23]  Vítor Santos Costa,et al.  Inductive Logic Programming , 2013, Lecture Notes in Computer Science.

[24]  Peter Clark,et al.  Transformers as Soft Reasoners over Language , 2020, ArXiv.

[25]  David R. Dowty,et al.  Introduction to Montague semantics , 1980 .

[26]  Xinlei Chen,et al.  Never-Ending Learning , 2012, ECAI.

[27]  W. V. Quine,et al.  Natural deduction , 2021, An Introduction to Proof Theory.

[28]  Jure Leskovec,et al.  Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings , 2020, ICLR.

[29]  Robert P. Goldman,et al.  A Bayesian Model of Plan Recognition , 1993, Artif. Intell..

[30]  Jennifer Chu-Carroll,et al.  To Test Machine Comprehension, Start by Defining Comprehension , 2020, ACL.

[31]  Peter Clark,et al.  ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language , 2020, FINDINGS.

[32]  Leon Henkin,et al.  Completeness in the theory of types , 1950, Journal of Symbolic Logic.

[33]  James Cussens,et al.  Parameter Estimation in Stochastic Logic Programs , 2001, Machine Learning.

[34]  Terence Parsons,et al.  Events in the Semantics of English: A Study in Subatomic Semantics , 1990 .

[35]  S. Muggleton.  Stochastic Logic Programs , 1996 .

[36]  Christian P. Robert,et al.  Monte Carlo Statistical Methods (Springer Texts in Statistics) , 2005 .

[37]  Ulrich Furbach,et al.  Tackling Benchmark Problems of Commonsense Reasoning , 2015, Bridging@CADE.

[38]  Fabrizio Riguzzi,et al.  Structure learning of probabilistic logic programs by searching the clause space , 2013, Theory and Practice of Logic Programming.

[39]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[40]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[41]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[42]  Luc De Raedt,et al.  Scalable Rule Learning in Probabilistic Knowledge Bases , 2019, AKBC.

[43]  Neng-Fa Zhou,et al.  Generative Modeling with Failure in PRISM , 2005, IJCAI.

[44]  Mohit Bansal,et al.  PRover: Proof Generation for Interpretable Reasoning over Rules , 2020, EMNLP.

[45]  Miriam R. L. Petruck,et al.  Language (Re)modelling: Towards Embodied Language Understanding , 2020, ACL.

[46]  Heiner Stuckenschmidt,et al.  Towards Distributed MCMC Inference in Probabilistic Knowledge Bases , 2012, AKBC-WEKEX@NAACL-HLT.

[47]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[48]  Emily M. Bender,et al.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data , 2020, ACL.

[49]  Thomas L. Griffiths,et al.  Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models , 2006, NIPS.

[50]  Staffan Larsson,et al.  Probabilistic Type Theory and Natural Language Semantics , 2015, LILT.

[51]  Jonathan Berant,et al.  On Making Reading Comprehension More Comprehensive , 2019, EMNLP.

[52]  M. E. Szabo,et al.  The collected papers of Gerhard Gentzen , 1969 .

[53]  John K. Tsotsos,et al.  40 years of cognitive architectures: core cognitive abilities and practical applications , 2018, Artificial Intelligence Review.

[54]  Tal Linzen,et al.  How Can We Accelerate Progress Towards Human-like Linguistic Generalization? , 2020, ACL.

[55]  Gerhard Gentzen,et al.  Investigations into Logical Deduction , 1970 .

[56]  Alonzo Church,et al.  A formulation of the simple theory of types , 1940, Journal of Symbolic Logic.

[57]  Doug Downey,et al.  Abductive Commonsense Reasoning , 2019, ICLR.