On the Capabilities and Limitations of Reasoning for Natural Language Understanding

Recent systems for natural language understanding are strong at overcoming linguistic variability for lookup style reasoning. Yet, their accuracy drops dramatically as the number of reasoning steps increases. We present the first formal framework to study such empirical observations, addressing the ambiguity, redundancy, incompleteness, and inaccuracy that the use of language introduces when representing a hidden conceptual space. Our formal model uses two interrelated spaces: a conceptual meaning space that is unambiguous and complete but hidden, and a linguistic symbol space that captures a noisy grounding of the meaning space in the symbols or words of a language. We apply this framework to study the connectivity problem in undirected graphs---a core reasoning problem that forms the basis for more complex multi-hop reasoning. We show that it is indeed possible to construct a high-quality algorithm for detecting connectivity in the (latent) meaning graph, based on an observed noisy symbol graph, as long as the noise is below our quantified noise level and only a few hops are needed. On the other hand, we also prove an impossibility result: if a query requires a large number (specifically, logarithmic in the size of the meaning graph) of hops, no reasoning system operating over the symbol graph is likely to recover any useful property of the meaning graph. This highlights a fundamental barrier for a class of reasoning problems and systems, and suggests the need to limit the distance between the two spaces, rather than investing in multi-hop reasoning with "many" hops.

[1]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[2]  Peter Jansen,et al.  What’s in an Explanation? Characterizing Knowledge and Inference Requirements for Elementary Science Exams , 2016, COLING.

[3]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[4]  B. Bollobás.  The evolution of random graphs , 1984 .

[5]  H. Chertkow,et al.  Semantic memory , 2002, Current neurology and neuroscience reports.

[6]  Danqi Chen,et al.  Observed versus latent features for knowledge base and text inference , 2015, CVSC.

[7]  John McCarthy,et al.  Programs with common sense , 1960 .

[8]  Dan Roth,et al.  A Linear Programming Formulation for Global Inference in Natural Language Tasks , 2004, CoNLL.

[9]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[10]  Peter Jansen,et al.  Framing QA as Building and Ranking Intersentence Answer Justifications , 2017, CL.

[11]  Philipp Koehn,et al.  Abstract Meaning Representation for Sembanking , 2013, LAW@ACL.

[12]  M. Baltin,et al.  The Mental representation of grammatical relations , 1985 .

[13]  Mark Steedman,et al.  Combinatory Categorial Grammar , 2011 .

[14]  Philip N. Johnson-Laird,et al.  Mental Models in Cognitive Science , 1980, Cogn. Sci..

[15]  Richard Socher,et al.  Multi-Hop Knowledge Graph Reasoning with Reward Shaping , 2018, EMNLP.

[16]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[17]  Christopher D. Manning,et al.  NaturalLI: Natural Logic Inference for Common Sense Reasoning , 2014, EMNLP.

[18]  Alan M. Frieze,et al.  Random graphs , 2006, SODA '06.

[19]  Charles S. Peirce,et al.  A theory of probable inference. , 1883 .

[20]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[21]  Peter Clark,et al.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , 2018, EMNLP.

[22]  Mariarosaria Taddeo,et al.  Solving the symbol grounding problem: a critical review of fifteen years of research , 2005, J. Exp. Theor. Artif. Intell..

[23]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[24]  Dan Roth,et al.  Question Answering as Global Reasoning Over Semantic Abstractions , 2018, AAAI.

[25]  Veronique Kiermer,et al.  Six degrees of separation , 2006, Nature Methods.

[26]  Charles J. Fillmore,et al.  SCENES- AND- FRAMES SEMANTICS. , 1977 .

[27]  Praveen Paritosh,et al.  Freebase: a collaboratively created graph database for structuring human knowledge , 2008, SIGMOD Conference.

[28]  Lynette Hirschman,et al.  Deep Read: A Reading Comprehension System , 1999, ACL.

[29]  Danqi Chen,et al.  Reasoning With Neural Tensor Networks for Knowledge Base Completion , 2013, NIPS.

[30]  Rada Mihalcea,et al.  Wikify!: linking documents to encyclopedic knowledge , 2007, CIKM '07.

[31]  Rajarshi Das,et al.  Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks , 2016, EACL.

[32]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[33]  Oren Etzioni,et al.  Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions , 2016, AAAI.

[34]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[35]  Oren Etzioni,et al.  Question Answering via Integer Programming over Semi-Structured Knowledge , 2016, IJCAI.

[36]  Dan Roth,et al.  Mapping Dependencies Trees: An Application to Question Answering , 2003 .

[37]  Clayton T. Morrison,et al.  WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference , 2018, LREC.

[38]  G. Kirk,et al.  The Presocratic Philosophers a Critical History with a Selection of Texts /by G.S. Kirk, J.E. Raven, M. Schofield. --. -- , 1957 .

[39]  P. Erdos,et al.  On the evolution of random graphs , 1984 .

[40]  Ronald M. Kaplan,et al.  Lexical Functional Grammar A Formal System for Grammatical Representation , 2004 .

[41]  Herbert A. Davidson,et al.  Alfarabi, Avicenna, and Averroes, on Intellect: Their Cosmologies, Theories of the Active Intellect, and Theories of Human Intellect , 1992 .

[42]  Peter A. Jansen.  A Study of Automatically Acquiring Explanatory Inference Patterns from Corpora of Explanations: Lessons from Elementary Science Exams , 2017, AKBC@NIPS.

[43]  Peter Clark,et al.  Answering Complex Questions Using Open Information Extraction , 2017, ACL.

[44]  Matt Gardner,et al.  Combining Vector Space Embeddings with Symbolic Logical Inference over Open-Domain Text , 2015, AAAI Spring Symposia.

[45]  Béla Bollobás,et al.  Random Graphs , 1985 .

[46]  Tiefeng Jiang,et al.  SPECTRAL DISTRIBUTIONS OF ADJACENCY AND LAPLACIAN MATRICES OF RANDOM GRAPHS , 2010, 1011.2608.

[47]  F. Chung,et al.  The average distances in random graphs with given expected degrees , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[48]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[49]  Mark Steedman,et al.  Universal Semantic Parsing , 2017, EMNLP.

[50]  Mihai Surdeanu,et al.  Higher-order Lexical Semantic Models for Non-factoid Answer Reranking , 2015, TACL.

[51]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[52]  Doug Downey,et al.  Local and Global Algorithms for Disambiguation to Wikipedia , 2011, ACL.