Functorial Language Models

We introduce functorial language models: a principled way to compute probability distributions over word sequences given a monoidal functor from grammar to meaning. This yields a method for training categorical compositional distributional (DisCoCat) models on raw text data. We provide a proof-of-concept implementation in DisCoPy, the Python toolbox for monoidal categories.

[1]  B. Coecke,et al.  Quantum Natural Language Processing on Near-Term Quantum Computers , 2020, QPL.

[2]  Martha Lewis,et al.  Generalized relations in linguistics & cognition , 2018, Theor. Comput. Sci..

[3]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[4]  Mehrnoosh Sadrzadeh,et al.  Experimental Support for a Categorical Compositional Distributional Model of Meaning , 2011, EMNLP.

[5]  Ronnie Cann,et al.  Grammars as Parsers: Meeting the Dialogue Challenge , 2006 .

[6]  Mehrnoosh Sadrzadeh,et al.  Exploring Semantic Incrementality with Dynamic Syntax and Vector Space Semantics , 2018, ArXiv.

[7]  Dimitri Kartsaklis,et al.  Reasoning about Meaning in Natural Language with Compact Closed Categories and Frobenius Algebras , 2014, ArXiv.

[8]  W. Bruce Croft,et al.  A Language Modeling Approach to Information Retrieval , 1998, SIGIR Forum.

[9]  Bob Coecke,et al.  Towards Compositional Distributional Discourse Analysis , 2018, CAPNS@QI.

[10]  Mehrnoosh Sadrzadeh,et al.  Lambek vs. Lambek: Functorial vector space semantics and string diagrams for Lambek calculus , 2013, Ann. Pure Appl. Log..

[11]  Dimitri Kartsaklis,et al.  Sentence entailment in compositional distributional semantics , 2015, Annals of Mathematics and Artificial Intelligence.

[12]  Giovanni de Felice,et al.  Functorial Question Answering , 2019, Electronic Proceedings in Theoretical Computer Science.

[13]  Mehrnoosh Sadrzadeh,et al.  Incremental Monoidal Grammars , 2020, ArXiv.

[14]  Stephen Clark,et al.  Mathematical Foundations for a Compositional Distributional Model of Meaning , 2010, ArXiv.

[15]  Joachim Lambek,et al.  Type Grammar Revisited , 1997, LACL.

[16]  S. Clark,et al.  A Compositional Distributional Model of Meaning , 2008 .

[17]  Joachim Lambek,et al.  Type Grammars as Pregroups , 2001, Grammars.

[18]  Bob Coecke,et al.  Foundations for Near-Term Quantum Natural Language Processing , 2020, ArXiv.

[19]  Y. Bar-Hillel.  A Quasi-Arithmetical Notation for Syntactic Description , 1953 .

[20]  J. Lambek.  The Mathematics of Sentence Structure , 1958 .

[21]  P. Selinger.  A Survey of Graphical Languages for Monoidal Categories , 2009, 0908.3347.

[22]  Dimitri Kartsaklis,et al.  Separating Disambiguation from Composition in Distributional Semantics , 2013, CoNLL.

[23]  Antonin Delpeuch.  Autonomization of Monoidal Categories , 2019, ACT.

[24]  Emil L. Post.  Recursive Unsolvability of a problem of Thue , 1947, Journal of Symbolic Logic.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.