An Enriched Category Theory of Language: From Syntax to Semantics

Given a piece of text, the ability to generate a coherent extension of it implies some sophistication, including a knowledge of grammar and semantics. In this paper, we propose a mathematical framework for passing from probability distributions on extensions of given texts to an enriched category containing semantic information. Roughly speaking, we model probability distributions on texts as a category enriched over the unit interval. Objects of this category are expressions in language and hom objects are conditional probabilities that one expression is an extension of another. This category is syntactical—it describes what goes with what. We then pass to the enriched category of unit intervalvalued copresheaves on this syntactical category to find semantic information.

[1]  Tai-Danae Bradley,et al.  Language Modeling with Reduced Densities , 2020, Compositionality.

[2]  Oleg Viro,et al.  Dequantization of Real Algebraic Geometry on Logarithmic Paper , 2000, math/0005163.

[3]  F. William Lawvere,et al.  Adjointness in Foundations , 1969 .

[4]  A. Louis,et al.  Documenta Mathematica , 1996 .

[5]  Paul M. Pietroski,et al.  Conjoining Meanings: Semantics Without Truth Values , 2018 .

[6]  Juan Luis Gastaldi,et al.  Why Can Computers Understand Natural Language? , 2020 .

[7]  G. Mikhalkin,et al.  Brief introduction to tropical geometry , 2015, 1502.05950.

[8]  Zellig S. Harris,et al.  Distributional Structure , 1954 .

[9]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[10]  O. Bagasra,et al.  Proceedings of the National Academy of Sciences , 1914, Science.

[11]  Stephen Clark,et al.  Mathematical Foundations for a Compositional Distributional Model of Meaning , 2010, ArXiv.

[12]  Proceedings of the IEEE , 2018, IEEE Journal of Emerging and Selected Topics in Power Electronics.

[13]  I. Moerdijk,et al.  Sheaves in geometry and logic: a first introduction to topos theory , 1992 .

[14]  Liwen Zhang,et al.  Tropical Geometry of Deep Neural Networks , 2018, ICML.

[15]  F. William Lawvere,et al.  Metric spaces, generalized logic, and closed categories , 1973 .

[16]  Petros Maragos,et al.  Tropical Geometry and Machine Learning , 2021, Proceedings of the IEEE.

[17]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[18]  Petros Maragos,et al.  A Tropical Approach to Neural Networks with Piecewise Linear Activations , 2018, ArXiv.

[19]  Guillaume Rabusseau,et al.  Tensor Networks for Probabilistic Sequence Modeling , 2020, AISTATS.

[20]  Robin Myers,et al.  Word , 2021, The Massachusetts Review.

[21]  B. Stöger.  An Invitation to Applied Category Theory. Seven Sketches in Compositionality. By Brendan Fong and David I. Spivak. Cambridge University Press, 2019. Paperback, pp. 348. Price GBP 37.99. ISBN 9781108711821. , 2021, Acta crystallographica. Section A, Foundations and advances.

[22]  A. Gabriel.  Editor , 2018, Best "New" African Poets 2018 Anthology.

[23]  David E. Speyer,et al.  The tropical Grassmannian , 2003, math/0304218.

[24]  Juan Luis Gastaldi,et al.  The calculus of language: explicit representation of emergent linguistic structure through type-theoretical paradigms , 2021, Interdisciplinary Science Reviews.

[25]  Petros Maragos,et al.  Tropical Polynomial Division and Neural Networks , 2019, ArXiv.

[26]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[27]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[28]  Editors , 2003 .

[29]  Omer Levy,et al.  Emergent linguistic structure in artificial neural networks trained by self-supervision , 2020, Proceedings of the National Academy of Sciences.

[30]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[31]  Fahd Ali Al-Agl,et al.  Theory and Applications of Categories , 1993 .

[32]  F. William Lawvere,et al.  Taking categories seriously , 1986 .

[33]  F. William Lawvere,et al.  Conceptual Mathematics: A First Introduction to Categories , 1997 .

[34]  Victor Nistor,et al.  Advances in Geometry , 1998 .

[35]  G. M. Kelly,et al.  BASIC CONCEPTS OF ENRICHED CATEGORY THEORY , 2022, Elements of ∞-Category Theory.

[36]  S. Willerton.  TIGHT SPANS, ISBELL COMPLETIONS AND SEMI-TROPICAL MODULES , 2013, 1302.4370.

[37]  Jungang Xu,et al.  A Survey on Neural Network Language Models , 2019, ArXiv.

[38]  B. Sturmfels,et al.  Tropical Convexity , 2003, math/0308254.

[39]  D. Maclagan.  Introduction to tropical algebraic geometry , 2012, 1207.1925.

[40]  P. Horwich.  A use theory of meaning , 2004 .

[41]  lawa Kanas,et al.  Metric Spaces , 2020, An Introduction to Functional Analysis.

[42]  吉富 修平.  Generators of modules in tropical geometry , 2010 .