Optimal Coding and the Origins of Zipfian Laws

The problem of compression in standard information theory consists of assigning codes as short as possible to numbers. Here we consider the problem of optimal coding -- under an arbitrary coding scheme -- and show that it predicts Zipf's law of abbreviation, namely a tendency in natural languages for more frequent words to be shorter. We apply this result to investigate optimal coding also under so-called non-singular coding, a scheme where unique segmentation is not warranted but codes stand for a distinct number. Optimal non-singular coding predicts that the length of a word should grow approximately as the logarithm of its frequency rank, which is again consistent with Zipf's law of abbreviation. Optimal non-singular coding in combination with the maximum entropy principle also predicts Zipf's rank-frequency distribution. Furthermore, our findings on optimal non-singular coding challenge common beliefs about random typing. It turns out that random typing is in fact an optimal coding process, in stark contrast with the common assumption that it is detached from cost cutting considerations. Finally, we discuss the implications of optimal coding for the construction of a compact theory of Zipfian laws and other linguistic laws.

[1]  J. Schilperoord,et al.  Linguistics , 1999 .

[2]  Amiyaal Ilany,et al.  The “Law of Brevity” in animal communication: Sex‐specific signaling optimization is determined by call amplitude rather than duration , 2019, Evolution letters.

[3]  Peter Elias,et al.  Universal codeword sets and representations of the integers , 1975, IEEE Trans. Inf. Theory.

[4]  Ramon Ferrer-i-Cancho,et al.  Random Texts Do Not Exhibit the Real Zipf's Law-Like Rank Distribution , 2010, PloS one.

[5]  Ramon Ferrer-i-Cancho,et al.  Optimization Models of Natural Communication , 2014, J. Quant. Linguistics.

[6]  G. A. Miller,et al.  Finitary models of language users , 1963 .

[7]  Eugene Kharitonov,et al.  Anti-efficient encoding in emergent communication , 2019, NeurIPS.

[8]  David Lusseau,et al.  Compression as a Universal Principle of Animal Behavior , 2013, Cogn. Sci..

[9]  Brockway McMillan,et al.  Two inequalities implied by unique decipherability , 1956, IRE Trans. Inf. Theory.

[10]  Ramon Ferrer-i-Cancho,et al.  The polysemy of the words that children learn over time , 2016, Interaction Studies.

[11]  J. Weijer,et al.  Word length, sentence length and frequency: Zipf revisited , 2004 .

[12]  Govindasamy Agoramoorthy,et al.  Efficiency of coding in macaque vocal communication , 2010, Biology Letters.

[13]  R. F. Cancho,et al.  The global minima of the communicative energy of natural communication systems , 2007 .

[14]  M. Newman.  Power laws, Pareto distributions and Zipf's law , 2005 .

[15]  Simon Kirby,et al.  Zipf’s Law of Abbreviation and the Principle of Least Effort: Language users optimise a miniature lexicon for efficient communication , 2017, Cognition.

[16]  H. K. Kesavan,et al.  Jaynes' Maximum Entropy Principle , 2009, Encyclopedia of Optimization.

[17]  Gabriel Altmann,et al.  Word Length and Word Frequency , 2007 .

[18]  David Lusseau,et al.  Efficient coding in dolphin surface behavioral patterns , 2009, Complex..

[19]  M. Porter,et al.  Critical Truths About Power Laws , 2012, Science.

[20]  W. Marsden.  I and J , 2012 .

[21]  S. Naranan,et al.  Information theoretic models in statistical linguistics. I: A model for word frequencies , 1992 .

[22]  Francesc Font-Clos,et al.  Large-Scale Analysis of Zipf’s Law in English Texts , 2015, PloS one.

[23]  Michael Cysouw,et al.  The Entropy of Words - Learnability and Expressivity across More than 1000 Languages , 2017, Entropy.

[24]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[25]  J. Hailman,et al.  The ‘chick-a-dee’ calls of Parus atricapillus: A recombinant system of animal communication compared with written English , 1985 .

[26]  Campbell B. Read,et al.  Zipf's Law , 2004 .

[27]  Cheng-shi Liu,et al.  MAXIMAL NON-SYMMETRIC ENTROPY LEADS NATURALLY TO ZIPF'S LAW , 2008 .

[28]  Jiang Feng,et al.  Brevity is prevalent in bat short-range communication , 2013, Journal of Comparative Physiology A.

[29]  Adrian Akmajian,et al.  Linguistics: An Introduction to Language and Communication , 1979 .

[30]  Peter Harremoës,et al.  Maximum Entropy Fundamentals , 2001, Entropy.

[31]  Henri Guiter.  Les relations /fréquence-longueur-sens/ des mots (langues romanes et anglais) , 1978 .

[32]  Khuram Shahzad,et al.  The organization of domains in proteins obeys Menzerath-Altmann’s law of language , 2015, BMC Systems Biology.

[33]  Marco Gamba,et al.  Do penguins’ vocal sequences conform to linguistic laws? , 2020, Biology Letters.

[34]  Mill Johannes G.A. Van,et al.  Transmission Of Information , 1961 .

[35]  Ramon Ferrer Cancho,et al.  The frequency spectrum of finite samples from the intermittent silence process , 2009 .

[36]  R. F. Cancho.  Euclidean distance between syntactically linked words. , 2004 .

[37]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[38]  G. Miller,et al.  Some effects of intermittent silence. , 1957, The American journal of psychology.

[39]  R. Ferrer i Cancho,et al.  The variation of Zipf's law in human language , 2005 .

[40]  A. Radford,et al.  Brevity is not always a virtue in primate communication , 2011, Biology Letters.

[41]  Paul A. Garber,et al.  Male gibbon loud morning calls conform to Zipf's law of brevity and Menzerath's law: insights into the origin of human language , 2020, Animal Behaviour.

[42]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[43]  R. Moreno-Sánchez,et al.  Exacerbating the Tragedy of the Commons: Private Inefficient Outcomes and Peer Effect in Experimental Games with Fishing Communities , 2016, PloS one.

[44]  Ramon Ferrer-i-Cancho,et al.  Zipf's law of abbreviation as a language universal , 2016 .

[45]  Alexa R. Romberg,et al.  Statistical learning and language acquisition. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[46]  Holger Klinck,et al.  Brevity is not a universal in animal communication: evidence for compression depends on the unit of analysis in small ape vocalizations , 2020, Royal Society Open Science.

[47]  Sebastian Bernhardsson,et al.  Zipf's law unzipped , 2011, ArXiv.

[48]  Ramon Ferrer-i-Cancho,et al.  The Failure of the Law of Brevity in Two New World Primates. Statistical Caveats. , 2012, ArXiv.

[49]  Michael Mitzenmacher,et al.  A Brief History of Generative Models for Power Law and Lognormal Distributions , 2004, Internet Math..

[50]  Jack P. Hailman,et al.  A model of repetitive behaviour illustrated by chickadee calling , 1978, Animal Behaviour.

[51]  Monica Borda,et al.  Fundamentals in Information Theory and Coding , 2011 .

[52]  Steven T. Piantadosi,et al.  The communicative function of ambiguity in language , 2011, Cognition.

[53]  John R. Buck,et al.  The use of Zipf's law in animal communication analysis , 2005, Animal Behaviour.

[54]  Michael Ramscar,et al.  Source codes in human communication , 2019, ArXiv.

[55]  Wentian Li,et al.  Random texts exhibit Zipf's-law-like word frequency distribution , 1992, IEEE Trans. Inf. Theory.

[56]  Anastasios A. Tsonis,et al.  Zipf's law and the structure and evolution of languages , 1997, Complex..

[57]  Ramon Ferrer-i-Cancho,et al.  The challenges of statistical patterns of language: The case of Menzerath's law in genomes , 2012, Complex..

[58]  Ramon Ferrer i Cancho,et al.  Decoding least effort and scaling in signal frequency distributions , 2005 .

[59]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[60]  Ramon Ferrer-i-Cancho,et al.  Linguistic laws in chimpanzee gestural communication , 2019, Proceedings of the Royal Society B.

[61]  Ramon Ferrer-i-Cancho,et al.  The frequency spectrum of finite samples from the intermittent silence process , 2009, J. Assoc. Inf. Sci. Technol..

[62]  Marco Beccuti,et al.  A versatile mathematical work-flow to explore how Cancer Stem Cell fate influences tumor progression , 2015, BMC Systems Biology.

[63]  Ramon Ferrer i Cancho,et al.  Euclidean distance between syntactically linked words. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[64]  J. N. Kapur,et al.  Entropy Optimization Principles and Their Applications , 1992 .

[65]  Ramon Ferrer-i-Cancho,et al.  Gelada vocal sequences follow Menzerath’s linguistic law , 2016, Proceedings of the National Academy of Sciences.

[66]  S. Naranan,et al.  Information theoretic models in statistical linguistics. II: Word frequencies and hierarchical structure in language-statistical tests , 1992 .

[67]  Matt Visser,et al.  Zipf's law, power laws and maximum entropy , 2012, 1212.5567.

[68]  David R. Anderson,et al.  Model Selection and Multimodel Inference , 2003 .

[69]  M. E. J. Newman,et al.  Power laws, Pareto distributions and Zipf's law , 2005 .

[70]  Olivier Morin,et al.  When iconicity stands in the way of abbreviation: No Zipfian effect for figurative signals , 2019, PloS one.