Constant conditional entropy and related hypotheses

Constant entropy rate (conditional entropies must remain constant as the sequence length increases) and uniform information density (conditional probabilities must remain constant as the sequence length increases) are two information theoretic principles that are argued to underlie a wide range of linguistic phenomena. Here we revise the predictions of these principles in the light of Hilberg's law on the scaling of conditional entropy in language and related laws. We show that constant entropy rate (CER) and two interpretations for uniform information density (UID), full UID and strong UID, are inconsistent with these laws. Strong UID implies CER but the reverse is not true. Full UID, a particular case of UID, leads to costly uncorrelated sequences that are totally unrealistic. We conclude that CER and its particular cases are incomplete hypotheses about the scaling of conditional entropies.

[1]  W. Hilberg,et al.  Der bekannte Grenzwert der redundanzfreien Information in Texten - eine Fehlinterpretation der Shannonschen Experimente? , 1990 .

[2]  George Kingsley Zipf,et al.  Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology , 2012 .

[3]  B. Sanders.  Review of entangled coherent states , 2011, 1112.1778.

[4]  Claude E. Shannon,et al.  Prediction and Entropy of Printed English , 1951 .

[5]  Lukasz Debowski,et al.  On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts , 2008, IEEE Transactions on Information Theory.

[6]  Eugene Charniak,et al.  Entropy Rate Constancy in Text , 2002, ACL.

[7]  T. Florian Jaeger,et al.  Redundancy and reduction: Speakers manage syntactic information density , 2010, Cognitive Psychology.

[8]  R. Baayen,et al.  Putting the bits together: an information theoretical perspective on morphological processing , 2004, Cognition.

[9]  Amy Perfors,et al.  Why are some word orders more common than others? A uniform information density account , 2010, NIPS.

[10]  A. S.,et al.  Estimating the Entropy of DNA Sequences , 1997 .

[11]  Daniel Polani,et al.  Phase transitions in least-effort communications , 2010 .

[12]  Thomas Hofmann,et al.  Speakers optimize information density through syntactic reduction , 2007 .

[13]  Fermin Moscoso del Prado.  The universal "shape" of human languages: spectral analysis beyond speech , 2011 .

[14]  Marcelo A. Montemurro,et al.  Long-range fractal correlations in literary corpora , 2002, ArXiv.

[15]  W. Ebeling,et al.  Entropy and Long-Range Correlations in Literary English , 1993, cond-mat/0204108.

[16]  R. Ferrer i Cancho,et al.  Zipf's law from a communicative phase transition , 2005 .

[17]  Aaron D. Wyner,et al.  Prediction and Entropy of Printed English , 1993 .

[18]  R. F. Cancho,et al.  The global minima of the communicative energy of natural communication systems , 2007 .

[19]  J-P Eckmann,et al.  Hierarchical structures induce long-range dynamical correlations in written texts. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Eduardo G. Altmann,et al.  On the origin of long-range correlations in texts , 2012, Proceedings of the National Academy of Sciences.

[21]  GUSTAV HERDAN.  QUANTITATIVE LINGUISTICS OR GENERATIVE GRAMMAR? , 1964 .

[22]  Rajesh P. N. Rao,et al.  Probabilistic Analysis of an Ancient Undeciphered Script , 2010, Computer.

[23]  F. M. D. P. Martín,et al.  The universal ‘ shape ’ of human languages : spectral analysis beyond speech , 2011 .

[24]  Lukasz Dkebowski,et al.  Excess entropy in natural language: present state and perspectives , 2011, Chaos.

[25]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .