Mutual information functions versus correlation functions

This paper studies one application of mutual information to symbolic sequences: the mutual information functionM(d). This function is compared with the more frequently used correlation functionΓ(d). An exact relation betweenM(d) andΓ(d) is derived for binary sequences. For sequences with more than two symbols, no such general relation exists; in particular,Γ(d)=0 may or may not lead toM(d)=0. This linear, but not general, independence between symbols separated by a distance is studied for ternary sequences. Also included is the estimation of the finite-size effect on calculating mutual information. Finally, the concept of “symbolic noise” is discussed.

[1]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[2]  Claude E. Shannon,et al.  Prediction and Entropy of Printed English , 1951 .

[3]  Ga Miller,et al.  Note on the bias of information estimates , 1955 .

[4]  Samuel Karlin,et al.  A First Course on Stochastic Processes , 1968 .

[5]  J. Kingman.  A FIRST COURSE IN STOCHASTIC PROCESSES , 1967 .

[6]  Aristid Lindenmayer,et al.  Mathematical Models for Cellular Interactions in Development , 1968 .

[7]  Samuel Karlin.  COMPOUNDING STOCHASTIC PROCESSES , 1968 .

[8]  A. Lindenmayer.  Mathematical models for cellular interactions in development. I. Filaments with one-sided inputs. , 1968, Journal of theoretical biology.

[9]  G. J. Chaltin,et al.  To a mathematical definition of 'life' , 1970, SIGA.

[10]  Lila L. Gatlin,et al.  Information theory and the living system , 1972 .

[11]  B. Harris.  The Statistical Estimation of Entropy in the Non-Parametric Case , 1975 .

[12]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[13]  G. Chaitin,et al.  TOWARD A MATHEMATICAL DEFINITION OF “ LIFE ” , 1979 .

[14]  守屋 悦朗,et al.  J.E.Hopcroft, J.D. Ullman 著, "Introduction to Automata Theory, Languages, and Computation", Addison-Wesley, A5変形版, X+418, \6,670, 1979 , 1980 .

[15]  S. Karlin,et al.  A second course in stochastic processes , 1981 .

[16]  V. Alekseev,et al.  Symbolic dynamics and hyperbolic dynamic systems , 1981 .

[17]  Robert Shaw,et al.  The Dripping Faucet As A Model Chaotic System , 1984 .

[18]  Fraser,et al.  Independent coordinates for strange attractors from mutual information. , 1986, Physical review. A, General physics.

[19]  P. Grassberger.  Toward a quantitative theory of self-generated complexity , 1986 .

[20]  Wentian Li.  Power Spectra of Regular Languages and Cellular Automata , 1987, Complex Syst..

[21]  Wentian Li,et al.  Mutual Information Functions of Natural Language Texts , 1989 .

[22]  A. Fraser.  Reconstructing attractors from scalar time series: A comparison of singular system and redundancy criteria , 1989 .

[23]  Wentian Li,et al.  Spatial 1/f spectra in open dynamical systems , 1989 .

[24]  Li,et al.  Expansion-modification systems: A model for spatial 1/f spectra. , 1991, Physical review. A, Atomic, molecular, and optical physics.