Scaling behaviour of entropy estimates

Entropy estimation of information sources is highly non-trivial for symbol sequences with strong long-range correlations. The rabbit sequence, related to the symbolic dynamics of the nonlinear circle map at the critical point as well as the logistic map at the Feigenbaum point, is known to produce long memory tails. For both dynamical systems the scaling behaviour of the block entropy of order n has been shown to increase ∝log n. In contrast to such probabilistic concepts, we investigate the scaling behaviour of certain non-probabilistic entropy estimation schemes suggested by Lempel and Ziv (LZ) in the context of algorithmic complexity and data compression. These are applied in a sequential manner with the scaling variable being the length N of the sequence. We determine the scaling law for the LZ entropy estimate applied to the case of the critical circle map and the logistic map at the Feigenbaum point in a binary partition.

[1]  An extended diffusion-limited aggregation model with repulsive and attractive interactions , 1992 .

[2]  Peter Grassberger,et al.  Entropy estimation of symbol sequences. , 1996, Chaos.

[3]  P. Grassberger.  Toward a quantitative theory of self-generated complexity , 1986 .

[4]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.

[5]  Aldo de Luca.  A Division Property of the Fibonacci Word , 1995, Inf. Process. Lett..

[6]  P. Grassberger.  Finite sample corrections to entropy and dimension estimates , 1988 .

[7]  Werner Ebeling,et al.  Statistical physics and thermodynamics of nonlinear nonequilibrium systems : Statistical Physics 18 Satellite Meeting : Gosen near Berlin, Germany, August 10-13, 1992 : 101.WE-Heraeus-Seminar , 1993 .

[8]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[9]  Procaccia,et al.  First-return maps as a unified renormalization scheme for dynamical systems. , 1987, Physical review. A, General physics.

[10]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[11]  Gregory J. Chaitin,et al.  On the Length of Programs for Computing Finite Binary Sequences , 1966, JACM.

[12]  Aleksandr Yakovlevich Khinchin,et al.  Mathematical foundations of information theory , 1959 .

[13]  Abraham Lempel,et al.  Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.

[14]  Yuri M. Suhov,et al.  Nonparametric Entropy Estimation for Stationary Processesand Random Fields, with Applications to English Text , 1998, IEEE Trans. Inf. Theory.

[15]  W. Hilberg,et al.  Der bekannte Grenzwert der redundanzfreien Information in Texten - eine Fehlinterpretation der Shannonschen Experimente? , 1990 .

[16]  Xavier Droubay,et al.  Palindromes in the Fibonacci Word , 1995, Inf. Process. Lett..

[17]  Donald E. Knuth,et al.  The Art of Computer Programming, Volume I: Fundamental Algorithms, 2nd Edition , 1997 .

[18]  Peter Grassberger,et al.  Estimating the information content of symbol sequences and efficient codes , 1989, IEEE Trans. Inf. Theory.

[19]  Abraham Lempel,et al.  On the Complexity of Finite Sequences , 1976, IEEE Trans. Inf. Theory.

[20]  Gramss.  Entropy of the symbolic sequence for critical circle maps. , 1994, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[21]  B. McMillan.  The Basic Theorems of Information Theory , 1953 .

[22]  Aldo de Luca,et al.  A Combinatorial Property of the Fibonacci Words , 1981, Inf. Process. Lett..

[23]  Werner Ebeling,et al.  Prediction and entropy of nonlinear dynamical systems and symbolic sequences with LRO , 1997 .

[24]  Paul C. Shields,et al.  Universal redundancy rates do not exist , 1993, IEEE Trans. Inf. Theory.

[25]  B. Burrows,et al.  Recursive procedures for measuring disorder in non-periodic sequences , 1995 .