A recursive Renyi's entropy estimator

Estimating the entropy of a sample set is required, in solving numerous learning scenarios involving information theoretic optimization criteria. A number of entropy estimators are available in the literature; however, these require a batch of samples to operate on in order to yield an estimate. We derive a recursive formula to estimate Renyi's (1970) quadratic entropy on-line, using each new sample to update the entropy estimate to obtain more accurate results in stationary situations or to track the changing entropy of a signal in nonstationary situations.

[1]  Ralph Linsker,et al.  An Application of the Principle of Maximum Information Preservation to Linear Systems , 1988, NIPS.

[2]  S. K. Datta.  Book Reviews : THE THEORY OF ELASTIC WAVES AND WAVEGUIDES J. Miklowitz Elsevier-North Holland Pub. Co. Amsterdam, The Netherlands and New York, NY, 1978 , 1980 .

[3]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[4]  D. Erdogmus,et al.  Nonlinear channel equalization using multilayer perceptrons with information-theoretic criterion , 2001, Neural Networks for Signal Processing XI: Proceedings of the 2001 IEEE Signal Processing Society Workshop (IEEE Cat. No.01TH8584).

[5]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[6]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[7]  Deniz Erdoğmuş.  INFORMATION THEORETIC LEARNING: RENYI'S ENTROPY AND ITS APPLICATIONS TO ADAPTIVE SYSTEM TRAINING , 2002 .

[8]  K. Loparo,et al.  Optimal state estimation for stochastic systems: an information theoretic approach , 1997, IEEE Trans. Autom. Control..

[9]  D. Erdogmus,et al.  Convergence analysis of the information potential criterion in Adaline training , 2001, Neural Networks for Signal Processing XI: Proceedings of the 2001 IEEE Signal Processing Society Workshop (IEEE Cat. No.01TH8584).

[10]  Joseph J. Atick,et al.  Convergent Algorithm for Sensory Receptive Field Development , 1993, Neural Computation.

[11]  Deniz Erdogmus,et al.  Generalized information potential criterion for adaptive system training , 2002, IEEE Trans. Neural Networks.

[12]  Deniz Erdogmus,et al.  AN ON-LINE ADAPTATION ALGORITHM FOR ADAPTIVE SYSTEM TRAINING WITH MINIMUM ERROR ENTROPY: STOCHASTIC INFORMATION GRADIENT , 2001 .

[13]  J. Príncipe,et al.  Nonlinear extensions to the minimum average correlation energy filter , 1997 .

[14]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..