Entropy and Kullback-Leibler divergence estimation based on Szegö's theorem

In this work, a new technique for the estimation of the Shannon's entropy and the Kullback-Leibler (KL) divergence for one dimensional data is presented. The estimator is based on the Szegö's theorem for sequences of Toeplitz matrices, which deals with the asymptotic behavior of the eigenvalues of those matrices, and the analogy between a probability density function (PDF) and a power spectral density (PSD), which allows us to estimate a PDF of bounded support using the well-known spectral estimation techniques. Specifically, an AR model is used for the PDF/PSD estimation, and the entropy is easily estimated as a function of the eigenvalues of the autocorrelation Toeplitz matrix. The performance of the Szegö's estimators is illustrated by means of Monte Carlo simulations and compared with previously proposed alternatives, showing a good performance.