Approximations for the entropy rate of a hidden Markov process

Let {Xt} be a stationary finite-alphabet Markov chain and {Zt} denote its noisy version when corrupted by a discrete memoryless channel. We present an approach to bounding the entropy rate of {Zt} by the construction and study of a related measure-valued Markov process. To illustrate its efficacy, we specialize it to the case of a BSC-corrupted binary Markov chain. The bounds obtained are sufficiently tight to characterize the behavior of the entropy rate in asymptotic regimes that exhibit a "concentration of the support". Examples include the 'high SNR', 'low SNR', 'rare spikes', and 'weak dependence' regimes. Our analysis also gives rise to a deterministic algorithm for approximating the entropy rate, achieving the best known precision-complexity tradeoff, for a significant subset of the process parameter space

[1]  Tsachy Weissman,et al.  The empirical distribution of rate-constrained source codes , 2004, IEEE Transactions on Information Theory.

[2]  Yuval Peres,et al.  Analytic dependence of Lyapunov exponents on transition probabilities , 1991 .

[3]  Philippe Jacquet,et al.  On the entropy of a hidden Markov process , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[4]  E. O. Elliott.  Estimates of error rates for codes on burst-noise channels , 1963 .

[5]  Tsachy Weissman,et al.  On the optimality of symbol-by-symbol filtering and denoising , 2004, IEEE Transactions on Information Theory.

[6]  Tsachy Weissman,et al.  New bounds on the entropy rate of hidden Markov processes , 2004, Information Theory Workshop.

[7]  L. Arnold,et al.  Evolutionary Formalism for Products of Positive Random Matrices , 1994 .

[8]  Michel Talagrand,et al.  The Sherrington–Kirkpatrick model: a challenge for mathematicians , 1998 .

[9]  Israel Bar-David,et al.  Capacity and coding for the Gilbert-Elliot channels , 1989, IEEE Trans. Inf. Theory.

[10]  John J. Birch.  Approximations for the Entropy for Functions of Markov Chains , 1962 .

[11]  P. Glynn,et al.  On Entropy and Lyapunov Exponents for Finite-State Channels , 2003 .

[12]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[13]  Eytan Domany,et al.  Asymptotics of the entropy rate for a hidden Markov process , 2005, Data Compression Conference.

[14]  Robert M. Gray,et al.  Information rates of autoregressive processes , 1970, IEEE Trans. Inf. Theory.

[15]  Predrag R. Jelenkovic,et al.  State Learning and Mixing in Entropy of Hidden Markov Processes and the Gilbert-Elliott Channel , 1999, IEEE Trans. Inf. Theory.

[16]  R. Gray.  Rate distortion functions for finite-state finite-alphabet Markov sources , 1971, IEEE Trans. Inf. Theory.

[17]  David J. C. MacKay,et al.  Equivalence of Linear Boltzmann Chains and Hidden Markov Models , 1996, Neural Computation.

[18]  Tsachy Weissman,et al.  Asymptotic filtering and entropy rate of a hidden Markov process in the rare transitions regime , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[19]  E. Gilbert.  Capacity of a burst-noise channel , 1960 .

[20]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[21]  Gene H. Golub,et al.  Matrix computations , 1983 .

[22]  Peter W. Glynn,et al.  Capacity of finite state markov channels with general inputs , 2003, IEEE International Symposium on Information Theory, 2003. Proceedings..