Shannon Entropy Rate of Hidden Markov Processes

Author(s): Jurgens, Alexandra M; Crutchfield, James P | Abstract: Hidden Markov chains are widely applied statistical models of stochastic processes, from fundamental physics and chemistry to finance, health, and artificial intelligence. The hidden Markov processes they generate are notoriously complicated, however, even if the chain is finite state: no finite expression for their Shannon entropy rate exists, as the set of their predictive features is generically infinite. As such, to date one cannot make general statements about how random they are nor how structured. Here, we address the first part of this challenge by showing how to efficiently and accurately calculate their entropy rates. We also show how this method gives the minimal set of infinite predictive features. A sequel addresses the challenge's second part on structure.

[1]  Alex Bateman,et al.  An introduction to hidden Markov models. , 2007, Current protocols in bioinformatics.

[2]  Aaron A. King,et al.  Time series analysis via mechanistic models , 2008, 0802.0021.

[3]  Michael F. Barnsley,et al.  Fractals everywhere , 1988 .

[4]  L. Goddard.  Information Theory , 1962, Nature.

[5]  Rolando Cavazos-Cadena,et al.  An alternative derivation of Birkhoff's formula for the contraction coefficient of a positive matrix , 2003 .

[6]  James P. Crutchfield,et al.  Nearly Maximally Predictive Features and Their Dimensions , 2017, Physical review. E.

[7]  James P. Crutchfield,et al.  Information Anatomy of Stochastic Equilibria , 2014, Entropy.

[8]  Susanne Still,et al.  Optimal causal inference: estimating stored information and approximating causal architecture. , 2007, Chaos.

[9]  Gerd Folkers,et al.  On computable numbers , 2016 .

[10]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[11]  Mohammad Rezaeian.  Hidden Markov Process: A New Representation, Entropy Rate and Estimation Entropy , 2006, ArXiv.

[12]  Jianxin Wu.  Hidden Markov model , 2018 .

[13]  J. Rogers.  Chaos , 1876 .

[14]  Tsachy Weissman,et al.  New bounds on the entropy rate of hidden Markov processes , 2004, Information Theory Workshop.

[15]  J. Crutchfield,et al.  Regularities unseen, randomness observed: levels of entropy convergence. , 2001, Chaos.

[16]  A. N. Kolmogorov.  Combinatorial foundations of information theory and the calculus of probabilities , 1983 .

[17]  A. Turing.  On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[18]  Hidden Markov models for stochastic thermodynamics , 2015 .

[19]  James P. Crutchfield,et al.  Functional Thermodynamics of Maxwellian Ratchets: Constructing and Deconstructing Patterns, Randomizing and Derandomizing Behaviors , 2020 .

[20]  J. Elton.  An ergodic theorem for iterated maps , 1987, Ergodic Theory and Dynamical Systems.

[21]  Philippe Jacquet,et al.  On the entropy of a hidden Markov process , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[22]  J. Crutchfield.  The calculi of emergence: computation, dynamics and induction , 1994 .

[23]  T. Rydén,et al.  Stylized Facts of Daily Return Series and the Hidden Markov Model , 1998 .

[24]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[25]  Elon Kohlberg,et al.  The Contraction Mapping Approach to the Perron-Frobenius Theory: Why Hilbert's Metric? , 1982, Math. Oper. Res..

[26]  James P. Crutchfield,et al.  Spectral Simplicity of Apparent Complexity, Part II: Exact Complexities and Complexity Spectra , 2017, Chaos.

[27]  Aaron D. Wyner,et al.  A Universal Turing Machine with Two Internal States , 1993 .

[28]  A. N. Kolmogorov,et al.  Foundations of the theory of probability , 1960 .

[29]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[30]  Naftali Tishby,et al.  Past-future information bottleneck in dynamical systems. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[31]  Young,et al.  Inferring statistical complexity. , 1989, Physical review letters.

[32]  James P. Crutchfield,et al.  Predictive Rate-Distortion for Infinite-Order Markov Processes , 2016 .

[33]  Ursula Dresdner,et al.  Computation Finite And Infinite Machines , 2016 .

[34]  Brian H. Marcus,et al.  Analyticity of Entropy Rate of Hidden Markov Chains , 2005, IEEE Transactions on Information Theory.

[35]  James Odell,et al.  Between order and chaos , 2011, Nature Physics.

[36]  Karol Zyczkowski,et al.  Entropy computing via integration over fractal measures. , 1998, Chaos.

[37]  Ewan Birney,et al.  Hidden Markov models in biological sequence analysis , 2001, IBM J. Res. Dev..

[38]  James P. Crutchfield,et al.  Exact Complexity: The Spectral Decomposition of Intrinsic Computation , 2013, ArXiv.

[39]  Vladimir B. Balakirsky,et al.  On the entropy rate of a hidden Markov model , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[40]  John J. Birch.  Approximations for the Entropy for Functions of Markov Chains , 1962 .

[41]  Ya. G. Sinai,et al.  On the Notion of Entropy of a Dynamical System , 2010 .

[42]  James P. Crutchfield,et al.  Computational Mechanics: Pattern and Prediction, Structure and Simplicity , 1999, ArXiv.

[43]  G. Birkhoff.  Extensions of Jentzsch’s theorem , 1957 .

[44]  J. Davenport.  Editor , 1960 .