Structure and Randomness of Continuous-Time, Discrete-Event Processes

Loosely speaking, the Shannon entropy rate is used to gauge a stochastic process’ intrinsic randomness; the statistical complexity gives the cost of predicting the process. We calculate, for the first time, the entropy rate and statistical complexity of stochastic processes generated by finite unifilar hidden semi-Markov models—memoryful, state-dependent versions of renewal processes. Calculating these quantities requires introducing novel mathematical objects ($$\epsilon $$ϵ-machines of hidden semi-Markov processes) and new information-theoretic methods to stochastic processes.

[1]  Valérie Girardin,et al.  On the Different Extensions of the Ergodic Theorem of Information Theory , 2005 .

[2]  Karoline Wiesner,et al.  A New Method for Inferring Hidden Markov Models from Noisy Time Sequences , 2012, PloS one.

[3]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[4]  A. N. Kolmogorov,et al.  Foundations of the theory of probability , 1960 .

[5]  James P. Crutchfield,et al.  Time resolution dependence of information measures for spiking neurons: scaling and universality , 2015, Front. Comput. Neurosci..

[6]  Erik Aurell,et al.  Universal Relation between the Kolmogorov-Sinai Entropy and the Thermodynamical Entropy in Simple Liquids , 1998 .

[7]  James P. Crutchfield,et al.  Informational and Causal Architecture of Continuous-time Renewal Processes , 2016, 1611.01099.

[8]  James Odell,et al.  Between order and chaos , 2011, Nature Physics.

[9]  J. Glenn Brookshear,et al.  Theory of Computation: Formal Languages, Automata, and Complexity , 1989 .

[10]  James P. Crutchfield,et al.  Enumerating Finitary Processes , 2010, ArXiv.

[11]  Tamiki Komatsuzaki,et al.  Aggregated markov model using time series of single molecule dwell times with minimum excessive information. , 2013, Physical review letters.

[12]  Christopher C. Strelioff,et al.  Inferring Markov chains: Bayesian estimation, model comparison, entropy rate, and out-of-class modeling. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[13]  James P. Crutchfield,et al.  Computational Mechanics: Pattern and Prediction, Structure and Simplicity , 1999, ArXiv.

[14]  Gregory J. Chaitin,et al.  On the Length of Programs for Computing Finite Binary Sequences , 1966, JACM.

[15]  Young,et al.  Inferring statistical complexity. , 1989, Physical review letters.

[16]  Mervyn P. Freeman,et al.  The application of computational mechanics to the analysis of geomagnetic data , 2001 .

[17]  Chun-Biu Li,et al.  Multiscale complex network of protein conformational fluctuations in single-molecule time series , 2008, Proceedings of the National Academy of Sciences.

[18]  Christopher W. Fairall,et al.  Complexity in the atmosphere , 2000, IEEE Trans. Geosci. Remote. Sens..

[19]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[20]  Mile Gu,et al.  Occam's Vorpal Quantum Razor: Memory Reduction When Simulating Continuous-Time Stochastic Processes With Quantum Devices , 2017 .

[21]  Takuma Akimoto,et al.  Characterization of intermittency in renewal processes: application to earthquakes. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[22]  Devavrat Shah,et al.  On entropy for mixtures of discrete and continuous variables , 2006, ArXiv.

[23]  Richard W Clarke,et al.  Application of computational mechanics to the analysis of natural data: an example in geomagnetism. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[24]  Richard G. Brown,et al.  Neurovascular coupling: a parallel implementation , 2015, Front. Comput. Neurosci..

[25]  David Darmon,et al.  Predictability of User Behavior in Social Media: Bottom-Up v. Top-Down Modeling , 2013, 2013 International Conference on Social Computing.

[26]  Albert-László Barabási,et al.  Understanding individual human mobility patterns , 2008, Nature.

[27]  J. Victor.  Binless strategies for estimation of information from neural data. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[28]  W. Goldburg,et al.  Information content of turbulence. , 2012, Physical review. E, Statistical, nonlinear, and soft matter physics.

[29]  Reynaldo D. Pinto,et al.  Inferring statistical complexity in the dripping faucet experiment , 1998 .

[30]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[31]  Shunzheng Yu,et al.  Hidden semi-Markov models , 2010, Artif. Intell..

[32]  James P. Crutchfield,et al.  Chaotic Crystallography: How the physics of information reveals structural order in materials , 2014, ArXiv.

[33]  Alexander B. Neiman,et al.  Characterizing the dynamics of stochastic bistable systems by measures of complexity , 1997 .