The three faces of entropy for complex systems – information , thermodynamics and the maxent principle

There are three ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a means for statistical inference on multinomial Bernoulli processes (Jaynes maximum entropy principle). Even though these notions are fundamentally different concepts, the functional form of the entropy for thermodynamic systems in equilibrium, for ergodic sources in information theory, and for independent sampling processes in statistical systems, is degenerate, H(p) = − ∑ i pi log pi. For many complex systems, which are typically history-dependent, non-ergodic and non-multinomial, this is no longer the case. Here we show that for such processes the three entropy concepts lead to different functional forms of entropy. We explicitly compute these entropy functionals for three concrete examples. For Pólya urn processes, which are simple self-reinforcing processes, the source information rate is SIT = 1 1−c 1 N logN , the thermodynamical (extensive) entropy is (c, d)-entropy, SEXT = S(c,0), and the entropy in the maxent principle (MEP) is SMEP(p) = − ∑ i log pi. For sample space reducing (SSR) processes, which are simple path-dependent processes that are associated with power law statistics, the information rate is SIT = 1 + 1 2 logW , the extensive entropy is SEXT = H(p), and the maxent result is SMEP(p) = H(p/p1) +H(1− p/p1). Finally, for multinomial mixture processes, the information rate is given by the conditional entropy 〈H〉f , with respect to the mixing kernel f , the extensive entropy is given by H, and the MEP functional corresponds one-to-one to the logarithm of the mixing kernel.