An information-theoretic primer on complexity, self-organization, and emergence

Complex Systems Science aims to understand concepts like complexity, self-organization, emergence and adaptation, among others. The inherent fuzziness in complex systems definitions is complicated by the unclear relation among these central processes: does self-organisation emerge or does it set the preconditions for emergence? Does complexity arise by adaptation or is complexity necessary for adaptation to arise? The inevitable consequence of the current impasse is miscommunication among scientists within and across disciplines. We propose a set of concepts, together with their possible information-theoretic interpretations, which can be used to facilitate the Complex Systems Science discourse. Our hope is that the suggested information-theoretic baseline may promote consistent communications among practitioners, and provide new insights into the field. Published 2008 Wiley Periodicals, Inc. Complexity, 2009 This article was submitted as an invited paper resulting from the “Understanding Complex Systems” conference held at the University of Illinois at Urbana-Champaign, May 2007.

[1]  Daniel Polani,et al.  Emergence of Genetic Coding: An Information-Theoretic Model , 2007, ECAL.

[2]  A. Wuensche.  Classifying Cellular Automata Automatically , 1998 .

[3]  C. Shalizi,et al.  Causal architecture, complexity and self-organization in time series and cellular automata , 2001 .

[4]  John Hallam,et al.  From Animals to Animats 10 , 2008 .

[5]  Mikhail Prokopenko,et al.  Self-Organizing Hierarchies in Sensor and Communication Networks , 2005, Artificial Life.

[6]  M. Ridley,et al.  Nature via Nurture: Genes, Experience, and What Makes Us Human , 2005 .

[7]  Nihat Ay,et al.  Robustness and complexity co-constructed in multimodal signalling networks , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[8]  P. Grassberger.  Toward a quantitative theory of self-generated complexity , 1986 .

[9]  Luigi Lancieri.  Reusing Implicit Cooperation. A Novel Approach to Knowledge Management , 1970 .

[10]  Bruce Edmonds,et al.  Gossip, Sexual Recombination and the El Farol bar: modelling the emergence of heterogeneity , 1998, J. Artif. Soc. Soc. Simul..

[11]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[12]  James P. Crutchfield,et al.  Revisiting the Edge of Chaos: Evolving Cellular Automata to Perform Computations , 1993, Complex Syst..

[13]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[14]  R. Solé,et al.  Information Theory of Complex Networks: On Evolution and Architectural Constraints , 2004 .

[15]  Robert Haslinger,et al.  Quantifying self-organization with optimal predictors. , 2004, Physical review letters.

[16]  Hugh F. Durrant-Whyte,et al.  Measuring Global Behaviour of Multi-agent Systems from Pair-Wise Mutual Information , 2005, KES.

[17]  Dirk V. Arnold,et al.  Information-theoretic Analysis of Phase Transitions , 1996, Complex Syst..

[18]  Gregory J. Chaitin,et al.  Algorithmic Information Theory , 1987, IBM J. Res. Dev..

[19]  Gregory J. Chaitin,et al.  Information-Theoretic Limitations of Formal Systems , 1974, JACM.

[20]  Fabio Boschetti,et al.  Mapping the complexity of ecological models , 2008 .

[21]  James P. Crutchfield,et al.  Computational Mechanics: Pattern and Prediction, Structure and Simplicity , 1999, ArXiv.

[22]  Robert Shaw,et al.  The Dripping Faucet As A Model Chaotic System , 1984 .

[23]  J. Crutchfield,et al.  Thermodynamic depth of causal states: Objective complexity via minimal representations , 1999 .

[24]  Mikhail Prokopenko,et al.  Defining and Detecting Emergence in Complex Networks , 2005, KES.

[25]  Christopher G. Langton,et al.  Computation at the edge of chaos: Phase transitions and emergent computation , 1990 .

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Naftali Tishby,et al.  Complexity through nonextensivity , 2001, physics/0103076.

[28]  Albert Y. Zomaya,et al.  Local information transfer as a spatiotemporal filter for complex systems. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[29]  C. Adami,et al.  Introduction To Artificial Life , 1997, IEEE Trans. Evol. Comput..

[30]  Stephen P. Hubbell,et al.  Foraging by Bucket-Brigade in Leaf-Cutter Ants , 1980 .

[31]  Tom De Wolf,et al.  Emergence Versus Self-Organisation: Different Concepts but Promising When Combined , 2004, Engineering Self-Organising Systems.

[32]  Atocha Aliseda.  Los límites de las matemáticas. Chaitin, Gregory. The limits of mathematics: a course on information theory and the limits of formal reasoning, Singapore: Springer, 1997 , 1997 .

[33]  Alfréd Rényi,et al.  Probability Theory , 1970 .

[34]  David A. Winkler,et al.  Consistent concepts of self-organization and self-assembly , 2008 .

[35]  J. Goldstein.  The Singular Nature of Emergent Levels: Suggestions for a Theory of Emergence , 2002 .

[36]  Young,et al.  Inferring statistical complexity. , 1989, Physical review letters.

[37]  Naftali Tishby,et al.  Predictability, Complexity, and Learning , 2000, Neural Computation.

[38]  Mikhail Prokopenko,et al.  On connectivity of reconfigurable impact networks in ageless aerospace vehicles , 2005, Robotics Auton. Syst..

[39]  A. Vulpiani,et al.  Predictability: a way to characterize complexity , 2001, nlin/0101029.

[40]  Chrystopher L. Nehaniv,et al.  Relevant information in optimized persistence vs. progeny strategies , 2006 .

[41]  J. Crutchfield,et al.  Structural information in two-dimensional patterns: entropy convergence and excess entropy. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[42]  Paul Ricoeur,et al.  MEMORY AND FORGETTING , 2002 .

[43]  P. Erdos,et al.  On the strength of connectedness of a random graph , 1964 .

[44]  Chrystopher L. Nehaniv,et al.  Organization of the information flow in the perception-action loop of evolved agents , 2004, Proceedings. 2004 NASA/DoD Conference on Evolvable Hardware, 2004..

[45]  Yicheng Zhang,et al.  On the minority game: Analytical and numerical studies , 1998, cond-mat/9805084.

[46]  Mats G. Nordahl,et al.  Complexity Measures and Cellular Automata , 1988, Complex Syst..

[47]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[48]  J. Crutchfield.  The calculi of emergence: computation, dynamics and induction , 1994 .

[49]  Mikhail Prokopenko,et al.  Phase Transitions in Self-Organising Sensor Networks , 2003, ECAL.

[50]  J. L. Casti.  Would-be worlds : the science and surprise of artificial worlds , 1999 .

[51]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[52]  N. Packard,et al.  Symbolic dynamics of noisy chaos , 1983 .

[53]  F. Slanina.  Social organization in the Minority Game model , 2000, cond-mat/0006098.

[54]  H. Van Dyke Parunak,et al.  Co-X: Defining what Agents Do Together , 2001 .

[55]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[56]  Mikhail Prokopenko,et al.  Evolving Spatiotemporal Coordination in a Modular Robotic System , 2006, SAB.

[57]  Stephen Wolfram,et al.  Universality and complexity in cellular automata , 1983 .

[58]  James P. Crutchfield,et al.  Discovering Noncritical Organization: Statistical Mechanical, Information Theoretic, and Computational Views of Patterns in One-Dimensional Spin Systems , 1998, Entropy.

[59]  M E J Newman.  Assortative mixing in networks. , 2002, Physical review letters.

[60]  A. Wagner.  Robustness and Evolvability in Living Systems , 2005 .

[61]  Daniel Polani,et al.  Modelling Stigmergic Gene Transfer , 2008, ALIFE.

[62]  Olaf Sporns,et al.  Mapping Information Flow in Sensorimotor Networks , 2006, PLoS Comput. Biol..

[63]  Chrystopher L. Nehaniv,et al.  All Else Being Equal Be Empowered , 2005, ECAL.

[64]  Werner Ebeling,et al.  Prediction and entropy of nonlinear dynamical systems and symbolic sequences with LRO , 1997 .

[65]  Wentian Li,et al.  On the Relationship between Complexity and Entropy for Markov Chains and Regular Languages , 1991, Complex Syst..

[66]  Cosma Rohilla Shalizi.  Optimal Nonlinear Prediction of Random Fields on Networks , 2003, DMCS.

[67]  Gregory. J. Chaitin,et al.  Algorithmic information theory , 1987, Cambridge tracts in theoretical computer science.

[68]  J. Crutchfield,et al.  Statistical complexity of simple one-dimensional spin systems , 1997, cond-mat/9702191.

[69]  J. Hopcroft,et al.  Are randomly grown graphs really random? , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[70]  Cosma Rohilla Shalizi,et al.  Blind Construction of Optimal Nonlinear Recursive Predictors for Discrete Sequences , 2004, UAI.

[71]  Marco Dorigo,et al.  Ant system: optimization by a colony of cooperating agents , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[72]  Peter A. Corning,et al.  The re-emergence of "emergence": A venerable concept in search of a theory , 2002, Complex..

[73]  Joaquín J. Torres,et al.  Control of neural chaos by synaptic noise , 2007, Biosyst..

[74]  Daniel Polani,et al.  Information Flows in Causal Networks , 2008, Adv. Complex Syst..

[75]  D. Hofstadter,et al.  Gödel, Escher, Bach: An Eternal Golden Braid@@@Godel, Escher, Bach: An Eternal Golden Braid , 1980 .

[76]  Daniel Polani,et al.  Optimizing Potential Information Transfer with Self-referential Memory , 2006, UC.

[77]  C. Adami.  What is complexity? , 2002, BioEssays : news and reviews in molecular, cellular and developmental biology.

[78]  Schreiber,et al.  Measuring information transfer , 2000, Physical review letters.

[79]  J. Crutchfield,et al.  Regularities unseen, randomness observed: levels of entropy convergence. , 2001, Chaos.

[80]  L V Beloussov,et al.  A common biomechanical model for the formation of stationary cell domains and propagating waves in the developing organisms , 2005, Computer methods in biomechanics and biomedical engineering.

[81]  K. Eriksson,et al.  Structural Information in Self-Organizing Systems , 1987 .

[82]  David Batten,et al.  Are some human ecosystems self-defeating? , 2007, Environ. Model. Softw..