A Review of Graph and Network Complexity from an Algorithmic Information Perspective

Information-theoretic-based measures have been useful in quantifying network complexity. Here we briefly survey and contrast (algorithmic) information-theoretic methods which have been used to characterize graphs and networks. We illustrate the strengths and limitations of Shannon’s entropy, lossless compressibility and algorithmic complexity when used to identify aspects and properties of complex networks. We review the fragility of computable measures on the one hand and the invariant properties of algorithmic measures on the other demonstrating how current approaches to algorithmic complexity are misguided and suffer of similar limitations than traditional statistical approaches such as Shannon entropy. Finally, we review some current definitions of algorithmic complexity which are used in analyzing labelled and unlabelled graphs. This analysis opens up several new opportunities to advance beyond traditional measures.

[1]  Thomas M. Cover,et al.  Elements of information theory (2. ed.) , 2006 .

[2]  Jean-Paul Delahaye,et al.  Two-dimensional Kolmogorov complexity and an empirical validation of the Coding theorem method by compressibility , 2012, PeerJ Comput. Sci..

[3]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[4]  A. Turing.  On Computable Numbers, with an Application to the Entscheidungsproblem. , 1937 .

[5]  Chiara Orsini,et al.  Quantifying randomness in real networks , 2015, Nature Communications.

[6]  Hector Zenil,et al.  An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems , 2017, bioRxiv.

[7]  Hector Zenil,et al.  Algorithmic complexity of motifs clusters superfamilies of networks , 2013, 2013 IEEE International Conference on Bioinformatics and Biomedicine.

[8]  Yilun Shang,et al.  Bounding Extremal Degrees of Edge-Independent Random Graphs Using Relative Entropy , 2016, Entropy.

[9]  Hector Zenil,et al.  Small Data Matters, Correlation versus Causation and Algorithmic Data Analytics , 2013 .

[10]  Gregory J. Chaitin,et al.  On the Length of Programs for Computing Finite Binary Sequences , 1966, JACM.

[11]  Matthias Dehmer,et al.  A history of graph entropy measures , 2011, Inf. Sci..

[12]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.

[13]  Hector Zenil,et al.  The Thermodynamics of Network Coding, and an Algorithmic Refinement of the Principle of Maximum Entropy † , 2018, Entropy.

[14]  Hector Zenil,et al.  Methods of information theory and algorithmic complexity for network biology. , 2014, Seminars in cell & developmental biology.

[15]  Hector Zenil,et al.  Coding-theorem like behaviour and emergence of the universal distribution from resource-bounded algorithmic probability , 2017, Int. J. Parallel Emergent Distributed Syst..

[16]  Przemyslaw Kazienko,et al.  On Measuring the Complexity of Networks: Kolmogorov Complexity versus Entropy , 2017, Complex..

[17]  M. Dehmer,et al.  Entropy Bounds for Hierarchical Molecular Networks , 2008, PloS one.

[18]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[19]  Ernesto Estrada,et al.  Walk entropies in graphs , 2013, 1303.6203.

[20]  Jean-Paul Delahaye,et al.  Numerical evaluation of algorithmic complexity for short strings: A glance into the innermost structure of randomness , 2011, Appl. Math. Comput..

[21]  Z. Wang,et al.  The structure and dynamics of multilayer networks , 2014, Physics Reports.

[22]  Cristian Claude,et al.  Information and Randomness: An Algorithmic Perspective , 1994 .

[23]  Jean-Paul Delahaye,et al.  Calculating Kolmogorov Complexity from the Output Frequency Distributions of Small Turing Machines , 2012, PloS one.

[24]  Yongtang Shi,et al.  Entropy bounds for dendrimers , 2014, Appl. Math. Comput..

[25]  Cristian S. Calude.  Information and Randomness: An Algorithmic Perspective , 1994 .

[26]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[27]  Yilun Shang,et al.  The Estrada index of evolving graphs , 2015, Appl. Math. Comput..

[28]  Harry Buhrman,et al.  Kolmogorov Random Graphs and the Incompressibility Method , 1999, SIAM J. Comput..

[29]  Hector Zenil,et al.  Low Algorithmic Complexity Entropy-deceiving Graphs , 2016, Physical review. E.

[30]  Paul M. B. Vitányi,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1993, Graduate Texts in Computer Science.

[31]  S. Shen-Orr,et al.  Network motifs: simple building blocks of complex networks. , 2002, Science.

[32]  Christopher G. Langton,et al.  Studying artificial life with cellular automata , 1986 .

[33]  János Körner,et al.  Random access communication and graph entropy , 1988, IEEE Trans. Inf. Theory.

[34]  Hector Zenil,et al.  Quantifying loss of information in network-based dimensionality reduction techniques , 2015, J. Complex Networks.

[35]  Dipendra C. Sengupta,et al.  Application of Graph Entropy in CRISPR and Repeats Detection in DNA Sequences , 2016 .

[36]  U. Alon.  Network motifs: theory and experimental approaches , 2007, Nature Reviews Genetics.

[37]  Jean-Paul Delahaye,et al.  Two-Dimensional Kolmogorov Complexity and Validation of the Coding Theorem Method by Compressibility , 2012, ArXiv.

[38]  Paul M. B. Vitányi,et al.  The miraculous universal distribution , 1997 .

[39]  Per Martin-Löf,et al.  The Definition of Random Sequences , 1966, Inf. Control..

[40]  Hector Zenil,et al.  Correlation of automorphism group size and topological properties with program−size complexity evaluations of graphs and complex networks , 2013, 1306.0322.

[41]  Hector Zenil,et al.  A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity , 2016, Entropy.

[42]  Alan M. Frieze,et al.  Random graphs , 2006, SODA '06.

[43]  G. Bianconi.  The entropy of randomized network ensembles , 2007, 0708.0153.