On Measuring the Complexity of Networks: Kolmogorov Complexity versus Entropy

One of the most popular methods of estimating the complexity of networks is to measure the entropy of network invariants, such as adjacency matrices or degree sequences. Unfortunately, entropy and all entropy-based information-theoretic measures have several vulnerabilities. These measures neither are independent of a particular representation of the network nor can capture the properties of the generative process, which produces the network. Instead, we advocate the use of the algorithmic entropy as the basis for complexity definition for networks. Algorithmic entropy (also known as Kolmogorov complexity or -complexity for short) evaluates the complexity of the description required for a lossless recreation of the network. This measure is not affected by a particular choice of network features and it does not depend on the method of network representation. We perform experiments on Shannon entropy and -complexity for gradually evolving networks. The results of these experiments point to -complexity as the more robust and reliable measure of network complexity. The original contribution of the paper includes the introduction of several new entropy-deceiving networks and the empirical comparison of entropy and -complexity as fundamental quantities for constructing complexity measures for networks.

[1]  Matthias Dehmer,et al.  A history of graph entropy measures , 2011, Inf. Sci..

[2]  Stephen F. Bush,et al.  Detecting Distributed Denial-of-Service Attacks Using Kolmogorov Complexity Metrics , 2005, Journal of Network and Systems Management.

[3]  Jean-Paul Delahaye,et al.  Calculating Kolmogorov Complexity from the Output Frequency Distributions of Small Turing Machines , 2012, PloS one.

[4]  Gregory J. Chaitin,et al.  On the Length of Programs for Computing Finite Binary Sequences , 1966, JACM.

[5]  Ming Li,et al.  Kolmogorov Complexity and its Applications , 1991, Handbook of Theoretical Computer Science, Volume A: Algorithms and Complexity.

[6]  Yongtang Shi,et al.  Extremality of degree-based graph entropies , 2014, Inf. Sci..

[7]  Thomas Wilhelm,et al.  What is a complex graph , 2008 .

[8]  Albert-László Barabási,et al.  Scale-Free Networks: A Decade and Beyond , 2009, Science.

[9]  Jan Mendling,et al.  A Discourse on Complexity of Process Models , 2006, Business Process Management Workshops.

[10]  P. Erdos,et al.  On the evolution of random graphs , 1984 .

[11]  Yongtang Shi,et al.  A Note on Distance-based Graph Entropies , 2014, Entropy.

[12]  Hector Zenil,et al.  Low Algorithmic Complexity Entropy-deceiving Graphs , 2016, Physical review. E.

[13]  Hector Zenil,et al.  A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity , 2016, Entropy.

[14]  周涛,et al.  Network Entropy Based on Topology Configuration and Its Computation to Random Networks , 2008 .

[15]  Jean-Paul Delahaye,et al.  Numerical evaluation of algorithmic complexity for short strings: A glance into the innermost structure of randomness , 2011, Appl. Math. Comput..

[16]  T. Rado.  On non-computable functions , 1962 .

[17]  Mark E. J. Newman,et al.  The Structure and Function of Complex Networks , 2003, SIAM Rev..

[18]  Matthias Dehmer,et al.  Entropy and the Complexity of Graphs Revisited , 2012, Entropy.

[19]  Jürgen Schmidhuber,et al.  Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability , 1997, Neural Networks.

[20]  Sezer Sorgun.  On Randic Energy of Graphs , 2014 .

[21]  Hector Zenil,et al.  Algorithmic complexity for psychology: a user-friendly implementation of the coding theorem method , 2016, Behavior research methods.

[22]  Jorge S. Cardoso,et al.  Complexity analysis of BPEL Web processes , 2007, Softw. Process. Improv. Pract..

[23]  A. Mowshowitz,et al.  Entropy and the complexity of graphs. I. An index of the relative complexity of a graph. , 1968, The Bulletin of mathematical biophysics.

[24]  Danail Bonchev,et al.  Quantitative Measures of Network Complexity , 2005 .

[25]  Tomasz Kajdanowicz,et al.  Using Kolmogorov Complexity with Graph and Vertex Entropy to Measure Similarity of Empirical Graphs with Theoretical Graph Models , 2015 .

[26]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part I , 1964, Inf. Control..

[27]  Christos Faloutsos,et al.  On data mining, compression, and Kolmogorov complexity , 2007, Data Mining and Knowledge Discovery.

[28]  J. Körner.  Fredman-Kolmo´s bounds and information theory , 1986 .

[29]  Michael E. Orrison,et al.  The Linear Complexity of a Graph , 2006, Electron. J. Comb..

[30]  Gregory M. Constantine,et al.  Graph complexity and the laplacian matrix in blocked experiments , 1990 .

[31]  Shujuan Cao,et al.  Network Entropies Based on Independent Sets and Matchings , 2017, Appl. Math. Comput..

[32]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[33]  Matthias Dehmer,et al.  Information processing in complex networks: Graph entropy and information functionals , 2008, Appl. Math. Comput..

[34]  I. Gutman,et al.  Laplacian energy of a graph , 2006 .

[35]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[36]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[37]  H. Geiringer.  On the Foundations of Probability Theory , 1967 .

[38]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[39]  N. J. A. Sloane,et al.  The On-Line Encyclopedia of Integer Sequences , 2003, Electron. J. Comb..

[40]  Antti Latva-Koivisto,et al.  Finding a Complexity Measure for Business Process Models , 2001 .

[41]  Todd L. Veldhuizen.  Software Libraries and Their Reuse: Entropy, Kolmogorov Complexity, and Zipf's Law , 2005, ArXiv.