From big data to important information

Advances in science are being sought in newly available opportunities to collect massive quantities of data about complex systems. While key advances are being made in detailed mapping of systems, how to relate this data to solving many of the challenges facing humanity is unclear. The questions we often wish to address require identifying the impact of interventions on the system and that impact is not apparent in the detailed data that is available. Here we review key concepts and motivate a general framework for building larger scale views of complex systems and for characterizing the importance of information in physical, biological and social systems. We provide examples of its application to evolutionary biology with relevance to ecology, biodiversity, pandemics, and human lifespan, and in the context of social systems with relevance to ethnic violence, global food prices, and stock market panic. Framing scientific inquiry as an effort to determine what is important and unimportant is a means for advancing our understanding and addressing many practical concerns, such as economic development or treating disease.

[1]  Dan Braha,et al.  Anticipating Economic Market Crises Using Measures of Collective Panic , 2015, PloS one.

[2]  Shigeru Kondo,et al.  Reaction-Diffusion Model as a Framework for Understanding Biological Pattern Formation , 2010, Science.

[3]  A. Turing.  The chemical basis of morphogenesis , 1990 .

[4]  Adrian Cho.  Econophysics: still controversial after all these years. , 2009, Science.

[5]  Michael R. Moldover,et al.  THERMODYNAMIC ANOMALIES AT CRITICAL POINTS OF FLUIDS , 1981 .

[6]  J. Feigenbaum,et al.  A Bayesian analysis of log-periodic precursors to financial crashes , 2001 .

[7]  Yaneer Bar-Yam,et al.  Computationally tractable pairwise complexity profile , 2012, Complex..

[8]  S. Hill,et al.  Dynamic model of time-dependent complex networks. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[9]  P. Gennes.  Scaling Concepts in Polymer Physics , 1979 .

[10]  James H. Brown,et al.  A General Model for the Origin of Allometric Scaling Laws in Biology , 1997, Science.

[11]  G. Sussman,et al.  Chaotic Evolution of the Solar System , 1992, Science.

[12]  David S. Brée,et al.  Fitting the Log Periodic Power Law to financial crashes: a critical analysis , 2010, 1002.1010.

[13]  C. Domb,et al.  The Critical Point: A Historical Introduction To The Modern Theory Of Critical Phenomena , 1996 .

[14]  Dan Braha,et al.  Dynamical Response of Networks Under External Perturbations: Exact Results , 2007, 0705.4607.

[15]  Didier Sornette,et al.  Market Fluctuations: multiplicative and percolation models, size effects and predictions , 2002 .

[16]  Rosario N. Mantegna,et al.  An Introduction to Econophysics: Contents , 1999 .

[17]  L. Summers,et al.  Positive Feedback Investment Strategies and Destabilizing Rational Speculation , 1989 .

[18]  Justin Werfel,et al.  Good Fences: The Importance of Setting Boundaries for Peaceful Coexistence , 2011, PloS one.

[19]  Leo P. Kadanoff.  Scaling, Universality and Operator Algebras , 1993 .

[20]  Armin Bunde,et al.  The science of disasters : climate disruptions, heart attacks, and market crashes , 2002 .

[21]  W. Brian Arthur,et al.  Self-Reinforcing Mechanisms in Economics , 2018 .

[22]  K. Kaski,et al.  Dynamics of market correlations: taxonomy and portfolio analysis. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[23]  J. Lawton,et al.  Rare species, the coincidence of diversity hotspots and conservation strategies , 1993, Nature.

[24]  Daniel P. Faith,et al.  Genetic diversity and taxonomic priorities for conservation , 1994 .

[25]  Philip H. Dybvig,et al.  Bank Runs, Deposit Insurance, and Liquidity , 1983, Journal of Political Economy.

[26]  Yaneer Bar-Yam,et al.  Networks of Economic Market Interdependence and Systemic Risk , 2010, 1011.3707.

[27]  H. Eugene Stanley,et al.  Inverse Cubic Law for the Probability Distribution of Stock Price Variations , 1998 .

[28]  Suzanne K. Linder,et al.  A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic , 2013, PloS one.

[29]  Justin Werfel,et al.  Multilevel and kin selection in a connected world , 2010, Nature.

[30]  M. Nowak,et al.  The evolution of eusociality , 2010, Nature.

[31]  Yaneer Bar-Yam,et al.  Information flow through a chaotic channel: prediction and postdiction at finite resolution. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[32]  Irving R. Epstein,et al.  TURING STRUCTURES IN SIMPLE CHEMICAL REACTIONS , 1993 .

[33]  P. Ehrlich,et al.  Biodiversity Studies: Science and Policy , 1991, Science.

[34]  Margaret Nichols.  Trans , 2015, De-centering queer theory.

[35]  D. Sornette,et al.  Stock Market Crashes, Precursors and Replicas , 1995, cond-mat/9510036.

[36]  K. Wilson.  The renormalization group and critical phenomena , 1983 .

[37]  R. Lewontin.  ‘The Selfish Gene’ , 1977, Nature.

[38]  D. Sandbach.  All systems go. , 1986, The Health service journal.

[39]  H. Jane Brockmann,et al.  The selfish gene (2nd edn) , 1990 .

[40]  John Kenneth Galbraith,et al.  The Great Crash, 1929 , 1956 .

[41]  Yaneer Bar-Yam,et al.  Global Pattern Formation and Ethnic/Cultural Violence , 2007, Science.

[42]  R. Mantegna,et al.  Scaling behaviour in the dynamics of an economic index , 1995, Nature.

[43]  A. Bray.  Theory of phase-ordering kinetics , 1994, cond-mat/9501089.

[44]  Yaneer Bar-Yam,et al.  Analytically solvable model of probabilistic network dynamics. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[45]  Luis Bettencourt.  A unified theory of urban living It is time for a science of how city growth affects society , 2010 .

[46]  Wei-Hao Wang,et al.  Studies , 1926 .

[47]  David R. Nelson,et al.  Recent developments in phase transitions and critical phenomena , 1977, Nature.

[48]  Justin Werfel,et al.  The evolution of reproductive restraint through social communication. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[49]  Yaneer Bar-Yam,et al.  Long-range interactions and evolutionary stability in a predator-prey system. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[50]  Olivier V. Pictet,et al.  From the bird's eye to the microscope: A survey of new stylized facts of the intra-daily foreign exchange markets , 1997, Finance Stochastics.

[51]  F. Lillo,et al.  Topology of correlation-based minimal spanning trees in real and model markets. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[52]  K. Gaston.  Global patterns in biodiversity , 2000, Nature.

[53]  Yaneer Bar-Yam,et al.  Beyond the Mean Field in Host-Pathogen Spatial Ecology , 2011 .

[54]  S. Carpenter,et al.  Early-warning signals for critical transitions , 2009, Nature.

[55]  Y. Bar-Yam,et al.  Multiscale analysis of information correlations in an infinite-range, ferromagnetic Ising system. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[56]  K. Wilson.  Renormalization Group and Critical Phenomena. I. Renormalization Group and the Kadanoff Scaling Picture , 1971 .

[57]  Albert-László Barabási,et al.  Statistical mechanics of complex networks , 2001, ArXiv.

[58]  D. E. Aspnes,et al.  Static Phenomena Near Critical Points: Theory and Experiment , 1967 .

[59]  R. Mittermeier,et al.  Biodiversity hotspots for conservation priorities , 2000, Nature.

[60]  H Sayama,et al.  Relationship between measures of fitness and time scale in evolution. , 2002, Physical review letters.

[61]  Florian Wagener,et al.  Complex Evolutionary Systems in Behavioral Finance , 2008 .

[62]  P B Hamilton,et al.  Genetic variation in strains of zebrafish (Danio rerio) and the implications for ecotoxicology studies , 2009, Ecotoxicology.

[63]  Y. Bar-Yam.  Improving the effectiveness of health care and public health: a multiscale complex systems analysis. , 2006, American journal of public health.

[64]  A. M. Turing,et al.  The chemical basis of morphogenesis , 1952, Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences.

[65]  I. Cockburn,et al.  The Economics of Reproducibility in Preclinical Research , 2015, PLoS biology.

[66]  D. Sornette,et al.  Significance of log-periodic precursors to financial crashes , 2001, cond-mat/0106520.

[67]  C. Kindleberger.  MANIAS, PANICS, AND CRASHES , 1996 .

[68]  E. Ben-Jacob,et al.  Dominating Clasp of the Financial Sector Revealed by Partial Correlation Analysis of the Stock Market , 2010, PloS one.

[69]  R. Rajesh,et al.  Persistence properties of a system of coagulating and annihilating random walkers. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[70]  A. Galecki,et al.  Exotic mice as models for aging research: polemic and prospectus. , 1999, Neurobiology of aging.

[71]  E. L. Quarantelli,et al.  Panic, Sociology of , 2001 .

[72]  Franziska Wulf,et al.  Mathematical Population Genetics , 2016 .

[73]  Benoit B. Mandelbrot,et al.  Fractal Geometry of Nature , 1984 .

[74]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[75]  R. Crozier.  Preserving the Information Content of Species: Genetic Diversity, Phylogeny, and Conservation Worth , 1997 .

[76]  Yaneer Bar-Yam,et al.  UPDATE February 2012 - The Food Crises: Predictive validation of a quantitative model of food prices including speculators and ethanol conversion , 2012 .

[77]  K. L. Judd.  Complex Evolutionary Systems in Behavioral Finance, In: Handbook of Financial Markets: Dynamics and Evolution, Edited by T. Hens and K.R. , 2011 .

[78]  E. L. Quarantelli,et al.  The Sociology Of Panic , 2001 .

[79]  R. Palmer,et al.  Time series properties of an artificial stock market , 1999 .

[80]  Edwin H. Lowe,et al.  Do population size bottlenecks reduce evolutionary potential? , 1999 .

[81]  R M May,et al.  Extinction and the loss of evolutionary history. , 1997, Science.

[82]  Charles W. Calomiris,et al.  The Origins of Banking Panics: Models, Facts, and Bank Regulation , 1991 .

[83]  M Arunachalam,et al.  Population genomics of wild and laboratory zebrafish (Danio rerio) , 2011, Molecular ecology.

[84]  H. Stanley,et al.  Spontaneous recovery in dynamical networks , 2013, Nature Physics.

[85]  B. Widom,et al.  Book Review:The Critical Point. A Historical Introduction to the Modern Theory of Critical Phenomena. Cyril Domb, Taylor and Francis, London, 1996 , 1998 .

[86]  Hiroki Sayama,et al.  Introduction to the Modeling and Analysis of Complex Systems , 2015 .

[87]  E. Fama.  EFFICIENT CAPITAL MARKETS: A REVIEW OF THEORY AND EMPIRICAL WORK* , 1970 .

[88]  P. Gopikrishnan,et al.  Inverse cubic law for the distribution of stock price variations , 1998, cond-mat/9803374.

[89]  Y. Bar-Yam.  Making Things Work: Solving Complex Problems in a Complex World , 2004 .

[90]  Dan Braha,et al.  Predicting Economic Market Crises Using Measures of Collective Panic , 2011, ArXiv.

[91]  James A. Feigenbaum.  More on a statistical analysis of log-periodic precursors to financial crashes , 2001 .

[92]  A. Mawson.  Understanding Mass Panic and Other Collective Responses to Threat and Disaster , 2005, Psychiatry.

[93]  Eshel Ben-Jacob,et al.  Dependency Network and Node Influence: Application to the Study of Financial Markets , 2012, Int. J. Bifurc. Chaos.

[94]  L Mann,et al.  A study of economic panic: the "run" on the Hindmarsh Building Society. , 1976, Sociometry.

[95]  L Kaufman,et al.  Symmetry breaking and coarsening in spatially distributed evolutionary processes including sexual reproduction and disruptive selection. , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[96]  Yaneer Bar-Yam,et al.  An Information-Theoretic Formalism for Multiscale Structure in Complex Systems , 2014, 1409.4708.

[97]  Thomas Bury,et al.  Predicting trend reversals using market instantaneous state , 2013, 1310.8169.

[98]  M. Kardar.  Statistical physics of fields , 2007 .

[99]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[100]  R. Mantegna.  Hierarchical structure in financial markets , 1998, cond-mat/9802256.

[101]  Yaneer Bar-Yam,et al.  Multiscale complexity of correlated Gaussians. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[102]  Yaneer Bar-Yam,et al.  Dynamics Of Complex Systems , 2019 .

[103]  C. Moritz.  Defining 'Evolutionarily Significant Units' for conservation. , 1994, Trends in ecology & evolution.

[104]  James A. Feigenbaum,et al.  DISCRETE SCALE INVARIANCE IN STOCK MARKETS BEFORE CRASHES , 1996 .

[105]  W. Ewens.  Mathematical Population Genetics , 1980 .

[106]  Zhang,et al.  Dynamic scaling of growing interfaces. , 1986, Physical review letters.

[107]  Yaneer Bar-Yam,et al.  A mathematical theory of strong emergence using multiscale variety , 2004, Complex..

[108]  S. Carpenter,et al.  Anticipating Critical Transitions , 2012, Science.

[109]  Justin Werfel,et al.  Programed Death is Favored by Natural Selection in Spatial Systems. , 2015, Physical review letters.

[110]  C. Peng,et al.  Long-range correlations in nucleotide sequences , 1992, Nature.

[111]  L. Bettencourt,et al.  A unified theory of urban living , 2010, Nature.

[112]  Andy Gardner,et al.  Adaptation and the Evolution of Parasite Virulence in a Connected World , 2009 .

[113]  Shlomo Havlin,et al.  The structural role of weak and strong links in a financial market network , 2008, 0805.2477.

[114]  H. Stanley,et al.  Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. , 1995, Chaos.

[115]  N.Vandewalle,et al.  Self-organized critical topology of stock markets , 2000 .

[116]  Andrew Balmford,et al.  When does conservation genetics matter? , 2001, Heredity.

[117]  Mark Kon,et al.  A New Phylogenetic Diversity Measure Generalizing the Shannon Index and Its Application to Phyllostomid Bats , 2009, The American Naturalist.

[118]  Lev Muchnik,et al.  Statistical Mechanics of Conventional Traders May Lead to Non-Conventional Market Behavior , 2003 .

[119]  Yaneer Bar-Yam,et al.  Multiscale Complexity/Entropy , 2004, Adv. Complex Syst..

[120]  D. Sornette,et al.  Large financial crashes , 1997, cond-mat/9704127.

[121]  Dan Braha,et al.  From Centrality to Temporary Fame: Dynamic Centrality in Complex Networks , 2006, Complex..

[122]  P Heller,et al.  Experimental investigations of critical phenomena , 1967 .

[123]  D. Sornette,et al.  Endogenous versus Exogenous Origins of Crises , 2004, physics/0412026.

[124]  C. Tyler,et al.  Are toxicological responses in laboratory (inbred) zebrafish representative of those in outbred (wild) populations? - A case study with an endocrine disrupting chemical. , 2011, Environmental science & technology.

[125]  Yaneer Bar-Yam,et al.  Moran model as a dynamical process on networks and its implications for neutral speciation. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[126]  Ulf Dieckmann,et al.  Speciation along environmental gradients , 2003, Nature.

[127]  R. A. Fisher,et al.  The Genetical Theory of Natural Selection , 1931 .

[128]  Yaneer Bar-Yam,et al.  Multiscale variety in complex systems , 2004, Complex..

[129]  Y. Bar-Yam,et al.  Global patterns of speciation and diversity , 2009, Nature.

[130]  R. Punnett,et al.  The Genetical Theory of Natural Selection , 1930, Nature.

[131]  W. Arthur,et al.  The Economy as an Evolving Complex System II , 1988 .

[132]  J. Poterba,et al.  What moves stock prices? , 1988 .

[133]  M. Dehmer,et al.  Influence of the Time Scale on the Construction of Financial Networks , 2010, PloS one.

[134]  W. Ewens.  Mathematical Population Genetics : I. Theoretical Introduction , 2004 .

[135]  Yaneer Bar-Yam,et al.  The Food Crises: A Quantitative Model of Food Prices Including Speculators and Ethanol Conversion , 2011, 1109.4859.

[136]  N. Smelser.  Theory Of Collective Behavior , 1963 .

[137]  K. Wilson.  Problems in Physics with many Scales of Length , 1979 .

[138]  Sandra C. Greer,et al.  Thermodynamic anomalies near the critical point of steam , 1972 .

[139]  M. Marchesi,et al.  Scaling and criticality in a stochastic multi-agent model of a financial market , 1999, Nature.

[140]  Yaneer Bar-Yam,et al.  Formalizing the gene centered view of evolution , 1999, Adv. Complex Syst..

[141]  Laurent Lehmann,et al.  Inclusive fitness theory and eusociality , 2011, Nature.

[142]  Yaneer Bar-Yam,et al.  When systems engineering fails-toward complex systems engineering , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).