On Criticality in High-Dimensional Data

Data sets with high dimensionality such as natural images, speech, and text have been analyzed with methods from condensed matter physics. Here we compare recent approaches taken to relate the scale invariance of natural images to critical phenomena. We also examine the method of studying high-dimensional data through specific heat curves by applying the analysis to noncritical systems: 1D samples taken from natural images and 2D binary pink noise. Through these examples, we concluded that due to small sample sizes, specific heat is not a reliable measure for gauging whether high-dimensional data are critical. We argue that identifying order parameters and universality classes is a more reliable way to identify criticality in high-dimensional data.

[1]  K. Wilson,et al.  The Renormalization group and the epsilon expansion , 1973 .

[2]  Shang‐keng Ma.  Modern Theory of Critical Phenomena , 1976 .

[3]  K. Wilson.  Problems in Physics with many Scales of Length , 1979 .

[4]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[5]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[6]  P. Howe,et al.  Multicritical points in two dimensions, the renormalization group and the ϵ expansion , 1989 .

[7]  William Bialek,et al.  Statistics of Natural Images: Scaling in the Woods , 1993, NIPS.

[8]  J. V. van Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[9]  J. H. Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998 .

[10]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[11]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[12]  Michael J. Berry,et al.  Weak pairwise correlations imply strongly correlated network states in a neural population , 2005, Nature.

[13]  Michael J. Berry,et al.  Spin glass models for a network of real neurons , 2009, 0912.5409.

[14]  R. Segev,et al.  Sparse low-order interaction network underlies a highly correlated and learnable neural population code , 2011, Proceedings of the National Academy of Sciences.

[15]  M. Bethge,et al.  Common input explains higher-order correlations and entropy in a simple model of neural population activity. , 2011, Physical review letters.

[16]  H. B. Barlow,et al.  Possible Principles Underlying the Transformations of Sensory Messages , 2012 .

[17]  W. Bialek,et al.  Statistical thermodynamics of natural images. , 2008, Physical review letters.

[18]  Saeed Saremi,et al.  Hierarchical model of natural images and the origin of scale invariance , 2013, Proceedings of the National Academy of Sciences.

[19]  Terrence J. Sejnowski,et al.  Correlated Percolation, Fractal Structures, and Scale-Invariant Distribution of Clusters in Natural Images , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.