On the Number of Limit Cycles in Diluted Neural Networks

We consider the storage properties of temporal patterns, i.e. cycles of finite lengths, in neural networks represented by (generally asymmetric) spin glasses defined on random graphs. Inspired by the observation that dynamics on sparse systems have more basins of attractions than the dynamics of densely connected ones, we consider the attractors of a greedy dynamics in sparse topologies, considered as proxy for the stored memories. We enumerate them using numerical simulation and extend the analysis to large systems sizes using belief propagation. We find that the logarithm of the number of such cycles is a non monotonic function of the mean connectivity and we discuss the similarities with biological neural networks describing the memory capacity of the hippocampus.

[1]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[2]  Giorgio Parisi,et al.  On the number of limit cycles in asymmetric neural networks , 2018, 1810.09325.

[3]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[4]  Carlo Baldassi,et al.  Theory and learning protocols for the material tempotron model , 2013 .

[5]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[6]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[7]  G. Parisi,et al.  Relaxation, closing probabilities and transition from oscillatory to chaotic attractors in asymmetric neural networks , 1998, cond-mat/9803224.

[8]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[9]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[10]  Y. Kabashima,et al.  Dynamics of asymmetric kinetic Ising systems revisited , 2013, 1310.5003.

[11]  K Nutzel,et al.  The length of attractors in asymmetric random neural networks with deterministic dynamics , 1991 .

[12]  E. Gardner,et al.  Zero temperature parallel dynamics for infinite range spin glasses and neural networks , 1987 .

[13]  G. E. Alexander,et al.  Neuron Activity Related to Short-Term Memory , 1971, Science.

[14]  Menno P. Witter,et al.  Connectivity of the Hippocampus , 2010 .

[15]  G. Parisi,et al.  Attractors in fully asymmetric neural networks , 1997, cond-mat/9708215.

[16]  H. Gutfreund,et al.  The nature of attractors in an asymmetric spin glass with deterministic dynamics , 1988 .

[17]  Taro Toyoizumi,et al.  Structure of attractors in randomly connected networks. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[18]  Moritz Helias,et al.  Optimal Sequence Memory in Driven Random Networks , 2016, Physical Review X.

[19]  F. Tanaka,et al.  Analytic theory of the ground state properties of a spin glass. II. XY spin glass , 1980 .

[20]  D. Saad,et al.  Slow spin dynamics and self-sustained clusters in sparsely connected systems. , 2017, Physical Review E.

[21]  Sompolinsky,et al.  Dynamics of spin systems with randomly asymmetric bonds: Ising spins and Glauber dynamics. , 1988, Physical review. A, General physics.

[22]  Lenka Zdeborová,et al.  Dynamic message-passing equations for models with unidirectional dynamics , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[23]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[24]  Y. Miyashita.  Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[25]  H Sompolinsky,et al.  Dynamics of random neural networks with bistable units. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[26]  W. Heisenberg.  Zur Theorie des Ferromagnetismus , 1928 .

[27]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[28]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[29]  Brad E. Pfeiffer,et al.  Autoassociative dynamics in the generation of sequences of hippocampal place cells , 2015, Science.

[30]  A. Crisanti,et al.  Path integral approach to random neural networks , 2018, Physical Review E.

[31]  Edmund T. Rolls,et al.  Cortical Attractor Network Dynamics with Diluted Connectivity , 2011 .

[32]  Edmund T. Rolls,et al.  Advantages of dilution in the connectivity of attractor networks in the brain , 2012, BICA 2012.

[33]  Giancarlo Ruocco,et al.  Effect of dilution in asymmetric recurrent neural networks , 2018, Neural Networks.

[34]  M. Mézard,et al.  The Bethe lattice spin glass revisited , 2000, cond-mat/0009418.

[35]  M. Mézard,et al.  Spin Glass Theory And Beyond: An Introduction To The Replica Method And Its Applications , 1986 .

[36]  Schuster,et al.  Suppressing chaos in neural networks by noise. , 1992, Physical review letters.

[37]  Misha Tsodyks,et al.  Chaos in Highly Diluted Neural Networks , 1991 .

[38]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.