The dynamics of sparse random networks

Recurrent neural networks with full symmetric connectivity have been extensively studied as associative memories and pattern recognition devices. However, there is considerable evidence that sparse, asymmetrically connected, mainly excitatory networks with broadly directed inhibition are more consistent with biological reality. In this paper, we use the technique of return maps to study the dynamics of random networks with sparse, asymmetric connectivity and nonspecific inhibition. These networks show three qualitatively different kinds of behavior: fixed points, cycles of low period, and extremely long cycles verging on aperiodicity. Using statistical arguments, we relate these behaviors to network parameters and present empirical evidence for the accuracy of this statistical model. The model, in turn, leads to methods for controlling the level of activity in networks. Studying random, untrained networks provides an understanding of the intrinsic dynamics of these systems. Such dynamics could provide a substrate for the much more complex behavior shown when synaptic modification is allowed.

[1]  H. Gutfreund,et al.  The nature of attractors in an asymmetric spin glass with deterministic dynamics , 1988 .

[2]  G. G. Furman,et al.  Comparison of models for subtractive and shunting lateral-inhibition in receptor-neuron fields , 1965, Kybernetik.

[3]  Stephen Grossberg,et al.  Absolute stability of global pattern formation and parallel memory storage by competitive neural networks , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[5]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[6]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[7]  K. E. Kürten.  Critical phenomena in model neural networks , 1988 .

[8]  W. Kinzel,et al.  Hopfield network with directed bonds , 1989 .

[9]  Stephen Grossberg,et al.  Nonlinear neural networks: Principles, mechanisms, and architectures , 1988, Neural Networks.

[10]  B. Derrida,et al.  Random networks of automata: a simple annealed approximation , 1986 .

[11]  L. Squire,et al.  The medial temporal lobe memory system , 1991, Science.

[12]  Yong Yao,et al.  Model of biological pattern recognition with spatially chaotic dynamics , 1990, Neural Networks.

[13]  Ichiro Tsuda,et al.  Dynamic link of memory--Chaotic memory map in nonequilibrium neural networks , 1992, Neural Networks.

[14]  M. Segal,et al.  Serotonin attenuates a slow inhibitory postsynaptic potential in rat hippocampal neurons , 1990, Neuroscience.

[15]  J. L. van Hemmen,et al.  Hebbian learning reconsidered: Representation of static and dynamic objects in associative neural nets , 1989, Biological Cybernetics.

[16]  Shun-ichi Amari,et al.  A method of statistical neurodynamics , 1974, Kybernetik.

[17]  Gutfreund,et al.  Processing of temporal sequences in neural networks. , 1988, Physical review letters.

[18]  J Ambros-Ingerson,et al.  Simulation of paleocortex performs hierarchical clustering. , 1990, Science.

[19]  Edmund T. Rolls,et al.  The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain , 1990 .

[20]  D. Amaral,et al.  Organization of intrahippocampal projections originating from CA3 pyramidal cells in the rat , 1990, The Journal of comparative neurology.

[21]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[22]  E T Rolls,et al.  Computational constraints suggest the need for two distinct input systems to the hippocampal CA3 network , 1992, Hippocampus.

[23]  Wolfgang Kinzel,et al.  Freezing transition in asymmetric random neural networks with deterministic dynamics , 1989 .

[24]  U. Krey,et al.  On learning and recognition of temporal sequences of correlated patterns , 1990 .

[25]  J. Leo van Hemmen,et al.  Collective phenomena in neural networks , 1991 .

[27]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[28]  Riedel,et al.  Temporal sequences and chaos in neural nets. , 1988, Physical review. A, General physics.

[29]  Sompolinsky,et al.  Dynamics of spin systems with randomly asymmetric bonds: Langevin dynamics and a spherical model. , 1987, Physical review. A, General physics.

[30]  S. Amari,et al.  A Mathematical Foundation for Statistical Neurodynamics , 1977 .

[31]  Mitsuaki Yamamoto,et al.  Structural properties of network attractor associated with neuronal dynamics transition , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[32]  D Kleinfeld,et al.  Sequential state generation by model neural networks. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Tom Heskes,et al.  Retrieval of pattern sequences at variable speeds in a neural network with delays , 1992, Neural Networks.

[34]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[35]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[36]  D J Willshaw,et al.  An assessment of Marr's theory of the hippocampus as a temporary memory store. , 1990, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[37]  Franklin A. Graybill,et al.  Introduction to The theory , 1974 .

[38]  A. R. Gardner-Medwin.  The recall of events through the learning of associations between their parts , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[39]  H. Eichenbaum,et al.  The hippocampus--what does it do? , 1992, Behavioral and neural biology.

[40]  Jerry B. Marion,et al.  A mathematical foundation , 1966 .

[41]  D Marr,et al.  Simple memory: a theory for archicortex. , 1971, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[42]  Michael Schreckenberg,et al.  Glauber dynamics of the asymmetric SK-model , 1989 .

[43]  John Robinson,et al.  Statistical analysis of the dynamics of a sparse associative memory , 1992, Neural Networks.

[44]  Shun-ichi Amari,et al.  Characteristics of randomly connected threshold-element networks and network systems , 1971 .

[45]  K Nutzel,et al.  The length of attractors in asymmetric random neural networks with deterministic dynamics , 1991 .

[46]  D. Amaral,et al.  Neurons, numbers and the hippocampal network. , 1990, Progress in brain research.

[47]  A. R. Gardner-Medwin.  Doubly modifiable synapses: a model of short and long term auto-associative memory , 1989, Proceedings of the Royal Society of London. B. Biological Sciences.

[48]  E. Rolls.  Functions of neuronal networks in the hippocampus and neocortex in memory , 1989 .

[49]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[50]  B. McNaughton,et al.  Hebb-Marr networks and the neurobiological representation of action in space. , 1990 .

[51]  Joachim M. Buhmann,et al.  Storing sequences of biased patterns in neural networks with stochastic dynamics , 1988 .

[52]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[54]  W. Levy.  A computational approach to hippocampal function , 1989 .

[55]  John G. Taylor,et al.  Storing temporal sequences , 1991, Neural Networks.

[56]  J. A. Hertz,et al.  Irreversible spin glasses and neural networks , 1987 .

[57]  Kunihiko Fukushima,et al.  A model of associative memory in the brain , 1973, Kybernetik.