Finite-size effects in separable recurrent neural networks

We perform a systematic analytical study of finite-size effects in separable recurrent neural network models with sequential dynamics, away from saturation. We find two types of finite-size effects: thermal fluctuations, and disorder-induced `frozen' corrections to the mean-field laws. The finite-size effects are described by equations that correspond to a time-dependent Ornstein-Uhlenbeck process. We show how the theory can be used to understand and quantify various finite-size phenomena in recurrent neural networks, with and without detailed balance.

[1]  J. G. Taylor,et al.  Mathematical Approaches to Neural Networks , 1993 .

[2]  W. Ebeling.  Stochastic Processes in Physics and Chemistry , 1995 .

[3]  Christopher K. I. Williams.  Regression with Gaussian processes , 1997 .

[4]  M. Tsodyks,et al.  Capacity of networks with correlated attractors , 1994 .

[5]  D. Sherrington.  Stochastic Processes in Physics and Chemistry , 1983 .

[6]  R. F. Pawula,et al.  Approximation of the Linear Boltzmann Equation by the Fokker-Planck Equation , 1967 .

[7]  Haim Sompolinsky,et al.  STATISTICAL MECHANICS OF NEURAL NETWORKS , 1988 .

[8]  G. Kohring,et al.  COMMENT: Convergence time and finite size effects in neural networks , 1990 .

[9]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[10]  L. Viana.  Spin glass model for a neural network : associative memories stored with unequal weights , 1988 .

[11]  J. Zittartz,et al.  Glauber dynamics of the Little-Hopfield model , 1988 .

[12]  A. Coolen,et al.  Macroscopic Lyapunov functions for separable stochastic neural networks with detailed balance , 1995 .

[13]  H. Horner,et al.  Transients and basins of attraction in neutral network models , 1989 .

[14]  N. Kampen,et al.  Stochastic processes in physics and chemistry , 1981 .

[15]  C. Gardiner.  Handbook of Stochastic Methods , 1983 .

[16]  Laughton,et al.  Order-parameter flow in symmetric and nonsymmetric fully connected attractor neural networks near saturation. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[17]  Eytan Domany,et al.  Models of Neural Networks I , 1991 .

[18]  Sherrington,et al.  Dynamical replica theory for disordered spin systems. , 1995, Physical review. B, Condensed matter.

[19]  J. F. Mansergh,et al.  Physique , 1903, The Indian medical gazette.

[20]  A. Coolen,et al.  Image evolution in Hopfield networks. , 1988, Physical review. A, General physics.

[21]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.