The "off line learning approximation" in continuous time neural networks: An adiabatic theorem

In this paper we consider the justification of the ''Off Line Approximation'' in continuous time neural networks from a rigorous mathematical point of view. In real time models, the behavior of a network is characterized by two distinct dynamics evolving according different time scales, the weight dynamics which are the ''slow'' and the activation dynamics which are the ''fast.'' The ''off-line approximation'' assumes that during the learning process, neural activities are in their steady states. Such an approximation is a common dogma often used to provide an analysis of network behavior. In this paper we consider convergent networks and prove that this approximation is valid on the time scale 1/@e where @e is the learning rate parameter which controls the learning velocity. We apply these results to prove the stability of Hebbian learning in a two-layered neural network which can be seen as a continuous time version of the self-organizing Kohonen's model.

[2]  Amir F. Atiya.  Learning on a General Network , 1987, NIPS.

[3]  Morris W. Hirsch,et al.  Convergent activation dynamics in continuous time networks , 1989, Neural Networks.

[4]  M. Samuelides,et al.  Dynamical properties of neural nets using competitive activation mechanisms , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[5]  Schürmann.  Stability and adaptation in artificial neural systems. , 1989, Physical review. A, General physics.

[6]  C. Conley.  Isolated Invariant Sets and the Morse Index , 1978 .

[7]  D. McLeish.  A Maximal Inequality and Dependent Strong Laws , 1975 .

[8]  E. Coddington,et al.  Theory of Ordinary Differential Equations , 1955 .

[9]  Michail Zak,et al.  Terminal attractors in neural networks , 1989, Neural Networks.

[10]  M. Freidlin,et al.  Random Perturbations of Dynamical Systems , 1984 .

[11]  F. Verhulst,et al.  Averaging Methods in Nonlinear Dynamical Systems , 1985 .

[12]  Gail A. Carpenter,et al.  Neural network models for pattern recognition and associative memory , 1989, Neural Networks.

[13]  Fernando J. Pineda,et al.  Dynamics and architecture for neural computation , 1988, J. Complex..

[14]  J. Palis,et al.  Geometric theory of dynamical systems , 1982 .

[15]  Stephen Grossberg,et al.  Nonlinear neural networks: Principles, mechanisms, and architectures , 1988, Neural Networks.

[16]  M. Hirsch.  Systems of Differential Equations that are Competitive or Cooperative II: Convergence Almost Everywhere , 1985 .

[17]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Todd W. Troyer.  Dynamics of the Amari-Takeuchi competitive learning model , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[19]  M. Benaim,et al.  A stochastic model of neural network for unsupervised learning , 1992 .

[20]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[21]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[22]  M. Hirsch,et al.  Differential Equations, Dynamical Systems, and Linear Algebra , 1974 .

[23]  D. Kelly,et al.  Stability in contractive nonlinear neural networks , 1990, IEEE Transactions on Biomedical Engineering.

[24]  Allon Guez,et al.  On the stability, storage capacity, and design of nonlinear continuous neural networks , 1988, IEEE Trans. Syst. Man Cybern..

[25]  N. Bogolyubov,et al.  Asymptotic Methods in the Theory of Nonlinear Oscillations , 1961 .

[26]  Kiyotoshi Matsuoka,et al.  Stability conditions for nonlinear continuous neural networks with asymmetric connection weights , 1992, Neural Networks.