Large Memory Capacity in Chaotic Artificial Neural Networks: A View of the Anti-Integrable Limit

In the literature, it was reported that the chaotic artificial neural network model with sinusoidal activation functions possesses a large memory capacity as well as a remarkable ability of retrieving the stored patterns, better than the conventional chaotic model with only monotonic activation functions such as sigmoidal functions. This paper, from the viewpoint of the anti-integrable limit, elucidates the mechanism inducing the superiority of the model with periodic activation functions that includes sinusoidal functions. Particularly, by virtue of the anti-integrable limit technique, this paper shows that any finite-dimensional neural network model with periodic activation functions and properly selected parameters has much more abundant chaotic dynamics that truly determine the model's memory capacity and pattern-retrieval ability. To some extent, this paper mathematically and numerically demonstrates that an appropriate choice of the activation functions and control scheme can lead to a large memory capacity and better pattern-retrieval ability of the artificial neural network models.

[1]  Shigetoshi Nara,et al.  Memory search using complex dynamics in a recurrent neural network model , 1993, Neural Networks.

[2]  Hong Zhao,et al.  Designing asymmetric neural networks with associative memory. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[3]  M. K. Ali,et al.  PATTERN RECOGNITION IN A NEURAL NETWORK WITH CHAOS , 1998 .

[4]  E. Zeidler.  Nonlinear functional analysis and its applications , 1988 .

[5]  Emilio Del-Moral-Hernandez,et al.  Chaotic Neural Networks , 2009, Encyclopedia of Artificial Intelligence.

[6]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[7]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[8]  P. Davis,et al.  Chaotic wandering and search in a cycle-memory neural network , 1992 .

[9]  Douglas Lind,et al.  An Introduction to Symbolic Dynamics and Coding , 1995 .

[10]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[11]  Christian Van den Broeck,et al.  Statistical Mechanics of Learning , 2001 .

[12]  T. Kohonen.  Self-Organized Formation of Correct Feature Maps , 1982 .

[13]  I. Tsuda.  Toward an interpretation of dynamic neural activity in terms of chaotic dynamical systems. , 2001, The Behavioral and brain sciences.

[14]  Shun-ichi Amari,et al.  Auto-associative memory with two-stage dynamics of nonmonotonic neurons , 1996, IEEE Trans. Neural Networks.

[15]  Kazuyuki Aihara,et al.  Threshold control of chaotic neural network , 2008, Neural Networks.

[16]  Guanrong Chen,et al.  Heteroclinical Repellers Imply Chaos , 2006, Int. J. Bifurc. Chaos.

[17]  Kazuyuki Aihara,et al.  Associative Dynamics in a Chaotic Neural Network , 1997, Neural Networks.

[18]  Liu,et al.  Associative memory with spatiotemporal chaos control. , 1996, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[19]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[20]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[21]  S. Aubry,et al.  Chaotic trajectories in the standard map. The concept of anti-integrability , 1990 .

[22]  Masahiro Nakagawa,et al.  Statistical Properties of Chaos Associative Memory , 2001 .

[23]  Shigetoshi Nara,et al.  Novel tracking function of moving target using chaotic dynamics in a recurrent neural network model , 2008, Cognitive Neurodynamics.

[24]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[25]  L. Olsen,et al.  Chaos in biological systems , 1985, Quarterly Reviews of Biophysics.

[26]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[27]  J C Sprott,et al.  Persistent chaos in high dimensions. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[28]  Guanrong Chen,et al.  On Chaotification of Discrete Systems , 2003, Int. J. Bifurc. Chaos.

[29]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[30]  Michael Menzinger,et al.  Bistable gradient networks. I. Attractors and pattern retrieval at low loading in the thermodynamic limit. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[31]  D. J. Albers,et al.  Structural stability and hyperbolicity violation in high-dimensional dynamical systems , 2004 .

[32]  Guanrong Chen,et al.  Generalized snap-back repeller and semi-conjugacy to shift operators of piecewise continuous transformations , 2007 .

[33]  S. Aubry,et al.  Breathers in nonlinear lattices: existence, linear stability and quantization , 1997 .

[34]  Masahiko Morita,et al.  Capacity of associative memory using a nonmonotonic neuron model , 1993, Neural Networks.

[35]  C. Robinson.  Dynamical Systems: Stability, Symbolic Dynamics, and Chaos , 1994 .

[36]  Steven H. Strogatz,et al.  Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering , 1994 .

[37]  J. J. Hopfield,et al.  ‘Unlearning’ has a stabilizing effect in collective memories , 1983, Nature.

[38]  Masahiro Nakagawa.  A Novel Chaos Neuron Model with a Periodic Mapping , 1997 .

[39]  K. Aihara,et al.  Associative dynamics in chaotic neural networks , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[40]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[41]  Bruno A. Olshausen,et al.  Book Review , 2003, Journal of Cognitive Neuroscience.

[42]  James D. Meiss,et al.  Homoclinic bifurcations for the Hénon map , 1999, chao-dyn/9904019.

[43]  Steven H. Strogatz,et al.  Nonlinear Dynamics and Chaos , 2024 .

[44]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[45]  S. Nara.  Can potentially useful dynamics to solve complex problems emerge from constrained chaos and/or chaotic itinerancy? , 2003, Chaos.

[46]  James D. Meiss,et al.  Computing periodic orbits using the anti-integrable limit , 1998, chao-dyn/9802014.

[47]  Kazuyuki Aihara,et al.  Strange attractors in chaotic neural networks , 2000 .

[48]  Jack W. Tsao,et al.  Observed brain dynamics, P.P. Mitra, H. Bokil. Oxford University Press (2008), ISBN-13: 978-0-19-517808-1, 381 pages, $65.00 , 2009 .

[49]  Ichiro Tsuda,et al.  Dynamic link of memory--Chaotic memory map in nonequilibrium neural networks , 1992, Neural Networks.

[50]  Kanter,et al.  Neural networks and the solution of nonlinear equations. , 1990, Physical review letters.

[51]  Kazuyuki Aihara,et al.  Global searching ability of chaotic neural networks , 1999 .

[52]  K. Aihara,et al.  Chaos and asymptotical stability in discrete-time neural networks , 1997, chao-dyn/9701020.

[53]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[54]  Masahiko Morita,et al.  Associative memory with nonmonotone dynamics , 1993, Neural Networks.

[55]  James D. Meiss,et al.  Cantori for symplectic maps near the anti-integrable limit , 1992 .

[56]  Kazuyuki Aihara,et al.  Global bifurcation structure of chaotic neural networks and its application to traveling salesman problems , 1995, Neural Networks.

[57]  Nonlinear functional analysis and its applications, part I: Fixed-point theorems , 1991 .

[58]  Jason Brownlee,et al.  Complex adaptive systems , 2007 .

[59]  B. Cessac,et al.  Mean-field equations, bifurcation map and route to chaos in discrete time neural networks , 1994 .

[60]  D. J. Albers,et al.  Mathematik in den Naturwissenschaften Leipzig Routes to chaos in high-dimensional dynamical systems : a qualitative numerical study , 2006 .

[61]  Yi-Chiuan Chen.  Bernoulli shift for second order recurrence relations near theanti-integrable limit , 2005 .

[62]  Robert S. MacKay,et al.  Proof of existence of breathers for time-reversible or Hamiltonian networks of weakly coupled oscillators , 1994 .

[63]  Ping Zhu,et al.  Controlling chaos in a chaotic neural network , 2003, Neural Networks.

[64]  Kazuyuki Aihara,et al.  Chaotic simulated annealing by a neural network model with transient chaos , 1995, Neural Networks.

[65]  Ping Zhu,et al.  CONTROLLING CHAOS IN A NEURAL NETWORK BASED ON THE PHASE SPACE CONSTRAINT , 2003 .