Energy-Based Computation with Symmetric Hopfield Nets

We propose a unifying approach to the analysis of computatio n l aspects of symmetric Hopfield nets which is based on the concept of “en ergy source”. Within this framework we present different results concerning the computational power of various Hopfield model classes. It is shown that polynomialtime computations by nondeterministic Turing machines can be reduced to the proc ess of minimizing the energy in Hopfield nets (the MIN ENERGY problem). Furthermor e, external and internal sources of energy are distinguished. The external so urces include e.g. energizing inputs from so-called Hopfield languages, and also ce rtain external oscillators that prove finite analog Hopfield nets to be computationally T uring universal. On the other hand, the internal source of energy can be implemented by a symmetric clock subnetwork producing an exponential number of oscillation s which are used to energize the simulation of convergent asymmetric networks by Ho pfield nets. This shows that infinite families of polynomial-size Hopfield nets comp ute the complexity class PSPACE/poly. A special attention is paid to generalizing th ese results for analog states and continuous time to point out alternative sources of effic i nt computation.

[1]  S.-I. Amari,et al.  Neural theory of association and concept-formation , 1977, Biological Cybernetics.

[2]  Marcelo Bianconi.  Dynamics I: Discrete Time , 2003 .

[3]  Pekka Orponen,et al.  Computing with continuous-time Liapunov systems , 2001, STOC '01.

[4]  Jiří Šíma,et al.  The Computational Capabilities of Neural Networks , 2001 .

[5]  Pekka Orponen,et al.  On the Computational Complexity of Binary and Analog Symmetric Hopfield Nets , 2000, Neural Computation.

[6]  Pekka Orponen.  An Overview Of The Computational Power Of Recurrent Neural Networks , 2000 .

[7]  Hava T. Siegelmann,et al.  Noisy Neural Networks and Generalizations , 1999, NIPS.

[8]  Sanjeev Mahajan,et al.  Derandomizing Approximation Algorithms Based on Semidefinite Programming , 1999, SIAM J. Comput..

[9]  Eduardo D. Sontag,et al.  Analog Neural Nets with Gaussian or Other Common Noise Distributions Cannot Recognize Arbitrary Regular Languages , 1999, Neural Computation.

[10]  Pekka Orponen,et al.  On the Effect of Analog Noise in Discrete-Time Analog Computations , 1996, Neural Computation.

[11]  Jirí Wiedermann,et al.  Theory of neuromata , 1998, JACM.

[12]  Pekka Orponen,et al.  The Computational Power of Continuous Time Neural Networks , 1997, SOFSEM.

[13]  Paola Campadelli,et al.  Approximability of the Ground State Problem for Certain Ising Spin Glasses , 1997, J. Complex..

[14]  Hava T. Siegelmann,et al.  Computational power of neural networks: a characterization in terms of Kolmogorov complexity , 1997, IEEE Trans. Inf. Theory.

[15]  Pekka Orponen,et al.  Computing with Truly Asynchronous Threshold Logic Networks , 1997, Theor. Comput. Sci..

[16]  J. R,et al.  Analog Stable Simulation of Discrete Neural Networks , 1997 .

[17]  Hava T. Siegelmann,et al.  RECURRENT NEURAL NETWORKS AND FINITE AUTOMATA , 1996, Comput. Intell..

[18]  Pekka Orponen,et al.  The Computational Power of Discrete Hopfield Nets with Hidden Units , 1996, Neural Computation.

[19]  Don R. Hush,et al.  Bounds on the complexity of recurrent neural network implementations of finite state machines , 1993, Neural Networks.

[20]  Jirí Síma.  Hopfield Languages , 1995, SOFSEM.

[21]  Piotr Indyk.  Optimal Simulation of Automata by Neural Nets , 1995, STACS.

[22]  T. Kailath,et al.  Discrete Neural Computation: A Theoretical Foundation , 1995 .

[23]  Eduardo Sontag,et al.  Computational power of neural networks , 1995 .

[24]  K. Siu,et al.  Theoretical Advances in Neural Computation and Learning , 1994, Springer US.

[25]  Paola Campadelli,et al.  On the approximability of the energy function of Ising spin glasses , 1994 .

[26]  Johan Håstad,et al.  On the Size of Weights for Threshold Gates , 1994, SIAM J. Discret. Math..

[27]  Pascal Koiran.  Dynamics of Discrete Time, Continuous State Hopfield Networks , 1994, Neural Computation.

[28]  Pekka Orponen,et al.  Computational complexity of neural networks: a survey , 1994 .

[29]  Ian Parberry,et al.  Circuit complexity and neural networks , 1994 .

[30]  Pekka Orponen,et al.  Complexity Issues in Discrete Hopfield Networks , 1994 .

[31]  Hava T. Siegelmann,et al.  Analog computation via neural networks , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[32]  Patrik Floréen.  Worst-case convergence times for Hopfield memories , 1991, IEEE Trans. Neural Networks.

[33]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[34]  Mihalis Yannakakis,et al.  Simple Local Search Problems That are Hard to Solve , 1991, SIAM J. Comput..

[35]  Eric Goles Ch.,et al.  Energy Functions in Neural Networks with Continuous Local Functions , 1989, Complex Systems.

[36]  Eric Goles Ch.,et al.  Exponential Transient Classes of Symmetric Neural Networks for Synchronous and Sequential Updating , 1989, Complex Syst..

[37]  Pekka Orponen,et al.  On the Computational Complexity of Analyzing Hopfield Nets , 1989, Complex Syst..

[38]  Joseph W. Goodman,et al.  A generalized convergence theorem for neural networks , 1988, IEEE Trans. Inf. Theory.

[39]  Michael Luby,et al.  Steepest Descent Can Take Exponential Time for Symmetric Connection Networks , 1988, Complex Syst..

[40]  János Komlós,et al.  Convergence results in an associative memory model , 1988, Neural Networks.

[41]  Eric Goles,et al.  Lyapunov functions associated to automata networks , 1987 .

[42]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[43]  Ingo Wegener,et al.  The complexity of Boolean functions , 1987 .

[44]  Maurice Tchuente.  Sequential Simulation of Parallel Iterations and Applications , 1986, Theor. Comput. Sci..

[45]  Eric Goles Ch.,et al.  Dynamics of Positive Automata Networks , 1985, Theor. Comput. Sci..

[46]  Eric Goles Ch.,et al.  Decreasing energy functions as a tool for studying threshold networks , 1985, Discret. Appl. Math..

[47]  Mihalis Yannakakis,et al.  How easy is local search? , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[48]  D Psaltis,et al.  Optical implementation of the Hopfield model. , 1985, Applied optics.

[49]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[50]  Stephen Grossberg,et al.  Absolute stability of global pattern formation and parallel memory storage by competitive neural networks , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[51]  Eric Goles Ch.,et al.  Transient length in sequential iteration of threshold functions , 1983, Discret. Appl. Math..

[52]  Svatopluk Poljak,et al.  On periodical behaviour in societies with symmetric influences , 1983, Comb..

[53]  F. Barahona.  On the computational complexity of Ising spin glass models , 1982 .

[54]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Eric Goles Ch.,et al.  The Convergence of Symmetric Threshold Automata , 1981, Inf. Control..

[56]  F. Tanaka,et al.  Analytic theory of the ground state properties of a spin glass. II. XY spin glass , 1980 .

[57]  L. Bieche,et al.  On the ground states of the frustration model of a spin glass by a matching method of graph theory , 1980 .

[58]  W. A. Little,et al.  Analytic study of the memory storage capacity of a neural network , 1978 .

[59]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[60]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[61]  Saburo Muroga,et al.  Threshold logic and its applications , 1971 .

[62]  S. Muroga,et al.  Theory of majority decision elements , 1961 .

[63]  S C Kleene,et al.  Representation of Events in Nerve Nets and Finite Automata , 1951 .