Collective phenomena in neural networks

In this paper we review some central notions of the theory of neural networks. In so doing we concentrate on collective aspects of the dynamics of large networks. The neurons are usually taken to be formal but this is not a necessary requirement for the central notions to be applicable. Formal neurons just make the theory simpler.

[1]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[2]  O. Lanford.  ENTROPY AND EQUILIBRIUM STATES IN CLASSICAL STATISTICAL MECHANICS , 1973 .

[3]  D. Amit,et al.  Perceptron learning with sign-constrained weights , 1989 .

[4]  Thomas B. Kepler,et al.  Optimal learning in neural network memories , 1989 .

[5]  R. Palmer,et al.  The replica method and solvable spin glass model , 1979 .

[6]  J. Hemmen,et al.  Elementary solution of Classical Spin-Glass models , 1986 .

[7]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[8]  Kanter,et al.  Associative recall of memory without errors. , 1987, Physical review. A, General physics.

[9]  J. L. van Hemmen,et al.  Equilibrium theory of spin glasses: Mean-field theory and beyond , 1983 .

[10]  Coolen,et al.  Image evolution in Hopfield networks. , 1988, Physical review. A, General physics.

[11]  Boes,et al.  Statistical mechanics for networks of graded-response neurons. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[12]  R. Penrose.  A Generalized inverse for matrices , 1955 .

[13]  Baldi,et al.  Number of stable points for spin-glasses and neural networks of higher orders. , 1987, Physical review letters.

[14]  R. Ellis,et al.  Entropy, large deviations, and statistical mechanics , 1985 .

[15]  Stephen Grossberg,et al.  Nonlinear neural networks: Principles, mechanisms, and architectures , 1988, Neural Networks.

[16]  M. A. Virasoro,et al.  The Effect of Synapses Destruction on Categorization by Neural Networks , 1988 .

[17]  J. Lamperti.  Stochastic processes : a survey of the mathematical theory , 1979 .

[18]  J. Hemmen.  Nonlinear neural networks near saturation. , 1987 .

[19]  Alessandro Treves,et al.  Metastable states in asymmetrically diluted Hopfield networks , 1988 .

[20]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[21]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[22]  K. H. Lee,et al.  Correlation of cell body size, axon size, and signal conduction velocity for individually labelled dorsal root ganglion cells in the cat , 1986, The Journal of comparative neurology.

[23]  Westervelt,et al.  Dynamics of iterated-map neural networks. , 1989, Physical review. A, General physics.

[24]  J. J. Hopfield,et al.  ‘Unlearning’ has a stabilizing effect in collective memories , 1983, Nature.

[25]  D. Grensing,et al.  Random-site spin-glass models , 1986 .

[26]  N. D. Hayes.  Roots of the Transcendental Equation Associated with a Certain Difference‐Differential Equation , 1950 .

[27]  V. Braitenberg.  Two Views of the Cerebral Cortex , 1986 .

[28]  Kinzel.  Remanent magnetization of the infinite-range Ising spin glass. , 1986, Physical review. B, Condensed matter.

[29]  Néstor Parga,et al.  The ultrametric organization of memories in a neural network , 1986 .

[30]  J. Doob.  What is a Martingale , 1971 .

[31]  J. Hemmen,et al.  The Hebb rule: storing static and dynamic objects in an associative neural network , 1988 .

[32]  G. V. Van Hoesen,et al.  Prosopagnosia , 1982, Neurology.

[33]  Alessandro Treves,et al.  Low firing rates: an effective Hamiltonian for excitatory neurons , 1989 .

[34]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[35]  Wulfram Gerstner,et al.  Encoding and Decoding of Patterns which are Correlated in Space and Time , 1990, ÖGAI.

[36]  Francis Crick,et al.  The function of dream sleep , 1983, Nature.

[37]  A. Hatley.  Mathematics in Science and Engineering , Volume 6: Differential- Difference Equations. Richard Bellman and Kenneth L. Cooke. Academic Press, New York and London. 462 pp. 114s. 6d. , 1963, The Journal of the Royal Aeronautical Society.

[38]  Robert Miller.  Representation of brief temporal patterns, Hebbian synapses, and the left-hemisphere dominance for phoneme recognition , 1987, Psychobiology.

[39]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[40]  Sompolinsky,et al.  Dynamics of spin systems with randomly asymmetric bonds: Langevin dynamics and a spherical model. , 1987, Physical review. A, General physics.

[41]  D. Hansel,et al.  Information processing in three-state neural networks , 1989 .

[42]  V S Dotsdernko,et al.  'Ordered' spin glass: a hierarchical memory machine , 1985 .

[43]  J. L. Hemmen,et al.  Nonlinear neural networks. , 1986, Physical review letters.

[44]  R. Malinow,et al.  Postsynaptic hyperpolarization during conditioning reversibly blocks induction of long-term potentiation , 1986, Nature.

[45]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[46]  B. Forrest.  Content-addressability and learning in neural networks , 1988 .

[47]  J. L. van Hemmen,et al.  Storing patterns in a spin-glass model of neural networks nears saturation , 1987 .

[48]  Opper,et al.  Learning of correlated patterns in spin-glass networks by local learning rules. , 1987, Physical review letters.

[49]  J. A. Hertz,et al.  Irreversible spin glasses and neural networks , 1987 .

[50]  L. Personnaz,et al.  Collective computational properties of neural networks: New learning mechanisms. , 1986, Physical review. A, General physics.

[51]  S. Kelso,et al.  Hebbian synapses in hippocampus. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[52]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[53]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[54]  J. L. van Hemmen,et al.  Classical Spin-Glass Model , 1982 .

[55]  J. L. van Hemmen,et al.  Non-linear neural networks with external noise , 1987 .

[56]  Sompolinsky,et al.  Neural networks with nonlinear synapses and a static noise. , 1986, Physical review. A, General physics.

[57]  W. A. Little,et al.  Analytic study of the memory storage capacity of a neural network , 1978 .

[58]  E. Gardner.  The space of interactions in neural network models , 1988 .

[59]  J. Marcinkiewicz.  Sur une propriété de la loi de Gauß , 1939 .

[60]  J. L. van Hemmen,et al.  Spin-glass models of a neural network. , 1986 .

[61]  J. Hertz,et al.  Mean-field analysis of hierarchical associative networks with 'magnetisation' , 1988 .

[62]  C. Stevens.  Strengthening the synapses , 1989, Nature.

[63]  D. Thouless,et al.  Stability of the Sherrington-Kirkpatrick solution of a spin glass model , 1978 .

[64]  I. Morgenstern,et al.  Heidelberg Colloquium on Spin Glasses , 1983 .

[65]  Riedel,et al.  Temporal sequences and chaos in neural nets. , 1988, Physical review. A, General physics.

[66]  Haim Sompolinsky,et al.  The Theory of Neural Networks: The Hebb Rule and Beyond , 1987 .

[67]  H. Horner,et al.  Transients and basins of attraction in neutral network models , 1989 .

[68]  A. Crisanti,et al.  Saturation Level of the Hopfield Model for Neural Network , 1986 .

[69]  Jean-Pierre Changeux,et al.  Learning by Selection , 1984 .

[70]  A. Komoda,et al.  On equivalent-neighbour, random site models of disordered systems , 1986 .

[71]  J. L. van Hemmen,et al.  Martingale approach to neural networks with hierarchically structured information , 1988 .

[72]  J. L. van Hemmen,et al.  Hebbian learning reconsidered: Representation of static and dynamic objects in associative neural nets , 1989, Biological Cybernetics.

[73]  Gutfreund.  Neural networks with hierarchically correlated patterns. , 1988, Physical review. A, General physics.

[74]  Andrew M. Odlyzko,et al.  On subspaces spanned by random selections of plus/minus 1 vectors , 1988, Journal of combinatorial theory. Series A.

[75]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[76]  G. Parisi.  A memory which forgets , 1986 .

[77]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[78]  Fontanari,et al.  Information storage and retrieval in synchronous neural networks. , 1987, Physical review. A, General physics.

[79]  D Kleinfeld,et al.  Sequential state generation by model neural networks. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[80]  Frumkin,et al.  Physicality of the Little model. , 1986, Physical review. A, General physics.

[81]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[82]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[83]  Pierre Peretto,et al.  On the dynamics of memorization processes , 1988, Neural Networks.

[84]  E. Gardner,et al.  Zero temperature parallel dynamics for infinite range spin glasses and neural networks , 1987 .

[85]  R. Glauber.  Time‐Dependent Statistics of the Ising Model , 1963 .

[86]  J. Nadal,et al.  Learning and forgetting on asymmetric, diluted neural networks , 1987 .

[87]  J. L. van Hemmen,et al.  Storing extensively many weighted patterns in a saturated neural network , 1987 .

[88]  Teuvo Kohonen,et al.  An Adaptive Associative Memory Principle , 1974, IEEE Transactions on Computers.

[89]  S Dehaene,et al.  Spin glass model of learning by selection. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[90]  van Aernout Enter,et al.  On a classical spin glass model , 1983 .

[91]  Freeman J. Dyson,et al.  Existence of a phase-transition in a one-dimensional Ising ferromagnet , 1969 .

[92]  G. A. Kohring,et al.  A high-precision study of the hopfield model in the phase of broken replica symmetry , 1990 .

[93]  Marc Mézard,et al.  Solvable models of working memories , 1986 .

[94]  I. Guyon,et al.  Information storage and retrieval in spin-glass like neural networks , 1985 .

[95]  Corinna Cortes,et al.  Hierarchical associative networks , 1987 .

[96]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[97]  Charles M. Newman,et al.  Memory capacity in neural network models: Rigorous lower bounds , 1988, Neural Networks.

[98]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[99]  Jan Leonard van Hemmen,et al.  Statistical-mechanical formalism for spin-glasses , 1984 .

[100]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[101]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[102]  J. Hale.  Theory of Functional Differential Equations , 1977 .

[103]  Haim Sompolinsky,et al.  STATISTICAL MECHANICS OF NEURAL NETWORKS , 1988 .

[104]  H. Stanley,et al.  Introduction to Phase Transitions and Critical Phenomena , 1972 .

[105]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[106]  J. Hemmen,et al.  Nonlinear neural networks. I. General theory , 1988 .

[107]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[108]  Lev B. Ioffe,et al.  The Augmented Models of Associative Memory Asymmetric Interaction and Hierarchy of Patterns - Int. J. Mod. Phys. B1, 51 (1987) , 1987 .