Layered neural networks

Some of the recent work done on layered feed-forward networks is reviewed. First we describe exact solutions for the dynamics of such networks, which are expected to respond to an input by going through a sequence of preassigned states on the various layers. The family of networks considered has a variety of interlayer couplings: linear and nonlinear Hebbian, Hebbian with Gaussian synaptic noise and with various kinds of dilution, and the pseudoinverse (projector) matrix of couplings. In all cases our solutions take the form of layer-to-layer recursions for the mean overlap with a (random) key pattern and for the width of the embedding field distribution. Dynamics is governed by the fixed points of these recursions. For all cases nontrivial domains of attraction of the memory states are found. Next we review studies of unsupervised leaming in such networks and the emergence of orientation-selective cells. Finally the main ideas of three supervised leaming procedures, recendy introduced for layered networks, are oudined. All three procedures are based on a search in the space of intemal representations; one is designed for leaming in networks with fixed architecture and has no associated convergence theorem, whereas the other two are guaranteed to converge but may require expansion of the network by an uncontrolled number of hidden units.

[1]  Ron Meir,et al.  Storing and Retrieving Information in a Layered Spin System , 1986 .

[2]  Werner Krauth,et al.  Critical storage capacity of the J = ± 1 neural network , 1989 .

[3]  Thomas B. Kepler,et al.  Optimal learning in neural network memories , 1989 .

[4]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[5]  Kanter,et al.  Associative recall of memory without errors. , 1987, Physical review. A, General physics.

[6]  E. Gardner.  The space of interactions in neural network models , 1988 .

[7]  W. Kinzel,et al.  Layered neural networks , 1989 .

[8]  Wolfgang Kinzel,et al.  Statistical Mechanics of Neural Networks , 1989 .

[9]  Eytan Domany.  Neural networks: A biased overview , 1988 .

[10]  Haim Sompolinsky,et al.  The Theory of Neural Networks: The Hebb Rule and Beyond , 1987 .

[11]  Anders Krogh,et al.  Dynamics of Learning in Simple Perceptrons , 1989 .

[12]  H. Horner,et al.  Transients and basins of attraction in neutral network models , 1989 .

[13]  Thomas B. Kepler,et al.  Domains of attraction in neural networks , 1988 .

[14]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[15]  J. L. Hemmen,et al.  Nonlinear neural networks. , 1986, Physical review letters.

[16]  D. J. Wallace,et al.  Training with noise and the storage of correlated patterns in a neural network model , 1989 .

[17]  Marc Mézard,et al.  The roles of stability and symmetry in the dynamics of neural networks , 1988 .

[18]  Opper,et al.  Learning of correlated patterns in spin-glass networks by local learning rules. , 1987, Physical review letters.

[19]  Marvin Minsky,et al.  Perceptrons: expanded edition , 1988 .

[20]  Meir,et al.  Chaotic behavior of a layered neural network. , 1988, Physical review. A, General physics.

[21]  Eytan Domany,et al.  Stochastic dynamics of a layered neural network. Exact solution , 1987 .

[22]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[23]  Sompolinsky,et al.  Neural networks with nonlinear synapses and a static noise. , 1986, Physical review. A, General physics.

[24]  Bernard Derrida,et al.  Statistical properties of randomly broken objects and of multivalley structures in disordered systems , 1987 .

[25]  and C.L. Coates Lewis,et al.  Threshold Logic , 1967 .

[26]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[27]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[28]  R Linsker,et al.  From basic network principles to neural architecture: emergence of spatial-opponent cells. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[29]  R Linsker,et al.  From basic network principles to neural architecture: emergence of orientation-selective cells. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[30]  Professor Moshe Abeles,et al.  Local Cortical Circuits , 1982, Studies of Brain Function.

[31]  Eytan Domany,et al.  Learning by Choice of Internal Representations , 1988, Complex Syst..

[32]  Wolfgang Kinzel,et al.  Basins of attraction near the critical storage capacity for neural networks with constant stabilities , 1989 .

[33]  Bernard Widrow,et al.  Neural nets for adaptive filtering and adaptive pattern recognition , 1988, Computer.

[34]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[35]  Marc Mézard,et al.  Basins of Attraction in a Perception-like Neural Network , 1988, Complex Syst..

[36]  John E. Hopcroft,et al.  Synthesis of Minimal Threshold Logic Networks , 1965, IEEE Trans. Electron. Comput..

[37]  Kanter.  Convergence time in infinite-range neural networks with parallel dynamics at zero temperature. , 1989, Physical review. A, General physics.

[38]  Meir,et al.  Layered feed-forward neural network with exactly soluble dynamics. , 1988, Physical review. A, General physics.

[39]  Meir,et al.  Iterated learning in a layered feed-forward neural network. , 1988, Physical review. A, General physics.

[40]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[41]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[42]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[43]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[44]  Ron Meir.  Extensions of a solvable feed forward neural network , 1988 .

[45]  Meir,et al.  Exact solution of a layered neural network model. , 1987, Physical review letters.

[46]  R Linsker,et al.  From basic network principles to neural architecture: emergence of orientation columns. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[47]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[48]  Pierre Peretto,et al.  On the dynamics of memorization processes , 1988, Neural Networks.

[49]  M. Opper.  Learning in Neural Networks: Solvable Dynamics , 1989 .

[50]  J. Hemmen.  Nonlinear neural networks near saturation. , 1987 .

[51]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[52]  B. Derrida.  Dynamical phase transition in nonsymmetric spin glasses , 1987 .

[53]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.