Statistical mechanical analysis of the dynamics of learning in perceptrons

We describe the application of tools from statistical mechanics to analyse the dynamics of various classes of supervised learning rules in perceptrons. The character of this paper is mostly that of a cross between a biased non-encyclopaedic review and lecture notes: we try to present a coherent and self-contained picture of the basics of this field, to explain the ideas and tricks, to show how the predictions of the theory compare with (simulation) experiments, and to bring together scattered results. Technical details are given explicitly in an appendix. In order to avoid distraction we concentrate the references in a final section. In addition this paper contains some new results: (i) explicit solutions of the macroscopic equations that describe the error evolution for on-line and batch learning rules; (ii) an analysis of the dynamics of arbitrary macroscopic observables (for complete and incomplete training sets), leading to a general Fokker–Planck equation; and (iii) the macroscopic laws describing batch learning with complete training sets. We close the paper with a preliminary expose´ of ongoing research on the dynamics of learning for the case where the training set is incomplete (i.e. where the number of examples scales linearly with the network size).

[1]  Reimann,et al.  Unsupervised learning by examples: On-line versus off-line. , 1996, Physical review letters.

[2]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[3]  Opper,et al.  Generalization performance of Bayes optimal classification algorithm for learning a perceptron. , 1991, Physical review letters.

[4]  October I.  Physical Review Letters , 2022 .

[5]  N. Caticha,et al.  On-line learning in parity machines , 1996 .

[6]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[7]  Saad,et al.  Exact solution for on-line learning in multilayer neural networks. , 1995, Physical review letters.

[8]  Haim Sompolinsky,et al.  STATISTICAL MECHANICS OF NEURAL NETWORKS , 1988 .

[9]  Wolfgang Kinzel,et al.  Improving a Network Generalization Ability by Selecting Examples , 1990 .

[10]  Physical Review , 1965, Nature.

[11]  Sherrington,et al.  Dynamical replica theory for disordered spin systems. , 1995, Physical review. B, Condensed matter.

[12]  Michael Biehl,et al.  The AdaTron: An Adaptive Perceptron Algorithm , 1989 .

[13]  D. Sherrington,et al.  Order-Parameter Flow in the SK Spin-Glass II: Inclusion of Microscopic Memory Effects , 1996 .

[14]  J. Hertz,et al.  Generalization in a linear perceptron in the presence of noise , 1992 .

[15]  Michael Biehl,et al.  On-Line Learning of a Time-Dependent Rule , 1992 .

[16]  J. Hertz,et al.  Phase transitions in simple learning , 1989 .

[17]  Michael Biehl,et al.  On-Line Learning with a Perceptron , 1994 .

[18]  Kim,et al.  On-line Gibbs learning. , 1996, Physical review letters.

[19]  D. Saad.  Europhysics Letters , 1997 .

[20]  W. Kinzel.  Physics of Neural Networks , 1990 .

[21]  Saad,et al.  On-line learning in soft committee machines. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[22]  Albrecht Rau,et al.  Statistical mechanics of neural networks , 1992 .

[23]  E. Gardner.  The space of interactions in neural network models , 1988 .

[24]  Györgyi,et al.  Inference of a rule by a neural network with thermal noise. , 1990, Physical review letters.

[25]  Opper.  On-line versus Off-line Learning from Random Examples: General Results. , 1996, Physical review letters.

[26]  O. Kinouchi,et al.  Lower bounds on generalization errors for drifting rules , 1993 .

[27]  Michael Biehl,et al.  Learning by on-line gradient descent , 1995 .

[28]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[29]  N. Caticha,et al.  On-line learning in the committee machine , 1995 .

[30]  D. Bedeaux,et al.  On the Relation between Master Equations and Random Walks and Their Solutions , 1971 .

[31]  Jong-Hoon Oh,et al.  Neural networks : the statistical mechanics perspective : proceedings of the CTP-PBSRI Joint Workshop on Theoretical Physics, POSTECH, Pohang, Korea, 2-4 February 95 , 1995 .

[32]  Akademii︠a︡ medit︠s︡inskikh nauk Sssr.  Journal of physics , 1939 .

[33]  Michael Biehl,et al.  Transient dynamics of on-line learning in two-layered neural networks , 1996 .

[34]  O. Kinouchi,et al.  Optimal generalization in perceptions , 1992 .

[35]  Kinouchi,et al.  On-line versus off-line learning in the linear perceptron: A comparative study. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[36]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[37]  Heskes,et al.  Learning processes in neural networks. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[38]  Sompolinsky,et al.  Local and global convergence of on-line learning. , 1995, Physical review letters.