Storage capacity of a constructive learning algorithm

Upper and lower bounds for the typical storage capacity of a constructive algorithm, the tilinglike learning algorithm for the parity machine (Biehl M and Opper M 1991 Phys. Rev. A 44 6888), are determined in the asymptotic limit of large training set sizes. The properties of a perceptron with threshold, learning a training set of patterns having a biased distribution of targets, needed as an intermediate step in the capacity calculation, are determined analytically. The lower bound for the capacity, determined with a cavity method, is proportional to the number of hidden units. The upper bound, obtained with the hypothesis of replica symmetry, is close to the one predicted by Mitchinson and Durbin (1989 Biol. Cybern. 60 345).

[1]  E. Gardner.  The space of interactions in neural network models , 1988 .

[2]  Michael Biehl,et al.  The AdaTron: An Adaptive Perceptron Algorithm , 1989 .

[3]  Jong-Hoon Oh,et al.  Storage capacity of a fully-connected parity machine with continuous weights , 1998 .

[4]  A. Engel,et al.  Perceptrons above saturation , 1993 .

[5]  Opper,et al.  Tilinglike learning in the parity machine. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[6]  R. Erichsen,et al.  Optimal storage of a neural network model: a replica symmetry-breaking solution , 1993 .

[7]  Thomas B. Kepler,et al.  Optimal learning in neural network memories , 1989 .

[8]  D. Martinez,et al.  The Offset Algorithm: Building and Learning Method for Multilayer Neural Networks , 1992 .

[9]  H. Gutfreund,et al.  Learning and retrieval in attractor neural networks above saturation , 1991 .

[10]  D. Saad,et al.  Threshold-induced phase transitions in perceptrons , 1997 .

[11]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[12]  Thomas B. Kepler,et al.  Universality in the space of interactions for network models , 1989 .

[13]  Paul C. Bressloff,et al.  Low firing-rates in a compartmental model neuron , 1993 .

[14]  Eytan Domany,et al.  Learning by Choice of Internal Representations , 1988, Complex Syst..

[15]  David Saad,et al.  The statistical mechanics of constructive algorithms , 1998 .

[16]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[17]  B. Derrida,et al.  Replica symmetry instability in perceptron models , 1994 .

[18]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[19]  Replica Symmetry Breaking and the Kuhn-Tucker Cavity Method in Simple and Multilayer Perceptrons , 1996, cond-mat/9610165.

[20]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[21]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[22]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .