The learning of a set of p random patterns in a linear perceptron is studied in the limit of a large (N) of input units with noise on the weights, inputs and output. The problem is formulated in continuous time as a Langevin equation, and the first task is to evaluate the response or Green function for the system. White noise on the output is shown to correspond to spatially correlated weight noise acting only in a subspace of the weight space. It is shown that the input noise acts as a simple weight decay with a size proportional to the load parameter alpha =p/N. With no weight decay, the relaxation time diverges at alpha =1. With a weight decay it becomes shorter, and finite for alpha =1, but at the cost of a larger asymptotic learning error that is found analytically. It is shown that a small weight decay decreases the effect of noise on the weights or outputs.
[1]
Thomas M. Cover,et al.
Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition
,
1965,
IEEE Trans. Electron. Comput..
[2]
D. Amit,et al.
Statistical mechanics of neural networks near saturation
,
1987
.
[3]
E. Gardner.
The space of interactions in neural network models
,
1988
.
[4]
Esther Levin,et al.
A statistical approach to learning and generalization in layered neural networks
,
1989,
Proc. IEEE.
[5]
M. Opper.
Learning in Neural Networks: Solvable Dynamics
,
1989
.
[6]
J. Hertz,et al.
Phase transitions in simple learning
,
1989
.
[7]
The Langevin method in the statistical dynamics of learning
,
1990
.
[8]
J. Hertz,et al.
Generalization in a linear perceptron in the presence of noise
,
1992
.