We study the learning from examples in a partially connected single layer perceptron and a two-layer network. Partially connected student networks learn from fully connected teacher networks. We study the generalization in the annealed approximation. We consider a single layer perceptron with binary weights. When a student is weakly diluted, there is a first order phase transition from the poor learning to the good learning state similar to that of fully connected perceptron. With a strong dilution, the first order phase transition disappears and the generalization error decreases continuously. We also study learning of a two-layer committee machine with binary weights. Contrary to the perceptron learning, there always exist a first order transition irrespective of dilution. The permutation symmetry is broken at the transition point and the generalization error is reduced to a non-zero minimum value.
[1]
W. Kinzel.
Physics of Neural Networks
,
1990
.
[2]
G. Kane.
Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models
,
1994
.
[3]
Vladimir Vapnik,et al.
Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics)
,
1982
.
[4]
M. V. Rossum,et al.
In Neural Computation
,
2022
.
[5]
Shun-ichi Amari,et al.
Four Types of Learning Curves
,
1992,
Neural Computation.