The behavior of a linear computing unit is analyzed during learning by gradient descent of a cost function equal to the sum of a variance maximization and a weight normalization term. The landscape of this cost function is shown to be composed of one local maximum, a set of saddle points, and one global minimum aligned with the principal components of the input patterns. It is possible to describe the cost landscape in terms of the hyperspheres, hypercrests, and hypervalleys associated with each of these principal components. Using this description, it is possible to show that the learning trajectory will converge to the global minimum of the landscape under certain conditions of the starting weights and learning rate of the descent procedure. Furthermore, it is possible to provide a precise description of the learning trajectory in this cost landscape. Extensions and implications of the algorithm are discussed by using networks of such cells.<<ETX>>
[1]
E. Oja.
Simplified neuron model as a principal component analyzer
,
1982,
Journal of mathematical biology.
[2]
R. Linsker,et al.
From basic network principles to neural architecture
,
1986
.
[3]
Geoffrey E. Hinton,et al.
Learning internal representations by error propagation
,
1986
.
[4]
R Linsker,et al.
From basic network principles to neural architecture: emergence of spatial-opponent cells.
,
1986,
Proceedings of the National Academy of Sciences of the United States of America.
[5]
M. Hirsch.
CONVERGENCE IN NEURAL NETS.
,
1987
.
[6]
Ralph Linsker,et al.
Self-organization in a perceptual network
,
1988,
Computer.
[7]
Kurt Hornik,et al.
Neural networks and principal component analysis: Learning from examples without local minima
,
1989,
Neural Networks.