Supervised and Unsupervised Learning in Linear Networks

We give an overview of the main facts on supervised and unsupervised learning in networks of linear units and present several new results and open questions. In the case of back-propagation, the complete structure of the landscape of the error function and its connections to known statistical techniques such as linear regression, principal component analysis and discriminant analysis have been established. Here, we examine the dynamical aspects of the learning process, how in certain cases the spectral properties of a covariance matrix are learnt according to the order defined by the eigenvalues, and the effects of noise. In the low noise limit, we prove that the strategy adopted by the networks are unchanged whereas in the high noise limit, the solution adopted is one of complete redundancy. In the case of unsupervised learning, several algorithms based on various hebbian and anti-hebbian mechanisms are reviewed together with the structure of their fixed points. We show that three “symmetric” algorithms suggested in the literature (Oja, 1982; Williams, 1985; Baldi, 1988) are in fact equivalent. Results of simulations are presented.