Applying Ockham's Razor to back-propagation

Back-propagation learning (bp) is known for its serious limitations in generalising knowledge from certain types of learning material. In this paper we describe a new learning algorithm, bp-som, which overcomes some of these limitations as is shown by application to four benchmark tasks. bp-som is a combination of a multi-layered feedforward network (mfn) trained with bp, and Kohonen's self-organising maps (soms). During the learning process, hidden-unit activations of the mfn are presented as learning vectors to self-organising maps trained in parallel. Information on classi cation errors contained in the maps is used when updating the connection weights of the network in addition to standard error-back-propagation. The additional update leads to an increasing similarity of hidden-unit activation patterns associated with the same output class. In a number of experiments, bp-som is shown (i) to improve generalisation performance over that of bp for tasks with which bp exhibits over tting; (ii) to provide indications of the number of hidden layers needed to learn a task; (iii) to increase the amount of hidden units that can be pruned without loss of generalisation, and (iv) to provide a means for automatic knowledge abstraction from trained networks. From these experiments, we conclude that the hybrid bp-som architecture and learning algorithm combines advantages of the bp and som learning algorithms.