Fast training of multilayer perceptrons

Training a multilayer perceptron by an error backpropagation algorithm is slow and uncertain. This paper describes a new approach which is much faster and certain than error backpropagation. The proposed approach is based on combined iterative and direct solution methods. In this approach, we use an inverse transformation for linearization of nonlinear output activation functions, direct solution matrix methods for training the weights of the output layer; and gradient descent, the delta rule, and other proposed techniques for training the weights of the hidden layers. The approach has been implemented and tested on many problems. Experimental results, including training times and recognition accuracy, are given. Generally, the approach achieves accuracy as good as or better than perceptrons trained using error backpropagation, and the training process is much faster than the error backpropagation algorithm and also avoids local minima and paralysis.

[1]  Anthony J. Robinson,et al.  Static and Dynamic Error Propagation Networks with Application to Speech Coding , 1987, NIPS.

[2]  Liping Yang,et al.  Backpropagation with Homotopy , 1993, Neural Computation.

[3]  L. Trefethen,et al.  Numerical linear algebra , 1997 .

[4]  J. J. Mulawka,et al.  A modified backpropagation algorithm , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[5]  A. Booth.  Numerical Methods , 1957, Nature.

[6]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[7]  Philip D. Wasserman,et al.  Neural computing - theory and practice , 1989 .

[8]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[9]  Brijesh Verma,et al.  Training of the Multilayer Perceptrons using Direct Solution Methods , 1994, Applied Informatics.

[10]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[11]  A. Neubauer,et al.  Robust learning algorithms for multi-layer perceptrons with discretized synaptic weights , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[12]  Lambert Spaanenburg,et al.  On the representation of data for optimal learning , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[13]  Frank Bärmann,et al.  A learning algorithm for multilayered neural networks based on linear least squares problems , 1993, Neural Networks.

[14]  Terrence J. Sejnowski,et al.  Analysis of hidden units in a layered network trained to classify sonar targets , 1988, Neural Networks.

[15]  Minoru Fukumi,et al.  A new back-propagation algorithm with coupled neuron , 1991, International 1989 Joint Conference on Neural Networks.

[16]  Tommy W. S. Chow,et al.  Extended backpropagation algorithm , 1993 .