An equalized error backpropagation algorithm for the on-line training of multilayer perceptrons

The error backpropagation (EBP) training of a multilayer perceptron (MLP) may require a very large number of training epochs. Although the training time can usually be reduced considerably by adopting an on-line training paradigm, it can still be excessive when large networks have to be trained on lots of data. In this paper, a new on-line training algorithm is presented. It is called equalized EBP (EEBP), and it offers improved accuracy, speed, and robustness against badly scaled inputs. A major characteristic of EEBP is its utilization of weight specific learning rates whose relative magnitudes are derived from a priori computable properties of the network and the training data.

[1]  Timothy Masters,et al.  Practical neural network recipes in C , 1993 .

[2]  Amro El-Jaroudi,et al.  A new error criterion for posterior probability estimation with neural nets , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[3]  Jean-Pierre Martens.  A stochastically motivated random initialization of pattern classifying MLPs , 2004, Neural Processing Letters.

[4]  Manfred Morari,et al.  Local Training for Radial Basis Function Networks: Towards Solving the Hidden Unit Problem , 1991, 1991 American Control Conference.

[5]  Jie Zhang,et al.  A Sequential Learning Approach for Single Hidden Layer Neural Networks , 1998, Neural Networks.

[6]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .

[7]  Elijah Polak,et al.  Computational methods in optimization , 1971 .

[8]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[9]  Victor W. Zue,et al.  Phonetic classification using multi-layer perceptrons , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[10]  Esther Levin,et al.  Accelerated Learning in Layered Neural Networks , 1988, Complex Syst..

[11]  H. Bourlard,et al.  Links Between Markov Models and Multilayer Perceptrons , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[13]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[14]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[15]  James L. McClelland,et al.  James L. McClelland, David Rumelhart and the PDP Research Group, Parallel distributed processing: explorations in the microstructure of cognition . Vol. 1. Foundations . Vol. 2. Psychological and biological models . Cambridge MA: M.I.T. Press, 1987. , 1989, Journal of Child Language.

[16]  Frank Fallside,et al.  An adaptive training algorithm for back propagation networks , 1987 .

[17]  Jean-Pierre Martens,et al.  A new dynamic programming/multi-layer perceptron hybrid for continuous speech recognition , 1993, EUROSPEECH.

[18]  P. Vandamme,et al.  Identification of EF group 22 campylobacters from gastroenteritis cases as Campylobacter concisus , 1989, Journal of clinical microbiology.

[19]  Jean-Pierre Martens,et al.  On the initialization and optimization of multilayer perceptrons , 1994, IEEE Trans. Neural Networks.

[20]  Richard M. Golden,et al.  Mathematical Methods for Neural Network Analysis and Design , 1996 .

[21]  Jean-Pierre Martens,et al.  A fast and robust learning algorithm for feedforward neural networks , 1991, Neural Networks.

[22]  Geoffrey E. Hinton,et al.  Learning sets of filters using back-propagation , 1987 .

[23]  G. R. Walsh,et al.  Methods Of Optimization , 1976 .