An Adaptive Momentum Back Propagation (AMBP)

An algorithm for fast minimum search is proposed, which achieves very satisfying performance harmonising the Vogl's and the Conjugate Gradient algorithms. Such effectiveness is achieved by making adaptive, in a very simple and satisfactory way, both the learning rate and the momentum term, and by executing controls and corrections both on the possible cost function increase and on moves opposite to the direction of the negative of the gradient. Thanks to these improvements, we can obtain a good scaling relationship in learning. As regards the real world context, a musical application showed favourable results: besides the good convergence speed, a high generalisation capability has been achieved, as confirmed both by subjective musical evaluations and by objective tests.

[1]  Alberto L. Sangiovanni-Vincentelli,et al.  Efficient Parallel Learning Algorithms for Neural Networks , 1988, NIPS.

[2]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[3]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[4]  Robert Hecht-Nielsen.  III.3 – Theory of the Backpropagation Neural Network* , 1992 .

[5]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[6]  Sandro Ridella,et al.  Statistically controlled activation weight initialization (SCAWI) , 1992, IEEE Trans. Neural Networks.

[7]  Roberto Battiti,et al.  Accelerated Backpropagation Learning: Two Optimization Methods , 1989, Complex Syst..

[8]  Martin Fodslette Meiller.  A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning , 1993 .

[9]  Bernd Fritzke,et al.  Growing cell structures--A self-organizing network for unsupervised and supervised learning , 1994, Neural Networks.

[10]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[11]  Gerald Tesauro,et al.  Scaling Relationships in Back-propagation Learning , 1988, Complex Syst..

[12]  Robert Hecht-Nielsen,et al.  Theory of the backpropagation neural network , 1989, International 1989 Joint Conference on Neural Networks.

[13]  E. Mingolla,et al.  Neurocomputing: Foundations of Research: Edited by James A. Anderson and Edward Rosenfeld, Cambridge, MA: MIT Press, 1988, $55.00, 729 pp. ISBN 0-262-01097-6 , 1989 .

[14]  E. Polak.  Introduction to linear and nonlinear programming , 1973 .

[15]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[16]  Farid U. Dowla,et al.  Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient Method , 1991, Int. J. Neural Syst..

[17]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[18]  L. Marconi,et al.  An application of backpropagation to medical diagnosis , 1989, International 1989 Joint Conference on Neural Networks.

[19]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[20]  A. K. Rigler,et al.  Accelerating the convergence of the back-propagation method , 1988, Biological Cybernetics.