Adaptive modified backpropagation algorithm based on differential errors

A new efficient modified back propagation algorithm with adaptive learning rate is proposed to increase the convergence speed and to minimize the error. The method eliminates initial fixing of learning rate through trial and error and replaces by adaptive learning rate. In each iteration, adaptive learning rate for output and hidden layer are determined by calculating differential linear and nonlinear errors of output layer and hidden layer separately. In this method, each layer has different learning rate in each iteration. The performance of the proposed algorithm is verified by the simulation results.

[1]  Yahya Zweiri.  Optimization of a Three-Term Backpropagation Algorithm Used for Neural Network Learning , 2007 .

[2]  T. Kathirvalavakumar,et al.  Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks , 2009, Neurocomputing.

[3]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[4]  C. Zhang,et al.  Variable Learning Rate LMS Based Linear Adaptive Inverse Control , 2006 .

[5]  Z. Zainuddin,et al.  Improving the Convergence of the Backpropagation Algorithm Using Local Adaptive Techniques , 2007, International Conference on Computational Intelligence.

[6]  Mohamed Najim,et al.  A fast feedforward training algorithm using a modified form of the standard backpropagation algorithm , 2001, IEEE Trans. Neural Networks.

[7]  Zhihong Man,et al.  A New Adaptive Backpropagation Algorithm Based on Lyapunov Stability Theory for Neural Networks , 2006, IEEE Transactions on Neural Networks.

[8]  Bin-Da Liu,et al.  A backpropagation algorithm with adaptive learning rate and momentum coefficient , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[9]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[10]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[11]  Vladimir B. Bajic,et al.  Adaptive On-line ANN Learning Algorithm and Application to Identification of Non-linear Systems , 1999, Informatica.

[12]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[13]  Samy Bengio,et al.  Use of genetic programming for the search of a new learning rule for neural networks , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[14]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[15]  Saeid Iranmanesh,et al.  A differential adaptive learning rate method for back-propagation neural networks , 2009 .

[16]  Guo-An Chen,et al.  Acceleration of backpropagation learning using optimised learning rate and momentum , 1993 .

[17]  Dilip Sarkar,et al.  Methods to speed up error back-propagation learning algorithm , 1995, CSUR.

[18]  Soo-Young Lee,et al.  Adaptive learning algorithms to incorporate additional functional constraints into neural networks , 1998, Neurocomputing.