Accelerating Learning Performance of Back Propagation Algorithm by Using Adaptive Gain Together with Adaptive Momentum and Adaptive Learning Rate on Classification Problems

The back propagation (BP) algorithm is a very popular learning approach in feedforward multilayer perceptron networks. However, the most serious problem associated with the BP is local minima problem and slow convergence speeds. Over the years, many improvements and modifications of the back propagation learning algorithm have been reported. In this research, we propose a new modified back propagation learning algorithm by introducing adaptive gain together with adaptive momentum and adaptive learning rate into weight update process. By computer simulations, we demonstrate that the proposed algorithm can give a better convergence rate and can find a good solution in early time compare to the conventional back propagation. We use two common benchmark classification problems to illustrate the improvement in convergence time.

[1]  Ramesh Sharda,et al.  Predicting box-office success of motion pictures with neural networks , 2006 .

[2]  Hiroki Tamura,et al.  Avoiding the Local Minima Problem in Backpropagation Algorithm with Modified Error Function , 2005, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[3]  Shadrokh Samavi,et al.  Steganalysis and payload estimation of embedding in pixel differences using neural networks , 2010, Pattern Recognit..

[4]  Kin Keung Lai,et al.  An Adaptive BP Algorithm with Optimal Learning Rates and Directional Error Correction for Foreign Exchange Market Trend Prediction , 2006, ISNN.

[5]  Harsha Sirisena,et al.  Performance improvement of backpropagation algorithm by automatic activation function gain tuning using fuzzy logic , 2003, Neurocomputing.

[6]  Shen Zhang,et al.  Improved BP Neural Network for Transformer Fault Diagnosis , 2007 .

[7]  Rozaida Ghazali,et al.  An Improved Back Propagation Neural Network Algorithm on Classification Problems , 2010, FGIT-DTA/BSBT.

[8]  Alan S. Morris,et al.  Soft computing methods applied to the control of a flexible robot manipulator , 2009, Appl. Soft Comput..

[9]  H. Tamura,et al.  An improved backpropagation algorithm to avoid the local minima problem , 2004, Neurocomputing.

[10]  Nikos E. Mastorakis,et al.  Multilayer perceptron and neural networks , 2009 .

[11]  Kok Wai Wong,et al.  Comparing the Performance of Different Neural Networks Architectures for the Prediction of Mineral Prospectivity , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[12]  Rozaida Ghazali,et al.  The Development of Improved Back-Propagation Neural Networks Algorithm for Predicting Patients with Heart Disease , 2010, ICICA.

[13]  John J. Paulos,et al.  The Effects of Precision Constraints in a Backpropagation Learning Network , 1990, Neural Computation.

[14]  Li Ji,et al.  Back-propagation network improved by conjugate gradient based on genetic algorithm in QSAR study on endocrine disrupting chemicals , 2008 .

[15]  D. Coomans,et al.  Comparison of Multivariate Discrimination Techniques for Clinical Data— Application to the Thyroid Functional State , 1983, Methods of Information in Medicine.

[16]  Holger R. Maier,et al.  The effect of internal parameters and geometry on the performance of back-propagation neural networks: an empirical study , 1998 .

[17]  David E. Booth,et al.  A comparison of supervised and unsupervised neural networks in predicting bankruptcy of Korean firms , 2005, Expert Syst. Appl..

[18]  Jacek M. Zurada,et al.  Advances in Neural Networks - ISNN 2006, Third International Symposium on Neural Networks, Chengdu, China, May 28 - June 1, 2006, Proceedings, Part I , 2006, International Symposium on Neural Networks.

[19]  Nazri Mohd Nawi,et al.  An Improved Conjugate Gradient Based Learning Algorithm for Back Propagation Neural Networks , 2008 .

[20]  S. Venugopal,et al.  Artificial neural network modeling to evaluate and predict the deformation behavior of stainless steel type AISI 304L during hot torsion , 2009, Appl. Soft Comput..

[21]  I Isc,et al.  Proceeding of the International Conference on Advanced Science, Engineering and Information Technology 2011 , 2011 .

[22]  Alsmadi M.Kh.S.,et al.  Back Propagation Algorithm: The Best Algorithm Among the Multi-layer Perceptron Algorithm , 2009 .

[23]  黄亚明,et al.  NOAH , 2012 .

[24]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[25]  Maode Ma,et al.  Information Computing and Applications - First International Conference, ICICA 2010, Tangshan, China, October 15-18, 2010. Proceedings , 2010, ICICA.

[26]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[27]  Rozaida Ghazali,et al.  The Effect of Adaptive Gain and Adaptive Momentum in Improving Training Time of Gradient Descent Back Propagation Algorithm on Classification Problems , 2011 .

[28]  D.S. Yeung,et al.  Pilot Study on the Localized Generalization Error Model for Single Layer Perceptron Neural Network , 2006, 2006 International Conference on Machine Learning and Cybernetics.

[29]  Emile Fiesler,et al.  The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks , 1996, Neural Computation.

[30]  Hongmei Shao,et al.  A New BP Algorithm with Adaptive Momentum for FNNs Training , 2009, 2009 WRI Global Congress on Intelligent Systems.

[31]  Mohammed A. Otair,et al.  Speeding Up Back-Propagation Neural Networks , 2005 .