Optimal Parameter Selection Using Three-term Back Propagation Algorithm for Data Classification

The back propagation (BP) algorithm is the most popular supervised learning method for multi-layered feed forward Neural Network. It has been successfully deployed in numerous practical problems and disciplines. Regardless of its popularity, BP is still known for some major drawbacks such as easily getting stuck in local minima and slow convergence; since, it uses Gradient Descent (GD) method to learn the network. Over the years, many improved modifications of the BP learning algorithm have been made by researchers but the local minima problem remains unresolved. Therefore, to resolve the inherent problems of BP algorithm, this paper proposed BPGD-A3T algorithm where the approach introduces three adaptive parameters which are gain, momentum and learning rate in BP. The performance of the proposed BPGD-A3T algorithm is then compared with BPGD two term parameters (BPGD-2T), BP with adaptive gain (BPGD-AG) and conventional BP algorithm (BPGD) by means of simulations on classification datasets. The simulation results show that the proposed BPGD-A3T shows better performance and performed highest accuracy for all dataset as compared to other.

[1]  Sagar V. Kamarthi,et al.  Accelerating neural network training using weight extrapolations , 1999, Neural Networks.

[2]  Shahnorbanun Sahran,et al.  A new initialization technique in polar coordinates for Particle Swarm Optimization and Polar PSO , 2017 .

[3]  Rozaida Ghazali,et al.  Learning Efficiency Improvement of Back Propagation Algorithm by Adaptively Changing Gain Parameter together with Momentum and Learning Rate , 2011, ICSECS.

[4]  D. E. Rumelhart,et al.  Learning internal representations by back-propagating errors , 1986 .

[5]  Mohd Saberi Mohamad,et al.  Specific Tuning Parameter for Directed Random Walk Algorithm Cancer Classification , 2017 .

[6]  Bin-Da Liu,et al.  A backpropagation algorithm with adaptive learning rate and momentum coefficient , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[7]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[8]  Mark A. Kramer,et al.  Improvement of the backpropagation algorithm for training neural networks , 1990 .

[9]  Dimitris A. Karras,et al.  An efficient constrained learning algorithm with momentum acceleration , 1995, Neural Networks.

[10]  Carl G. Looney.  Stabilization and speedup of convergence in training feedforward neural networks , 1996, Neurocomputing.

[11]  Rozaida Ghazali,et al.  THE EFFECT OF GAIN VARIATION IN IMPROVING LEARNING SPEED OF BACK PROPAGATION NEURAL NETWORK ALGORITHM ON CLASSIFICATION PROBLEMS , 2009 .

[12]  Rozaida Ghazali,et al.  The Effect of Adaptive Gain and Adaptive Momentum in Improving Training Time of Gradient Descent Back Propagation Algorithm on Classification Problems , 2011 .

[13]  Saeid Iranmanesh,et al.  A differential adaptive learning rate method for back-propagation neural networks , 2009 .

[14]  D. G. Sotiropoulos,et al.  A SPECTRAL VERSION OF PERRY'S CONJUGATE GRADIENT METHOD FOR NEURAL NETWORK TRAINING , 2002 .

[15]  Richard S. Johannes,et al.  Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus , 1988 .

[16]  Maslina Darus,et al.  An improved error signal for the backpropagation model for classification problems , 2001, Int. J. Comput. Math..

[17]  J. Ross Quinlan,et al.  Simplifying decision trees , 1987, Int. J. Hum. Comput. Stud..

[18]  Nazri Mohd Nawi,et al.  An Improved Conjugate Gradient Based Learning Algorithm for Back Propagation Neural Networks , 2008 .