A fast convergence algorithm for BPNN based on optimal control theory based learning rate

In this paper, a novel updating law for Backpropagation learning algorithm based on optimal control theory is proposed. The original Backpropagation algorithm composed of learning rate factor (LR). The coefficient in LR is called step size and indicates the rate of algorithm convergence which is selected by trial and error. In original BP the step size is constant. This fixed step size causes important incapabilities such as slow convergence and local minima problem. In Optimal Control Theory Based Learning Rate (OCLR)algorithm the step size is selected adaptively according to optimal control theory that makes Backpropagation learning algorithm convergence much faster than the original BP. To achieve the fastest possible answer, the Backpropagation learning algorithm is modeled as a minimum time control problem and the step size coefficient is considered as input. This consideration results a Bang-Bang control characteristics. The effectiveness of the proposed algorithm is evaluated via two examples. These examples are XOR, 3-bit parity. In all the problems, the proposed algorithm outperforms well in speed and the ability to escape from local minima.

[1]  D. E. Rumelhart,et al.  chapter Parallel Distributed Processing, Exploration in the Microstructure of Cognition , 1986 .

[2]  Eugenius Kaszkurewicz,et al.  A Control-Theoretic Approach to the Design of Zero Finding Numerical Methods , 2007, IEEE Transactions on Automatic Control.

[3]  Alexander S. Poznyak,et al.  Differential Neural Networks for Robust Nonlinear Control: Identification, State Estimation and Trajectory Tracking , 2001 .

[4]  H. Sira-Ramírez,et al.  A Sliding Mode Strategy for Adaptive Learning in Adalines , 1996 .

[5]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[6]  H. Handa,et al.  Utilization of hierarchical structure stochastic automata for the back propagation method with momentum , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[7]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[8]  Zhihong Man,et al.  A New Adaptive Backpropagation Algorithm Based on Lyapunov Stability Theory for Neural Networks , 2006, IEEE Transactions on Neural Networks.

[9]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[10]  Victor M. Becerra,et al.  Optimal control , 2008, Scholarpedia.

[11]  Yahya H. Zweiri,et al.  A three-term backpropagation algorithm , 2003, Neurocomputing.

[12]  Emanuel Todorov,et al.  Optimal Control Theory , 2006 .

[13]  Kaspar Althoefer,et al.  Stability analysis of a three-term backpropagation algorithm , 2005, Neural Networks.