Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks

This letter presents a unified convergence analysis of the split-complex nonlinear gradient descent (SCNGD) learning algorithms for complex-valued recurrent neural networks, covering three classes of SCNGD algorithms: standard SCNGD, normalized SCNGD, and adaptive normalized SCNGD. We prove that if the activation functions are of split-complex type and some conditions are satisfied, the error function is monotonically decreasing during the training iteration process, and the gradients of the error function with respect to the real and imaginary parts of the weights converge to zero. A strong convergence result is also obtained under the assumption that the error function has only a finite number of stationary points. The simulation results are given to support the theoretical analysis.

[1]  Danilo P. Mandic,et al.  A generalized normalized gradient descent algorithm , 2004, IEEE Signal Processing Letters.

[2]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[3]  Danilo P. Mandic,et al.  A fully adaptive normalized nonlinear gradient descent algorithm for complex-valued nonlinear adaptive filters , 2003, IEEE Trans. Signal Process..

[4]  Alois Knoll,et al.  Complex Valued Recurrent Neural Network: From Architecture to Training , 2012 .

[5]  C. Zhang,et al.  Convergence of BP algorithm for product unit neural networks with exponential weights , 2008, Neurocomputing.

[6]  Wei Wu,et al.  Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks , 2009, IEEE Transactions on Neural Networks.

[7]  Danilo P. Mandic,et al.  A Complex-Valued RTRL Algorithm for Recurrent Neural Networks , 2004, Neural Computation.

[8]  Pedro Henrique Gouvêa Coelho.  A complex EKF-RTRL neural network , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[9]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction , 2001 .

[10]  Akira Hirose.  Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.

[11]  Wei Wu,et al.  Deterministic convergence of an online gradient method for BP neural networks , 2005, IEEE Transactions on Neural Networks.

[12]  B. Widrow,et al.  The complex LMS algorithm , 1975, Proceedings of the IEEE.

[13]  M. Razaz,et al.  A normalized gradient descent algorithm for nonlinear adaptive filters using a gradient adaptive step size , 2001, IEEE Signal Processing Letters.

[14]  Danilo P. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters , 2009 .

[15]  D. Mandic,et al.  Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models , 2009 .

[16]  Wei Wu,et al.  Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update , 2007, Neural Computation.

[17]  Dongpo Xu,et al.  Convergence of gradient method for Elman networks , 2008 .

[18]  Zhiping Wang,et al.  Convergence of an Online Split-Complex Gradient Algorithm for Complex-Valued Neural Networks , 2010 .

[19]  Danilo P. Mandic,et al.  Stochastic Gradient-Adaptive Complex-Valued Nonlinear Neural Adaptive Filters With a Gradient-Adaptive Step Size , 2007, IEEE Transactions on Neural Networks.

[20]  Jacek M. Zurada,et al.  Deterministic convergence of conjugate gradient method for feedforward neural networks , 2011, Neurocomputing.

[21]  Gaofeng Zheng,et al.  Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks , 2011, Neural Processing Letters.

[22]  Kazuyuki Murase,et al.  Single-layered complex-valued neural network for real-valued classification problems , 2009, Neurocomputing.

[23]  Wei Wu,et al.  Convergence of gradient method for a fully recurrent neural network , 2010, Soft Comput..

[24]  Henry Leung,et al.  The complex backpropagation algorithm , 1991, IEEE Trans. Signal Process..

[25]  Akira Hirose,et al.  Complex-Valued Neural Networks (Studies in Computational Intelligence) , 2006 .

[26]  Elias S. Manolakos,et al.  Training fully recurrent neural networks with complex weights , 1994 .

[27]  Chao Zhang,et al.  Convergence of Batch Split-Complex Backpropagation Algorithm for Complex-Valued Neural Networks , 2009 .

[28]  Wu,et al.  CONVERGENCE OF GRADIENT METHOD WITH MOMENTUM FOR BACK-PROPAGATION NEURAL NETWORKS * , 2008 .

[29]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[30]  Akira Hirose,et al.  Complex-Valued Neural Networks , 2006, Studies in Computational Intelligence.

[31]  Marco Gori,et al.  Optimal convergence of on-line backpropagation , 1996, IEEE Trans. Neural Networks.

[32]  Wei Wu,et al.  Strong Convergence of Gradient Methods for BP Networks Training , 2005, 2005 International Conference on Neural Networks and Brain.

[33]  Sammy Siu,et al.  Analysis of the Initial Values in Split-Complex Backpropagation Algorithm , 2008, IEEE Transactions on Neural Networks.