Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks

Sigma-Pi (Σ-Π) neural networks (SPNNs) are known to provide more powerful mapping capability than traditional feed-forward neural networks. A unified convergence analysis for the batch gradient algorithm for SPNN learning is presented, covering three classes of SPNNs: Σ-Π-Σ, Σ-Σ-Π and Σ-Π-Σ-Π. The monotonicity of the error function in the iteration is also guaranteed.

[1]  Malcolm I. Heywood,et al.  A framework for improved training of Sigma-Pi networks , 1995, IEEE Trans. Neural Networks.

[2]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[3]  Burkhard Lenze,et al.  Note on interpolation on the hypercube by means of sigma-pi neural networks , 2004, Neurocomputing.

[4]  Wei Wu,et al.  Strong Convergence of Gradient Methods for BP Networks Training , 2005, 2005 International Conference on Neural Networks and Brain.

[5]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[6]  Wei Wu,et al.  Deterministic convergence of an online gradient method for BP neural networks , 2005, IEEE Transactions on Neural Networks.

[7]  H. Kushner,et al.  Analysis of adaptive step-size SA algorithms for parameter tracking , 1995, IEEE Trans. Autom. Control..

[8]  Yanchun Liang,et al.  Successive approximation training algorithm for feedforward neural networks , 2002, Neurocomputing.

[9]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[10]  Ya-Xiang Yuan,et al.  Optimization theory and methods , 2006 .

[11]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[12]  Xin Li,et al.  Training Multilayer Perceptrons Via Minimization of Sum of Ridge Functions , 2002, Adv. Comput. Math..

[13]  Burkhard Lenze,et al.  How to make sigma-pi neural networks perform perfectly on regular training sets , 1994, Neural Networks.