Kernel orthonormalization in radial basis function neural networks

This paper deals with optimization of the computations involved in training radial basis function (RBF) neural networks. The main contribution of the reported work is the method for network weights calculation, in which the key idea is to transform the RBF kernels into an orthonormal set of functions (using the standard Gram-Schmidt orthogonalization). This significantly reduces the computing time if the RBF training scheme, which relies on adding one kernel hidden node at a time to improve network performance, is adopted. Another property of the method is that, after the RBF network weights are computed, the original network structure can be restored back. An additional strength of the method is the possibility to decompose the proposed computing task into a number of parallel subtasks so gaining further savings on computing time. Also, the proposed weight calculation technique has low storage requirements. These features make the method very attractive for hardware implementation. The paper presents a detailed derivation of the proposed network weights calculation procedure and demonstrates its validity for RBF network training on a number of data classification and function approximation problems.

[1]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[2]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[3]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[4]  M. Korenberg,et al.  Orthogonal approaches to time-series analysis and system identification , 1991, IEEE Signal Processing Magazine.

[5]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[6]  M. J. D. Powell,et al.  Radial basis functions for multivariable interpolation: a review , 1987 .

[7]  Anthony Ralston,et al.  Mathematical Methods for Digital Computers , 1960 .

[8]  Partha Pratim Kanjilal,et al.  On the application of orthogonal transformation for the design and analysis of feedforward networks , 1995, IEEE Trans. Neural Networks.

[9]  D.R. Hush,et al.  Progress in supervised neural networks , 1993, IEEE Signal Processing Magazine.

[10]  Dimitry M. Gorinevsky,et al.  On the persistency of excitation in radial basis function network identification of nonlinear systems , 1995, IEEE Trans. Neural Networks.

[11]  K. S. Banerjee.  Generalized Inverse of Matrices and Its Applications , 1973 .

[12]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[13]  John E. Moody,et al.  Fast Pruning Using Principal Components , 1993, NIPS.

[14]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[15]  Chng Eng Siong,et al.  Gradient radial basis function networks for nonlinear and nonstationary time series prediction , 1996, IEEE Trans. Neural Networks.

[16]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[17]  Hong Chen,et al.  Approximation capability to functions of several variables, nonlinear functionals, and operators by radial basis function neural networks , 1993, IEEE Trans. Neural Networks.

[18]  Paolo Frasconi,et al.  Learning without local minima in radial basis function networks , 1995, IEEE Trans. Neural Networks.

[19]  Robert M. May,et al.  Simple mathematical models with very complicated dynamics , 1976, Nature.