Iterative fast orthogonal search algorithm for MDL-based training of generalized single-layer networks

The generalized single-layer network (GSLN) architecture, which implements a sum of arbitrary basis functions defined on its inputs, is potentially a flexible and efficient structure for approximating arbitrary nonlinear functions. A drawback of GSLNs is that a large number of weights and basis functions may be required to provide satisfactory approximations. In this paper, we present a new approach in which an algorithm known as iterative fast orthogonal search (IFOS) is coupled with the minimum description length (MDL) criterion to provide automatic structure selection and parameter estimation for GSLNs. The resulting algorithm, dubbed IFOS-MDL, performs both network growth and pruning to construct sparse GSLNs from potentially large spaces of candidate basis functions.

[1]  Somnath Mukhopadhyay,et al.  Iterative generation of higher-order nets in polynomial time using linear programming , 1997, IEEE Trans. Neural Networks.

[2]  A. Desrochers,et al.  On determining the structure of a non-linear system , 1984 .

[3]  Peter J. W. Rayner,et al.  Generalization and PAC learning: some new results for the class of generalized single-layer networks , 1995, IEEE Trans. Neural Networks.

[4]  M. O. Sunay,et al.  An orthogonal approach to the spatial-domain design of 2-D recursive and nonrecursive nonlinear filters , 1994 .

[5]  Mohamed I. Elmasry,et al.  Minimum description length pruning and maximum mutual information training of adaptive probabilistic neural networks , 1993, IEEE International Conference on Neural Networks.

[6]  Michael J. Korenberg,et al.  Iterative fast orthogonal search algorithm for sparse self-structuring generalized single-layer networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[7]  Robin D. Morris,et al.  Fast probabilistic self-structuring of generalized single-layer networks , 1996, IEEE Trans. Neural Networks.

[8]  M. Korenberg,et al.  Orthogonal approaches to time-series analysis and system identification , 1991, IEEE Signal Processing Magazine.

[9]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[10]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[11]  M. Korenberg,et al.  Fast orthogonal search for array processing and spectrum estimation , 1994 .

[12]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[13]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[14]  Horst Bischof,et al.  An efficient MDL-based construction of RBF networks , 1998, Neural Networks.

[15]  S. Chen,et al.  Fast orthogonal least squares algorithm for efficient subset model selection , 1995, IEEE Trans. Signal Process..

[16]  Mauricio Barahona,et al.  Detection of nonlinear dynamics in short, noisy time series , 1996, Nature.

[17]  Rosalind W. Picard,et al.  On the efficiency of the orthogonal least squares training method for radial basis function networks , 1996, IEEE Trans. Neural Networks.

[18]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[19]  Jirí Benes,et al.  On neural networks , 1990, Kybernetika.

[20]  M. Korenberg.  Statistical Identification of Parallel Cascades of Linear and Nonlinear Systems , 1982 .

[21]  Michael Shelley,et al.  Self-focussed optical structures in a nematic liquid crystal , 1996 .

[22]  E. C. Akeson,et al.  Mouse chromosome classification by radial basis function network with fast orthogonal search , 1998, Neural Networks.

[23]  Qi Li,et al.  Principal feature classification , 1997, IEEE Trans. Neural Networks.

[24]  Michael J. Korenberg,et al.  Fast orthogonal search for direction finding , 1992 .

[25]  Avinash C. Kak,et al.  Array signal processing , 1985 .

[26]  Claus-Peter Richter,et al.  Modeling stochastic spike train responses of neurons: an extended Wiener series analysis of pigeon auditory nerve fibers , 1997, Biological Cybernetics.

[27]  R J Cohen,et al.  Detection of chaotic determinism in time series from randomly forced maps. , 1997, Physica D. Nonlinear phenomena.

[28]  Alan A. Desrochers.  On an improved model reduction technique for nonlinear systems , 1981, Autom..

[29]  Ilan Ziskind,et al.  Maximum likelihood localization of multiple sources by alternating projection , 1988, IEEE Trans. Acoust. Speech Signal Process..