A sequential algorithm for feed-forward neural networks with optimal coefficients and interacting frequencies

An algorithm for sequential approximation with optimal coefficients and interacting frequencies (SAOCIF) for feed-forward neural networks is presented. SAOCIF combines two key ideas. The first one is the optimization of the coefficients (the linear part of the approximation). The second one is the strategy to choose the frequencies (the non-linear weights), taking into account the interactions with the previously selected ones. The resulting method combines the locality of sequential approximations, where only one frequency is found at every step, with the globality of non-sequential methods, where every frequency interacts with the others. The idea behind SAOCIF can be theoretically extended to general Hilbert spaces. Experimental results show a very satisfactory performance.

[1]  Tony R. Martinez,et al.  Heterogeneous radial basis function networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[2]  Visakan Kadirkamanathan,et al.  A Function Estimation Approach to Sequential Learning with Neural Networks , 1993, Neural Computation.

[3]  Julio Ortega Lopera,et al.  Improved RAN sequential prediction using orthogonal techniques , 2001, Neurocomputing.

[4]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[5]  T. Ash,et al.  Dynamic node creation in backpropagation networks , 1989, International 1989 Joint Conference on Neural Networks.

[6]  Kazuyuki Murase,et al.  A new algorithm to design compact two-hidden-layer artificial neural networks , 2001, Neural Networks.

[7]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[8]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[9]  L. Jones.  On a conjecture of Huber concerning the convergence of projection pursuit regression , 1987 .

[10]  J. M Varah,et al.  Computational methods in linear algebra , 1984 .

[11]  L. Jones.  A Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training , 1992 .

[12]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[13]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[14]  A. Barron.  Approximation and Estimation Bounds for Artificial Neural Networks , 1991, COLT '91.

[15]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[16]  Joydeep Ghosh,et al.  Ridge polynomial networks , 1995, IEEE Trans. Neural Networks.

[17]  Gunnar Rätsch,et al.  An Improvement of AdaBoost to Avoid Overfitting , 1998, ICONIP.

[18]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[19]  Y Lu,et al.  A Sequential Learning Scheme for Function Approximation Using Minimal Radial Basis Function Neural Networks , 1997, Neural Computation.

[20]  Alexander J. Smola,et al.  Sparse Greedy Gaussian Process Regression , 2000, NIPS.

[21]  Pascal Vincent,et al.  Kernel Matching Pursuit , 2002, Machine Learning.

[22]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[23]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[24]  Radakovič.  The theory of approximation , 1932 .

[25]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[26]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[27]  Helge Ritter,et al.  Cascade LLM Networks , 1992 .

[28]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[29]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[30]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[31]  Eric B. Bartlett,et al.  Dynamic node architecture learning: An information theoretic approach , 1994, Neural Networks.

[32]  Tamás D. Gedeon,et al.  Exploring constructive cascade networks , 1999, IEEE Trans. Neural Networks.

[33]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[34]  A. K. Rigler,et al.  Accelerating the convergence of the back-propagation method , 1988, Biological Cybernetics.

[35]  Luis Antonio,et al.  A Case study in neural network training with the breeder genetic algorithm , 2000 .

[36]  Lutz Prechelt,et al.  Investigation of the CasCor Family of Learning Algorithms , 1997, Neural Networks.

[37]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[38]  Sheng Chen,et al.  Regularized orthogonal least squares algorithm for constructing radial basis function networks , 1996 .

[39]  Brian D. Ripley,et al.  Statistical Ideas for Selecting Network Architectures , 1995, SNN Symposium on Neural Networks.

[40]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[41]  Martin G. Bello,et al.  Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks , 1992, IEEE Trans. Neural Networks.

[42]  Sheng Chen,et al.  Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks , 1999, IEEE Trans. Neural Networks.

[43]  J. Sopena,et al.  Neural networks with periodic and monotonic activation functions: a comparative study in classification problems , 1999 .

[44]  Bert Kappen,et al.  Neural Networks: Artificial Intelligence and Industrial Applications , 1995, Springer London.

[45]  Mikko Lehtokangas.  Modelling with constructive backpropagation , 1999, Neural Networks.

[46]  Julian Morris,et al.  A procedure for determining the topology of multilayer feedforward neural networks , 1994, Neural Networks.

[47]  J. Cooper,et al.  Theory of Approximation , 1960, Mathematical Gazette.

[48]  D. Faddeev,et al.  Computational methods of linear algebra , 1981 .

[49]  Jukka Saarinen,et al.  Evaluation of constructive neural networks with cascaded architectures , 2002, Neurocomputing.

[50]  Kevin Warwick,et al.  Incremental Approximation by Neural Networks , 1998 .

[51]  Shie Qian,et al.  Signal representation using adaptive normalized Gaussian functions , 1994, Signal Process..

[52]  Khashayar Khorasani,et al.  New training strategies for constructive neural networks with application to regression problems , 2004, Neural Networks.