Constructive feedforward neural networks for regression problems : a survey

In this paper we review the procedures for constructing feedforward neural networks in regression problems While standard back propagation performs gradient descent only in the weight space of a network with xed topology constructive procedures start with a small network and then grow additional hidden units and weights until a satisfactory solution is found The constructive procedures are categorized according to the resultant network architecture and the learning algorithm for the network weights The Hong Kong University of Science Technology Technical Report Series Department of Computer Science

[1]  R. A. Silverman,et al.  Introductory Real Analysis , 1972 .

[2]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[3]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[4]  Stanley J. Farlow,et al.  Self-Organizing Methods in Modeling: Gmdh Type Algorithms , 1984 .

[5]  L. Jones.  On a conjecture of Huber concerning the convergence of projection pursuit regression , 1987 .

[6]  Yves Chauvin,et al.  A Back-Propagation Algorithm with Optimal Use of Hidden Units , 1988, NIPS.

[7]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[8]  Lorien Y. Pratt,et al.  Comparing Biases for Minimal Network Construction with Back-Propagation , 1988, NIPS.

[9]  Yamashita,et al.  Backpropagation algorithm which varies the number of hidden units , 1989 .

[10]  T. Ash,et al.  Dynamic node creation in backpropagation networks , 1989, International 1989 Joint Conference on Neural Networks.

[11]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[12]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[13]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[14]  S. Fahlman.  Fast-learning variations on back propagation: an empirical study. , 1989 .

[15]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[16]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[17]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[18]  M. Golea,et al.  A Convergence Theorem for Sequential Learning in Two-Layer Perceptrons , 1990 .

[19]  R. Scott Crowder,et al.  Predicting the Mackey-Glass Timeseries With Cascade-Correlation Learning , 1990 .

[20]  James D. Keeler,et al.  Layered Neural Networks with Gaussian Hidden Units as Universal Approximations , 1990, Neural Computation.

[21]  Jerome H. Friedman,et al.  Adaptive Spline Networks , 1990, NIPS.

[22]  Manoel Fernando Tenorio,et al.  Self-organizing network for optimum supervised learning , 1990, IEEE Trans. Neural Networks.

[23]  Guillaume Deffuant.  Neural units recruitment algorithm for generation of decision trees , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[24]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[25]  Shigeo Abe.  Learning by parallel forward propagation , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[26]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[27]  Richard G. Priest,et al.  Pattern classification using projection pursuit , 1990, Pattern Recognit..

[28]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[29]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[30]  Norman Yarvin,et al.  Networks with Learned Unit Response Functions , 1991, NIPS.

[31]  Dennis Connolly,et al.  Self organizing modular neural networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[32]  Jude Shavlik,et al.  EXPERIMENTAL ANALYSIS OF ASPECTS OF THE CASCADE-CORRELATION LEARNING ARCHITECTURE , 1991 .

[33]  C. Jutten,et al.  Gal: Networks That Grow When They Learn and Shrink When They Forget , 1991 .

[34]  Joydeep Ghosh,et al.  The pi-sigma network: an efficient higher-order neural network for pattern classification and function approximation , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[35]  Sukhan Lee,et al.  A Gaussian potential function network with hierarchically self-organizing learning , 1991, Neural Networks.

[36]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[37]  M. Tummala,et al.  Identification of Volterra systems with a polynomial neural network , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[38]  Osamu Fujita,et al.  Optimization of the hidden unit function in feedforward neural networks , 1992, Neural Networks.

[39]  Helge Ritter,et al.  Cascade LLM Networks , 1992 .

[40]  F. Girosi.  Some extensions of radial basis functions and their applications in artificial intelligence , 1992 .

[41]  Terrence L. Fine,et al.  Forecasting Demand for Electric Power , 1992, NIPS.

[42]  Stephen J. McKenna,et al.  Cascade-correlation neural networks for the classification of cervical cells , 1992 .

[43]  S. Sjogaard.  Generalization in cascade-correlation networks , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[44]  Helge Ritter,et al.  Cascade network architectures , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[45]  James C. Bezdek,et al.  A comparison of neural network and fuzzy clustering techniques in segmenting magnetic resonance images of the brain , 1992, IEEE Trans. Neural Networks.

[46]  S. K. Rogers,et al.  A taxonomy of neural network optimality , 1992, Proceedings of the IEEE 1992 National Aerospace and Electronics Conference@m_NAECON 1992.

[47]  Vladimir Cherkassky,et al.  Neural networks and nonparametric regression , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[48]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[49]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[50]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[51]  Lei Xu,et al.  Recent Advances on Techniques of Static Feedforward Networks with Supervised Learning , 1992, Int. J. Neural Syst..

[52]  Henk Corporaal,et al.  Variations on the Cascade-Correlation Learning Architecture for Fast Convergence in Robot Control , 1992 .

[53]  Darrell Whitley,et al.  Prediction of software reliability using feedforward and recurrent neural nets , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[54]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[55]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[56]  Visakan Kadirkamanathan,et al.  A Function Estimation Approach to Sequential Learning with Neural Networks , 1993, Neural Computation.

[57]  Charles B. Roosen.  Logistic Response Projection Pursuit , 1993 .

[58]  Wj Fitzgerald,et al.  Optimization schemes for neural networks , 1993 .

[59]  Kurt Hornik,et al.  Some new results on neural network approximation , 1993, Neural Networks.

[60]  Pierre Courrieu.  A convergent generator of neural networks , 1993, Neural Networks.

[61]  Mahmood R. Azimi-Sadjadi,et al.  Recursive dynamic node creation in multilayer neural networks , 1993, IEEE Trans. Neural Networks.

[62]  Gustavo Deco,et al.  Coarse Coding Resource-Allocating Network , 1993, Neural Computation.

[63]  Dit-Yan Yeung,et al.  Constructive neural networks as estimators of bayesian discriminant functions , 1993, Pattern Recognit..

[64]  X. Yao.  A Review of Evolutionary Artiicial Neural Networks 1 2 , 1993 .

[65]  Avijit Saha,et al.  Approximation, Dimension Reduction, and Nonconvex Optimization Using Linear Superpositions of Gaussians , 1993, IEEE Trans. Computers.

[66]  Brian D. Ripley,et al.  Statistical aspects of neural networks , 1993 .

[67]  John Moody,et al.  Prediction Risk and Architecture Selection for Neural Networks , 1994 .

[68]  E. Fiesler,et al.  Comparative Bibliography of Ontogenic Neural Networks , 1994 .

[69]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[70]  K. Khorasani,et al.  Structure adaptation in feed-forward neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[71]  Stephen J. Roberts,et al.  A Probabilistic Resource Allocating Network for Novelty Detection , 1994, Neural Computation.

[72]  Chilukuri K. Mohan,et al.  An incremental network construction algorithm for approximating discontinuous functions , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[73]  Dhananjay S. Phatak,et al.  Connectivity and performance tradeoffs in the cascade correlation learning architecture , 1994, IEEE Trans. Neural Networks.

[74]  Hennie Daniels,et al.  Connectionist projection pursuit regression , 1994 .

[75]  D. M. Titterington,et al.  Neural Networks: A Review from a Statistical Perspective , 1994 .

[76]  Jerome H. Friedman,et al.  An Overview of Predictive Learning and Function Approximation , 1994 .

[77]  Byoung-Tak Zhang,et al.  An incremental learning algorithm that optimizes network size and sample size in one trial , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[78]  Warren S. Sarle,et al.  Neural Networks and Statistical Models , 1994 .

[79]  Jenq-Neng Hwang,et al.  Extensions to projection pursuit learning networks with parametric smoothers , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[80]  Bernd Fritzke,et al.  Growing cell structures--A self-organizing network for unsupervised and supervised learning , 1994, Neural Networks.

[81]  Benjamin W. Wah,et al.  An automated design system for finding the minimal configuration of a feed-forward neural network , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[82]  R. C. Lacher,et al.  Network complexity and learning efficiency of constructive learning algorithms , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[83]  Albert Y. Zomaya,et al.  Toward generating neural network structures for function approximation , 1994, Neural Networks.

[84]  Trevor Hastie,et al.  Automatic Smoothing Spline Projection Pursuit , 1994 .

[85]  Joydeep Ghosh,et al.  Ridge polynomial networks , 1995, IEEE Trans. Neural Networks.

[86]  S. Klinke,et al.  Exploratory Projection Pursuit , 1995 .

[87]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[88]  Sandeep Gulati,et al.  Parallelizing the cascade-correlation algorithm using time warp , 1995, Neural Networks.

[89]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.