Pruned Neural Networks for Regression

Neural networks have been widely used as a tool for regression. They are capable of approximating any function and they do not require any assumption about the distribution of the data. The most commonly used architectures for regression are the feedforward neural networks with one or more hidden layers. In this paper, we present a network pruning algorithm which determines the number of units in the input and hidden layers of the networks. We compare the performance of the pruned networks to four regression methods namely, linear regression (LR), Naive Bayes (NB), k-nearest-neighbor (kNN), and a decision tree predictor M5 . On 32 pubhcly available data sets tested, the neural network method outperforms NB and kNN if the prediction errors are computed in terms of the root mean squared errors. Under this measurement metric, it also performs as well as LR and M5'. On the other hand, using the mean absolute error as the measurement metric, the neural network method outperforms all four other regression methods.

[1]  Kenneth W. Bauer,et al.  Improved feature screening in feedforward neural networks , 1996, Neurocomputing.

[2]  Jacek M. Zurada,et al.  Perturbation method for deleting redundant inputs of perceptron networks , 1997, Neurocomputing.

[3]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[4]  Timur Ash,et al.  Dynamic node creation in backpropagation networks , 1989 .

[5]  Tor Guimaraes,et al.  Integrating artificial neural networks with rule-based expert systems , 1994, Decis. Support Syst..

[6]  Ian H. Witten,et al.  Induction of model trees for predicting continuous classes , 1996 .

[7]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[8]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[9]  Huan Liu,et al.  Neural-network feature selector , 1997, IEEE Trans. Neural Networks.

[10]  Zhi-Hong Mao,et al.  Function approximation with spiked random networks , 1999, IEEE Trans. Neural Networks.

[11]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[12]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[13]  Kenneth W. Bauer,et al.  Determining input features for multilayer perceptrons , 1995, Neurocomputing.

[14]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[15]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[16]  Michael C. Mozer,et al.  Using Relevance to Reduce Network Size Automatically , 1989 .

[17]  Leonard E. Trigg,et al.  Naive Bayes for regression , 1998 .

[18]  Robert W. Blanning,et al.  An empirical measure of element contribution in neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part C.