An Approach To Generate Rules From Neural Networks for Regression Problems

Abstract Artificial neural networks have been successfully applied to a variety of business application problems involving classification and regression. They are especially useful for regression problems as they do not require prior knowledge about the data distribution. In many applications, it is desirable to extract knowledge from trained neural networks so that the users can gain a better understanding of the solution. Existing research works have focused primarily on extracting symbolic rules for classification problems with few methods devised for regression problems. In order to fill this gap, we propose an approach to extract rules from neural networks that have been trained to solve regression problems. The extracted rules divide the data samples into groups. For all samples within a group, a linear function of the relevant input attributes of the data approximates the network output. The approach is illustrated with two examples on various application problems. Experimental results show that the proposed approach generates rules that are more accurate than the existing methods based on decision trees and linear regression.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  D. Kilpatrick,et al.  Numeric Prediction Using Instance-Based Learning with Encoding Length Selection , 1997, ICONIP.

[3]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[4]  Rudy Setiono,et al.  Generating Concise Sets of Linear Regression Rules from Artificial Neural Networks , 2002, Int. J. Artif. Intell. Tools.

[5]  Carol E. Brown,et al.  Artificial neural networks applied to ratio analysis in the analytical review process , 1993 .

[6]  Joachim Diederich,et al.  The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks , 1998, IEEE Trans. Neural Networks.

[7]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[8]  Melody Y. Kiang,et al.  Managerial Applications of Neural Networks: The Case of Bank Failure Predictions , 1992 .

[9]  Luís Torgo,et al.  Search-Based Class Discretization , 1997, ECML.

[10]  Phillip Ein-Dor,et al.  Attributes of the performance of central processing units: a relative performance prediction model , 1987, CACM.

[11]  Luís Torgo,et al.  Functional Models for Regression Tree Leaves , 1997, ICML.

[12]  Dayanand N. Naik,et al.  Applied Multivariate Statistics with SAS Software , 1997 .

[13]  E. Mine Cinar,et al.  Neural Networks: A New Tool for Predicting Thrift Failures , 1992 .

[14]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[15]  Wee Kheng Leow,et al.  Pruned Neural Networks for Regression , 2000, PRICAI.

[16]  Jacek M. Zurada,et al.  Extraction of rules from artificial neural networks for nonlinear regression , 2002, IEEE Trans. Neural Networks.

[17]  Jacques de Villiers,et al.  Backpropagation neural nets with one and two hidden layers , 1993, IEEE Trans. Neural Networks.

[18]  Ramesh Sharda,et al.  Bankruptcy prediction using neural networks , 1994, Decis. Support Syst..

[19]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[20]  Soumitra Dutta,et al.  Decision support in non-conservative domains: Generalization with neural networks , 1992, Decis. Support Syst..

[21]  James L. McClelland,et al.  James L. McClelland, David Rumelhart and the PDP Research Group, Parallel distributed processing: explorations in the microstructure of cognition . Vol. 1. Foundations . Vol. 2. Psychological and biological models . Cambridge MA: M.I.T. Press, 1987. , 1989, Journal of Child Language.

[22]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[23]  Rudy Setiono,et al.  A Penalty-Function Approach for Pruning Feedforward Neural Networks , 1997, Neural Computation.

[24]  Choon Hong. Lee,et al.  A multi-layer perceptron model of credit scoring. , 1998 .

[25]  Efraim Turban,et al.  Neural Networks in Finance and Investing: Using Artificial Intelligence to Improve Real-World Performance , 1992 .

[26]  Vijay S. Desai,et al.  The Efficacy of Neural Networks in Predicting Returns on Stock and Bond Indices , 1998 .

[27]  Gerhard Widmer,et al.  Relative Unsupervised Discretization for Regresseion Problems , 2000, ECML.

[28]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[29]  Ron Kohavi,et al.  Irrelevant Features and the Subset Selection Problem , 1994, ICML.