Greedy rule generation from discrete data and its use in neural network rule extraction

This paper proposes GRG (greedy rule generation) algorithm for generating classification rules from a data set with discrete attributes. The algorithm is "greedy" in the sense that at every iteration, it searches for the best rule to generate. The criteria for the best rule include the number of samples that it covers, the number of attributes involved in the rule, and the size of the input subspace it covers. This method is applied for extracting rules from neural networks that have been trained and pruned for solving classification problems. Neural networks with one hidden layer are trained and the proposed GRG algorithm is applied to their discretized hidden unit activation values. Our results show that rule extraction with the GRG method produces rule sets that are more accurate and concise compared to those obtained by a decision tree method and an existing neural network rule extraction method.

[1]  Hongjun Lu,et al.  NeuroRule: A Connectionist Approach to Data Mining , 1995, VLDB.

[2]  Bart Baesens,et al.  Using Neural Network Rule Extraction and Decision Tables for Credit - Risk Evaluation , 2003, Manag. Sci..

[3]  Rudy Setiono,et al.  Generating concise and accurate classification rules for breast cancer diagnosis , 2000, Artif. Intell. Medicine.

[4]  Huan Liu,et al.  Chi2: feature selection and discretization of numeric attributes , 1995, Proceedings of 7th IEEE International Conference on Tools with Artificial Intelligence.

[5]  Zhi-Hua Zhou,et al.  Extracting symbolic rules from trained neural network ensembles , 2003, AI Commun..

[6]  J.-F. Remm,et al.  Knowledge extraction using artificial neural networks: application to radar target identification , 2002, Signal Process..

[7]  Huan Liu,et al.  Neural-network feature selector , 1997, IEEE Trans. Neural Networks.

[8]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[9]  Huan Liu,et al.  NeuroLinear: From neural networks to oblique decision rules , 1997, Neurocomputing.

[10]  Minoru Fukumi,et al.  A new rule extraction method from neural networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[11]  Jacek M. Zurada,et al.  Extraction of rules from artificial neural networks for nonlinear regression , 2002, IEEE Trans. Neural Networks.

[12]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[13]  Huan Liu,et al.  Symbolic Representation of Neural Networks , 1996, Computer.

[14]  R. Rivest.  Learning Decision Lists , 1987, Machine Learning.

[15]  Masumi Ishikawa.  Rule extraction by successive regularization , 2000, Neural Networks.

[16]  C. Bradshaw,et al.  Using artificial neural networks to model the suitability of coastline for breeding by New Zealand fur seals (Arctocephalus forsteri ) , 2002 .

[17]  Antony Browne,et al.  Biological data mining with neural networks: implementation and application of a flexible decision tree extraction algorithm to genomic problem domains , 2004, Neurocomputing.

[18]  M. Esmel ElAlami,et al.  Extracting rules from trained neural network using GA for managing E-business , 2004, Appl. Soft Comput..

[19]  Nada Lavrac,et al.  The Multi-Purpose Incremental Learning System AQ15 and Its Testing Application to Three Medical Domains , 1986, AAAI.

[20]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[21]  Hao Xing,et al.  Extract intelligible and concise fuzzy rules from neural networks , 2002, Fuzzy Sets Syst..

[22]  LiMin Fu,et al.  Rule Generation from Neural Networks , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[23]  Huan Liu,et al.  X2R: a fast rule generator , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[24]  José Manuel Benítez,et al.  Interpretation of artificial neural networks by means of fuzzy rules , 2002, IEEE Trans. Neural Networks.

[25]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[26]  Joachim Diederich,et al.  The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks , 1998, IEEE Trans. Neural Networks.

[27]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[28]  Rudy Setiono,et al.  Extracting -of- Rules from Trained Neural Networks , 2000 .

[29]  Wlodzislaw Duch,et al.  A new methodology of extraction, optimization and application of crisp and fuzzy logical rules , 2001, IEEE Trans. Neural Networks.

[30]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[31]  R. Krishnan,et al.  A search technique for rule extraction from trained neural networks , 1999, Pattern Recognit. Lett..

[32]  Rudy Setiono,et al.  A Penalty-Function Approach for Pruning Feedforward Neural Networks , 1997, Neural Computation.

[33]  Guido Bologna,et al.  A model for single and multiple knowledge based networks , 2003, Artif. Intell. Medicine.