Generating rules with predicates, terms and variables from the pruned neural networks

Artificial neural networks (ANN) have demonstrated good predictive performance in a wide range of applications. They are, however, not considered sufficient for knowledge representation because of their inability to represent the reasoning process succinctly. This paper proposes a novel methodology Gyan that represents the knowledge of a trained network in the form of restricted first-order predicate rules. The empirical results demonstrate that an equivalent symbolic interpretation in the form of rules with predicates, terms and variables can be derived describing the overall behaviour of the trained ANN with improved comprehensibility while maintaining the accuracy and fidelity of the propositional rules.

[1]  Thomas L. Ainscough,et al.  Neural networks in business. , 1997 .

[2]  U. M. Feyyad.  Data mining and knowledge discovery: making sense out of data , 1996 .

[3]  Hector J. Levesque,et al.  Proceedings of the first international conference on Principles of knowledge representation and reasoning , 1989 .

[4]  Shlomo Geva,et al.  Rules and Local Function Networks , 2007 .

[5]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[6]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[7]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[8]  J. Ross Quinlan,et al.  Learning logical definitions from relations , 1990, Machine Learning.

[9]  Saso Dzeroski,et al.  Learning Nonrecursive Definitions of Relations with LINUS , 1991, EWSL.

[10]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[11]  Donald C. Wunsch,et al.  Neural network explanation using inversion , 2007, Neural Networks.

[12]  F. R. A. Hopgood,et al.  Machine Intelligence 5 , 1971, The Mathematical Gazette.

[13]  Guido Bologna,et al.  Is it worth generating rules from neural network ensembles? , 2004, J. Appl. Log..

[14]  Yves Kodratoff,et al.  Machine Learning — EWSL-91 , 1991, Lecture Notes in Computer Science.

[15]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[16]  Pavel Berkhin,et al.  A Survey of Clustering Data Mining Techniques , 2006, Grouping Multidimensional Data.

[17]  Kazumi Saito,et al.  Extracting regression rules from neural networks , 2002, Neural Networks.

[18]  G. Brewka.  Principles of Knowledge Representation , 1996 .

[19]  Joachim Diederich,et al.  The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks , 1998, IEEE Trans. Neural Networks.

[20]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[21]  F. R. A. Hopgood,et al.  Machine Intelligence 6 , 1972, The Mathematical Gazette.

[22]  Gordon Plotkin,et al.  A Further Note on Inductive Generalization , 2008 .

[23]  Rudy Setiono,et al.  Extracting Rules from Neural Networks by Pruning and Hidden-Unit Splitting , 1997, Neural Computation.

[24]  Rudy Setiono,et al.  Extracting rules from pruned networks for breast cancer diagnosis , 1996, Artif. Intell. Medicine.

[25]  José Manuel Benítez,et al.  Interpretation of artificial neural networks by means of fuzzy rules , 2002, IEEE Trans. Neural Networks.

[26]  Stephen I. Gallant,et al.  Perceptron-based learning algorithms , 1990, IEEE Trans. Neural Networks.

[27]  Richi Nayak.  Generating Predicate Rules from Neural Networks , 2005, IDEAL.

[28]  Joydeep Ghosh,et al.  Symbolic Interpretation of Artificial Neural Networks , 1999, IEEE Trans. Knowl. Data Eng..

[29]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[30]  Rudy Setiono,et al.  A Penalty-Function Approach for Pruning Feedforward Neural Networks , 1997, Neural Computation.

[31]  Wray L. Buntine.  Generalized Subsumption and Its Applications to Induction and Redundancy , 1986, Artif. Intell..

[32]  Joachim Diederich,et al.  Extracting rules for grammar recognition from Cascade-2 networks , 1995, Learning for Natural Language Processing.

[33]  Jude W. Shavlik,et al.  Extracting refined rules from knowledge-based neural networks , 2004, Machine Learning.

[34]  Ryszard S. Michalski,et al.  Knowledge acquisition by encoding expert rules versus computer induction from examples: a case study involving soybean pathology , 1999, Int. J. Hum. Comput. Stud..

[35]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[36]  Nelson F. F. Ebecken,et al.  Extracting rules from multilayer perceptrons in classification problems: A clustering-based approach , 2006, Neurocomputing.

[37]  Ryszard S. Michalski,et al.  Knowledge acquisition by encoding expert rules versus computer induction from examples: a case study , 1980 .

[38]  Alex S. Taylor,et al.  Machine intelligence , 2009, CHI.

[39]  Francesco Bergadano,et al.  Inductive Logic Programming: From Machine Learning to Software Engineering , 1995 .

[40]  Richi Nayak,et al.  GYAN: A methodology for rule extraction from artificial neural networks , 1999 .

[41]  Ellen Riloff,et al.  Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing , 1996, Lecture Notes in Computer Science.