Rules extraction from constructively trained neural networks based on genetic algorithms

The application of neural networks in the data mining has become wider. Although neural networks may have complex structure, long training time, and the representation of results is not comprehensible, neural networks have high acceptance ability for noisy data, high accuracy and are preferable in data mining. On the other hand, It is an open question as to what is the best way to train and extract symbolic rules from trained neural networks in domains like classification. In this paper, we train the neural networks by constructive learning and present the analysis of the convergence rate of the error in a neural network with and without threshold which have been learnt by a constructive method to obtain the simple structure of the network. The response of ANN is acquired but its result is not in understandable form or in a black box form. It is frequently desirable to use the model backwards and identify sets of input variable which results in a desired output value. The large numbers of variables and nonlinear nature of many materials models that can help finding an optimal set of difficult input variables. We will use a genetic algorithm to solve this problem. The method is evaluated on different public-domain data sets with the aim of testing the predictive ability of the method and compared with standard classifiers, results showed comparatively high accuracy.

[1]  Mark Craven,et al.  Extracting comprehensible models from trained neural networks , 1996 .

[2]  David Casasent,et al.  Neural closure associative processor , 1991, Neural Networks.

[3]  Pei-Chann Chang,et al.  Integration of Genetic Algorithm and Neural Network for Financial Early Warning System: An Example of Taiwanese Banking Industry , 2006, First International Conference on Innovative Computing, Information and Control - Volume I (ICICIC'06).

[4]  Khashayar Khorasani,et al.  A new strategy for adaptively constructing multilayer feedforward neural networks , 2003, Neurocomputing.

[5]  F. Smieja.  Neural network constructive algorithms: Trading generalization for learning efficiency? , 1993 .

[6]  Franz Wotawa,et al.  Deriving qualitative rules from neural networks - a case study for ozone forecasting , 2001, AI Commun..

[7]  R. Krishnan,et al.  A search technique for rule extraction from trained neural networks , 1999, Pattern Recognit. Lett..

[8]  Rudy Setiono,et al.  A Penalty-Function Approach for Pruning Feedforward Neural Networks , 1997, Neural Computation.

[9]  Rudy Setiono.  Extracting M-of-N rules from trained neural networks , 2000, IEEE Trans. Neural Networks Learn. Syst..

[10]  Ashish Darbari,et al.  Rule Extraction from Trained ANN: A Survey , 2000 .

[11]  Gordon I. McCalla,et al.  Tutoring bishop-pawn endgames: An experiment in using knowledge-based chess as a domain for intelligent tutoring , 1993, Applied Intelligence.

[12]  Hiroshi Tsukimoto,et al.  Extracting rules from trained neural networks , 2000, IEEE Trans. Neural Networks Learn. Syst..

[13]  Urszula Markowska-Kaczmar,et al.  Discovering the Mysteries of Neural Networks , 2004, Int. J. Hybrid Intell. Syst..

[14]  Jude Shavlik,et al.  THE EXTRACTION OF REFINED RULES FROM KNOWLEDGE BASED NEURAL NETWORKS , 1993 .

[15]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[16]  Edward Keedwell,et al.  CREATING RULES FROM TRAINED NEURAL NETWORKS USING GENETIC ALGORITHMS , 2000 .

[17]  Alex A. Freitas,et al.  Extracting comprehensible rules from neural networks via genetic algorithms , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[18]  Krzysztof J. Cios,et al.  Uniqueness of medical data mining , 2002, Artif. Intell. Medicine.

[19]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[20]  Lutz Prechelt,et al.  Investigation of the CasCor Family of Learning Algorithms , 1997, Neural Networks.

[21]  Jacek M. Zurada,et al.  Extraction of rules from artificial neural networks for nonlinear regression , 2002, IEEE Trans. Neural Networks.

[22]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[23]  H. Levent Akin,et al.  Rule extraction from trained neural networks using genetic algorithms , 1997 .

[24]  O. Mangasarian,et al.  Multisurface method of pattern separation for medical diagnosis applied to breast cytology. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[25]  Le Gruenwald,et al.  A survey of data mining and knowledge discovery software tools , 1999, SKDD.

[26]  Urszula Markowska-Kaczmar,et al.  Rule Extraction from Neural Network by Genetic Algorithm with Pareto Optimization , 2004, ICAISC.

[27]  Bart Baesens,et al.  Using Rule Extraction to Improve the Comprehensibility of Predictive Models , 2006 .

[28]  Rudy Setiono,et al.  Extracting Rules from Neural Networks by Pruning and Hidden-Unit Splitting , 1997, Neural Computation.

[29]  Guido Bologna,et al.  A model for single and multiple knowledge based networks , 2003, Artif. Intell. Medicine.

[30]  Shiro Usui,et al.  Mutation-based genetic neural network , 2005, IEEE Transactions on Neural Networks.

[31]  Sancho Salcedo-Sanz,et al.  A Hybrid Neural-Genetic Algorithm for the Frequency Assignment Problem in Satellite Communications , 2005, Applied Intelligence.

[32]  C. Lee Giles,et al.  Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution , 1995, IEEE Trans. Neural Networks.

[33]  Khashayar Khorasani,et al.  New training strategies for constructive neural networks with application to regression problems , 2004, Neural Networks.

[34]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[35]  Joachim Diederich,et al.  The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks , 1998, IEEE Trans. Neural Networks.

[36]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[37]  L. Darrell Whitley,et al.  International Workshop on Combinations of Genetic Algorithms and Neural Networks , 1992 .

[38]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[39]  M. Esmel ElAlami,et al.  Extracting rules from trained neural network using GA for managing E-business , 2004, Appl. Soft Comput..

[40]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[41]  Jude W. Shavlik,et al.  Extracting Refined Rules from Knowledge-Based Neural Networks , 1993, Machine Learning.

[42]  Khashayar Khorasani,et al.  Input-side training in constructive neural networks based on error scaling and pruning , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[43]  Zhi-Hua Zhou,et al.  Extracting symbolic rules from trained neural network ensembles , 2003, AI Commun..

[44]  Urszula Markowska-Kaczmar.  The influence of parameters in evolutionary based rule extraction method from neural network , 2005, 5th International Conference on Intelligent Systems Design and Applications (ISDA'05).

[45]  Mitchell A. Potter,et al.  A genetic cascade-correlation learning algorithm , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[46]  Colin R. Reeves,et al.  Genetic Algorithms: Principles and Perspectives: A Guide to Ga Theory , 2002 .

[47]  Jiawei Han,et al.  Data Mining: Concepts and Techniques , 2000 .

[48]  Huan Liu,et al.  Incremental Feature Selection , 1998, Applied Intelligence.

[49]  Helge J. Ritter,et al.  Learning and Generalization in Cascade Network Architectures , 1996, Neural Computation.

[50]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[51]  Alan Bundy.  IJCAI Policy on Multiple Publication of Papers , 1988, AI Commun..

[52]  Joydeep Ghosh,et al.  Three techniques for extracting rules from feedforward networks , 1996 .