A soft computing-based approach for integrated training and rule extraction from artificial neural networks: DIFACONN-miner

Artificial neural network (ANN) is one of the most widely used techniques in classification data mining. Although ANNs can achieve very high classification accuracies, their explanation capability is very limited. Therefore one of the main challenges in using ANNs in data mining applications is to extract explicit knowledge from them. Based on this motivation, a novel approach is proposed in this paper for generating classification rules from feed forward type ANNs. Although there are several approaches in the literature for classification rule extraction from ANNs, the present approach is fundamentally different from them. In the previous studies, ANN training and rule extraction is generally performed independently in a sequential (hierarchical) manner. However, in the present study, training and rule extraction phases are integrated within a multiple objective evaluation framework for generating accurate classification rules directly. The proposed approach makes use of differential evolution algorithm for training and touring ant colony optimization algorithm for rule extracting. The proposed algorithm is named as DIFACONN-miner. Experimental study on the benchmark data sets and comparisons with some other classical and state-of-the art rule extraction algorithms has shown that the proposed approach has a big potential to discover more accurate and concise classification rules.

[1]  Urszula Markowska-Kaczmar,et al.  Rule Extraction from Neural Network by Genetic Algorithm with Pareto Optimization , 2004, ICAISC.

[2]  A. M. M. Sharif Ullah,et al.  Knowledge extraction from time series and its application to surface roughness simulation , 2006, Inf. Knowl. Syst. Manag..

[3]  A. M. M. Sharif Ullah,et al.  A human-assisted knowledge extraction method for machining operations , 2006, Adv. Eng. Informatics.

[4]  Xing Zhang,et al.  A new approach to classification based on association rule mining , 2006, Decis. Support Syst..

[5]  H. Baxter Williams,et al.  A Survey , 1992 .

[6]  A. M. M. Sharif Ullah,et al.  Manufacturing process performance prediction by integrating crisp and granular information , 2005, J. Intell. Manuf..

[7]  M. Esmel ElAlami,et al.  Extracting rules from trained neural network using GA for managing E-business , 2004, Appl. Soft Comput..

[8]  Peter A. N. Bosman,et al.  Proceedings of the Genetic and Evolutionary Computation Conference - GECCO - 2006 , 2006 .

[9]  Peter I. Cowling,et al.  A greedy classification algorithm based on association rule , 2007, Appl. Soft Comput..

[10]  Lale Özbakir,et al.  MEPAR-miner: Multi-expression programming for classification rule mining , 2007, Eur. J. Oper. Res..

[11]  Nelson F. F. Ebecken,et al.  Extracting rules from multilayer perceptrons in classification problems: A clustering-based approach , 2006, Neurocomputing.

[12]  Ron Kohavi,et al.  The Power of Decision Tables , 1995, ECML.

[13]  Elsadig Musa Ahmed,et al.  Ict and Human Capital Role in Achieving Knowledge-Based Economy: Applications on Malaysia's Manufacturing , 2006, J. Inf. Knowl. Manag..

[14]  Dervis Karaboga,et al.  Designing digital IIR filters using ant colony optimisation algorithm , 2004, Eng. Appl. Artif. Intell..

[15]  Joni-Kristian Kämäräinen,et al.  Differential Evolution Training Algorithm for Feed-Forward Neural Networks , 2003, Neural Processing Letters.

[16]  Ian Witten,et al.  Data Mining , 2000 .

[17]  K. Bandurski,et al.  A parallel differential evolution algorithm for neural network training , 2006, International Symposium on Parallel Computing in Electrical Engineering (PARELEC'06).

[18]  M. Dorigo,et al.  1 Positive Feedback as a Search Strategy , 1991 .

[19]  Heikki Mannila,et al.  Principles of Data Mining , 2001, Undergraduate Topics in Computer Science.

[20]  Guido Bologna,et al.  Is it worth generating rules from neural network ensembles? , 2004, J. Appl. Log..

[21]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[22]  Rajib Mall,et al.  Predictive and comprehensible rule discovery using a multi-objective genetic algorithm , 2006, Knowl. Based Syst..

[23]  Ian H. Witten,et al.  Generating Accurate Rule Sets Without Global Optimization , 1998, ICML.

[24]  Lipo Wang,et al.  Rule extraction by genetic algorithms based on a simplified RBF neural network , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[25]  Rudy Setiono,et al.  An Approach To Generate Rules From Neural Networks for Regression Problems , 2004, Eur. J. Oper. Res..

[26]  Alex Alves Freitas,et al.  A new version of the ant-miner algorithm discovering unordered rule sets , 2006, GECCO '06.

[27]  Donald C. Wunsch,et al.  Neural network explanation using inversion , 2007, Neural Networks.

[28]  Lale Özbakir,et al.  Rule Extraction from Neural Networks Via Ant Colony Algorithm for Data Mining Applications , 2008, LION.

[29]  Krzysztof Bandurski,et al.  A Parallel Differential Evolution Algorithm A Parallel Differential Evolution Algorithm , 2006, PARELEC.

[30]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[31]  Hisao Ishibuchi,et al.  Three-objective genetics-based machine learning for linguistic rule extraction , 2001, Inf. Sci..

[32]  Bart Baesens,et al.  Inferring descriptive and approximate fuzzy rules for credit scoring using evolutionary algorithms , 2007, Eur. J. Oper. Res..

[33]  Xin Yao,et al.  Evolutionary Artificial Neural Networks , 1993, Int. J. Neural Syst..

[34]  Kay Chen Tan,et al.  A Dual-Objective Evolutionary Algorithm for Rules Extraction in Data Mining , 2006, Comput. Optim. Appl..

[35]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[36]  Rudy Setiono,et al.  A note on knowledge discovery using neural networks and its application to credit card screening , 2009, Eur. J. Oper. Res..

[37]  Alex A. Freitas,et al.  Extracting comprehensible rules from neural networks via genetic algorithms , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[38]  Yoshikazu Ikeda,et al.  Neural Network Rule Extraction by Using the Genetic Programming and Its Applications to Explanatory Classifications , 2005, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[39]  Vadlamani Ravi,et al.  Differential evolution trained wavelet neural networks: Application to bankruptcy prediction in banks , 2009, Expert Syst. Appl..

[40]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[41]  Hisao Ishibuchi,et al.  Evolutionary Multiobjective Optimization for Generating an Ensemble of Fuzzy Rule-Based Classifiers , 2003, GECCO.

[42]  Philip S. Yu,et al.  Data Mining: An Overview from a Database Perspective , 1996, IEEE Trans. Knowl. Data Eng..

[43]  Simon Parsons,et al.  Principles of Data Mining by David J. Hand, Heikki Mannila and Padhraic Smyth, MIT Press, 546 pp., £34.50, ISBN 0-262-08290-X , 2004, The Knowledge Engineering Review.

[44]  Urszula Markowska-Kaczmar.  The influence of parameters in evolutionary based rule extraction method from neural network , 2005, 5th International Conference on Intelligent Systems Design and Applications (ISDA'05).

[45]  A. F. Adams,et al.  The Survey , 2021, Dyslexia in Higher Education.

[46]  Kay Chen Tan,et al.  A coevolutionary algorithm for rules discovery in data mining , 2006, Int. J. Syst. Sci..

[47]  Duc Truong Pham,et al.  RULES-F: A fuzzy inductive learning algorithm , 2006 .

[48]  Novruz Allahverdi,et al.  Rule extraction from trained adaptive neural networks using artificial immune systems , 2009, Expert Syst. Appl..

[49]  Hisao Ishibuchi,et al.  Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems , 1997, Fuzzy Sets Syst..

[50]  Duc Truong Pham,et al.  Engineering Science Engineers, Part C: Journal of Mechanical , 2011 .

[51]  Duc Truong Pham,et al.  Computational Intelligence: for Engineering and Manufacturing , 2007 .

[52]  Hisao Ishibuchi,et al.  Comparison of Heuristic Criteria for Fuzzy Rule Selection in Classification Problems , 2004, Fuzzy Optim. Decis. Mak..

[53]  Renpu Li,et al.  Mining classification rules using rough sets and neural networks , 2004, Eur. J. Oper. Res..

[54]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..