Fuzzy rules extraction from support vector machines for multi-class classification

This paper proposes a new method for fuzzy rule extraction from trained support vector machines (SVMs) for multi-class problems, named FREx_SVM. SVMs have been used in a variety of applications. However, they are considered “black box models,” where no interpretation about the input–output mapping is provided. Some methods to reduce this limitation have already been proposed, but they are restricted to binary classification problems and to the extraction of symbolic rules with intervals or functions in their antecedents. In order to improve the interpretability of the generated rules, this paper presents a new model for extracting fuzzy rules from a trained SVM. The proposed model is suited for classification in multi-class problems and includes a wrapper feature selection algorithm. It is evaluated in four benchmark databases, and results obtained demonstrate its capacity to generate a reduced set of interpretable fuzzy rules that explains both the classification database and the influence of each input variable on the determination of the final class.

[1]  D. Sculley,et al.  Relaxed online SVMs for spam filtering , 2007, SIGIR.

[2]  Jerry M. Mendel,et al.  Generating fuzzy rules by learning from examples , 1992, IEEE Trans. Syst. Man Cybern..

[3]  Alexander J. Smola,et al.  Support Vector Method for Function Approximation, Regression Estimation and Signal Processing , 1996, NIPS.

[4]  S. Gunn.  Support Vector Machines for Classification and Regression , 1998 .

[5]  Jung-Hsien Chiang,et al.  Support vector learning mechanism for fuzzy rule-based modeling: a new approach , 2004, IEEE Trans. Fuzzy Syst..

[6]  Andrew P. Bradley,et al.  Rule extraction from support vector machines: A review , 2010, Neurocomputing.

[7]  Shigeo Abe,et al.  Fuzzy support vector machines for multiclass problems , 2002, ESANN.

[8]  Ron Kohavi,et al.  Irrelevant Features and the Subset Selection Problem , 1994, ICML.

[9]  Yiming Yang,et al.  Modified Logistic Regression: An Approximation to SVM and Its Applications in Large-Scale Text Categorization , 2003, ICML.

[10]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[11]  Daniel Cremers,et al.  Efficient Feature Subset Selection for Support Vector Machines , 2001 .

[12]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[13]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[14]  Andreu Català,et al.  Rule-Based Learning Systems for Support Vector Machines , 2006, Neural Processing Letters.

[15]  D Haussler,et al.  Knowledge-based analysis of microarray gene expression data by using support vector machines. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[17]  Koby Crammer,et al.  On the Learnability and Design of Output Codes for Multiclass Problems , 2002, Machine Learning.

[18]  Gunnar Rätsch,et al.  Predicting Time Series with Support Vector Machines , 1997, ICANN.

[19]  Ryan M. Rifkin,et al.  In Defense of One-Vs-All Classification , 2004, J. Mach. Learn. Res..

[20]  John E. Moody,et al.  Principled Architecture Selection for Neural Networks: Application to Corporate Bond Rating Prediction , 1991, NIPS.

[21]  Ulrich H.-G. Kreßel,et al.  Pairwise classification and support vector machines , 1999 .

[22]  Joachim Diederich,et al.  Eclectic Rule-Extraction from Support Vector Machines , 2005 .

[23]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[24]  T. Martin McGinnity,et al.  An approach for on-line extraction of fuzzy rules using a self-organising fuzzy neural network , 2005, Fuzzy Sets Syst..

[25]  Joachim Diederich,et al.  Learning-Based Rule-Extraction From Support Vector Machines: Performance On Benchmark Data Sets , 2004 .

[26]  J. Mendel.  Fuzzy logic systems for engineering: a tutorial , 1995, Proc. IEEE.

[28]  Andreu Català,et al.  Rule extraction from support vector machines , 2002, ESANN.

[29]  Veera Boonjing,et al.  Fuzzy Rules Generation and Extraction from Support Vector Machine Based on Kernel Function Firing Signals , 2010 .

[30]  Joachim Diederich,et al.  Rule Extraction from Support Vector Machines: An Introduction , 2008, Rule Extraction from Support Vector Machines.

[31]  M.M.B.R. Vellasco,et al.  Inverted hierarchical neuro-fuzzy BSP system: a novel neuro-fuzzy model for pattern classification and rule extraction in databases , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[32]  Jianping Li,et al.  A multiple kernel support vector machine scheme for feature selection and rule extraction from gene expression data of cancer tissue , 2007, Artif. Intell. Medicine.

[33]  Gregory Piatetsky-Shapiro,et al.  Advances in Knowledge Discovery and Data Mining , 2004, Lecture Notes in Computer Science.

[34]  Jason Weston,et al.  Multi-Class Support Vector Machines , 1998 .

[35]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[36]  Joachim Diederich,et al.  Rule Extraction from Support Vector Machines , 2008, Studies in Computational Intelligence.

[37]  Xiuju Fu,et al.  Extracting the knowledge embedded in support vector machines , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[38]  Harris Drucker,et al.  Support vector machines for spam categorization , 1999, IEEE Trans. Neural Networks.

[39]  Tong Zhang,et al.  An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods , 2001, AI Mag..

[40]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[41]  Juan Luis Castro,et al.  Extraction of fuzzy rules from support vector machines , 2007, Fuzzy Sets Syst..

[42]  Glenn Fung,et al.  Rule extraction from linear support vector machines , 2005, KDD '05.