Linear separability and classification complexity

We study the relationship between linear separability and the level of complexity of classification data sets. Linearly separable classification problems are generally easier to solve than non linearly separable ones. This suggests a strong correlation between linear separability and classification complexity. We propose a novel and simple method for quantifying the complexity of the classification problem. The method, which is shown below, reduces any two class classification problem to a sequence of linearly separable steps. The number of such reduction steps could be viewed as measuring the degree of non-separability and hence the complexity of the problem. This quantification in turn can be used as a measure for the complexity of classification data sets. Results obtained using several benchmarks are provided.

[1]  G. Gates,et al.  The reduced nearest neighbor rule (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[2]  Tuan D. Pham.  GeoEntropy: A measure of complexity and similarity , 2010, Pattern Recognit..

[3]  David A. Elizondo,et al.  The recursive deterministic perceptron neural network , 1998, Neural Networks.

[4]  Tacha Hicks,et al.  Forensic Interpretation of Glass Evidence , 2000 .

[5]  O. Mangasarian,et al.  Multisurface method of pattern separation for medical diagnosis applied to breast cytology. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Sholom M. Weiss,et al.  Computer Systems That Learn , 1990 .

[7]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[8]  Ravi Kothari,et al.  Feature subset selection using a new definition of classifiability , 2003, Pattern Recognit. Lett..

[9]  Ravi Kothari,et al.  Classifiability-based omnivariate decision trees , 2005, IEEE Transactions on Neural Networks.

[10]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[11]  Belur V. Dasarathy,et al.  Nosing Around the Neighborhood: A New System Structure and Classification Rule for Recognition in Partially Exposed Environments , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Tin Kam Ho,et al.  Domain of competence of XCS classifier system in complexity measurement space , 2005, IEEE Transactions on Evolutionary Computation.

[13]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[14]  G. Gates.  The Reduced Nearest Neighbor Rule , 1998 .

[15]  D. Elizondo.  Searching for linearly separable subsets using the class of linear separability method , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[16]  Ming Dong.  A New Measure of Classifiability and its Applications , 2001 .

[17]  Terrence J. Sejnowski,et al.  Analysis of hidden units in a layered network trained to classify sonar targets , 1988, Neural Networks.

[18]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[19]  David A. Elizondo,et al.  The linear separability problem: some testing methods , 2006, IEEE Transactions on Neural Networks.

[20]  Ivan Bratko,et al.  ASSISTANT 86: A Knowledge-Elicitation Tool for Sophisticated Users , 1987, EWSL.

[21]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[22]  David A. Elizondo,et al.  Current Trends on Knowledge Extraction and Neural Networks , 2005, ICANN.

[23]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[24]  O. Mangasarian,et al.  Robust linear programming discrimination of two linearly inseparable sets , 1992 .

[25]  José Martínez Sotoca,et al.  Data Characterization for Effective Prototype Selection , 2005, IbPRIA.

[26]  Francisco Herrera,et al.  Domains of competence of fuzzy rule based classification systems with data complexity measures: A case of study using a fuzzy hybrid genetic based machine learning method , 2010, Fuzzy Sets Syst..

[27]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[28]  P. Diaconis,et al.  Computer-Intensive Methods in Statistics , 1983 .

[29]  Mohamed Tajine,et al.  Adapting the 2-class recursive deterministic perceptron neural network to m-classes , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[30]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[31]  Tin Kam Ho,et al.  Complexity Measures of Supervised Classification Problems , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Richard Baumgartner,et al.  Data complexity assessment in undersampled classification of high-dimensional biomedical data , 2006, Pattern Recognit. Lett..