Similarity-based methods: a general framework for classification, approximation and association

Similarity-based methods (SBM) are a generalization of the minimal distance (MD) methods which form a basis of several machine learning and pattern recognition methods. Investigation of similarity leads to a fruitful framework in which many classification, approximation and association methods are accommodated. Probability p(C|X; M) of assign- ing class C to a vector X, given a classification model M, depends on adap- tive parameters and procedures used in construction of the model. Sys- tematic overview of choices available for model building is described and numerous improvements suggested. Similarity-Based Methods have nat- ural neural-network type realizations. Such neural network models as the Radial Basis Functions (RBF) and the Multilayer Perceptrons (MLPs) are included in this framework as special cases. SBM may also include several different submodels and a procedure to combine their results. Many new versions of similarity-based methods are derived from this framework. A search in the space of all methods belonging to the SBM framework finds a particular combination of parameterizations and procedures that is most appropriate for a given data. No single classification method can beat this approach. Preliminary implementation of SBM elements tested on a real- world datasets gave very good results.

[1]  Andreas Abecker,et al.  Optimized Nearest-Neighbor Classifiers Using Generated Instances , 1996, KI.

[2]  James C. Bezdek,et al.  Presupervised and post-supervised prototype classifier design , 1999, IEEE Trans. Neural Networks.

[3]  L. N. Kanal,et al.  Handbook of Statistics, Vol. 2. Classification, Pattern Recognition and Reduction of Dimensionality. , 1985 .

[4]  Tony R. Martinez,et al.  Improved Heterogeneous Distance Functions , 1996, J. Artif. Intell. Res..

[5]  Belur V. Dasarathy,et al.  Nearest neighbor (NN) norms: NN pattern classification techniques , 1991 .

[6]  Wolfram Schiffmann,et al.  Comparison of optimized backpropagation algorithms , 1993, ESANN.

[7]  Włodzisław Duch,et al.  Weighting and Selection of Features , 1999 .

[8]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[9]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[10]  Laveen N. Kanal,et al.  Classification, Pattern Recognition and Reduction of Dimensionality , 1982, Handbook of Statistics.

[11]  Antonio Bellacicco,et al.  Handbook of statistics 2: Classification, pattern recognition and reduction of dimensionality: P.R. KRISHNAIAH and L.N. KANAL (Eds.) North-Holland, Amsterdam, 1982, xxii + 903 pages, Dfl.275.00 , 1984 .

[12]  D. K. Y. Chiu,et al.  The ck-Nearest Neighbor Distance Network: A Network Using Class Boundary Feature Distances , 1997, ICONIP.

[13]  Yann LeCun,et al.  Efficient Pattern Recognition Using a New Transformation Distance , 1992, NIPS.

[14]  Yoichi Hayashi,et al.  Fuzzy and Crisp Logical Rule Extraction Methods in Application to Medical Data , 2000 .

[15]  L. Breiman.  Bias-variance, regularization, instability and stabilization , 1998 .

[16]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[17]  David W. Aha,et al.  Feature Weighting for Lazy Learning Algorithms , 1998 .

[18]  Sankar K. Pal,et al.  Knowledge-based fuzzy MLP for classification and rule generation , 1997, IEEE Trans. Neural Networks.

[19]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[20]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[21]  Sholom M. Weiss,et al.  An Empirical Comparison of Pattern Recognition, Neural Nets, and Machine Learning Classification Methods , 1989, IJCAI.

[22]  Hiroshi Motoda,et al.  Feature Extraction, Construction and Selection: A Data Mining Perspective , 1998 .

[23]  David L. Waltz,et al.  Memory-based reasoning , 1998 .

[24]  Włodzisław Duch.  A framework for similarity-based classification methods , 2000 .

[25]  D. Rubin.  Multiple Imputation After 18+ Years , 1996 .

[26]  Erkki Oja,et al.  Classification with learning k-nearest neighbors , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[27]  L I Kuncheva,et al.  A FUZZY GENERALIZED NEAREST PROTOTYPE CLASSIFIER , 1997 .

[28]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[29]  Włodzisław Duch,et al.  Neural minimal distance methods , 1997 .

[30]  Wlodzislaw Duch,et al.  Feature space mapping as a universal adaptive system , 1995 .

[31]  Thomas G. Dietterich,et al.  Readings in Machine Learning , 1991 .

[32]  Karol Grudzinski,et al.  A framework for similarity-based methods. , 1998 .

[33]  K. Gr,et al.  A general purpose separability criterion for classification systems , 1999 .

[34]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[35]  Christopher M. Bishop,et al.  Neural networks and machine learning , 1998 .

[36]  Lester Ingber,et al.  Adaptive simulated annealing (ASA): Lessons learned , 2000, ArXiv.

[37]  Thomas G. Dietterich,et al.  Locally Adaptive Nearest Neighbor Algorithms , 1993, NIPS.

[38]  Patrik Floréen.  The convergence of Hamming memory networks , 1991, IEEE Trans. Neural Networks.

[39]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification and Regression , 1995, NIPS.

[40]  J. Bezdek,et al.  Generalized k -nearest neighbor rules , 1986 .

[41]  Wlodzislaw Duch,et al.  Search and global minimization in similarity-based methods , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[42]  Jihoon Yang,et al.  DistAl: an inter-pattern distance-based constructive learning algorithm , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[43]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[44]  Wlodzislaw Duch,et al.  SBL-PM: A Simple Algorithm for Selection of Reference Instances for Similarity Based Methods , 2000, Intelligent Information Systems.

[45]  David G. Lowe,et al.  Similarity Metric Learning for a Variable-Kernel Classifier , 1995, Neural Computation.

[46]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[47]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[48]  Michal Morciniec,et al.  A Theoretical and Experimental Account of n-Tuple Classifier Performance , 1996, Neural Computation.

[49]  Huan Liu,et al.  Book review: Machine Learning, Neural and Statistical Classification Edited by D. Michie, D.J. Spiegelhalter and C.C. Taylor (Ellis Horwood Limited, 1994) , 1996, SGAR.

[50]  Wlodzislaw Duch,et al.  Extraction of crisp logical rules using constrained backpropagation networks , 1997, ESANN.

[51]  Ryszard S. Michalski,et al.  AQ-PM: A System for Partial Memory Learning , 1999 .

[52]  David W. Aha,et al.  Weighting Features , 1995, ICCBR.