A fast iterative algorithm for fisher discriminant using heterogeneous kernels

We propose a fast iterative classification algorithm for Kernel Fisher Discriminant (KFD) using heterogeneous kernel models. In contrast with the standard KFD that requires the user to predefine a kernel function, we incorporate the task of choosing an appropriate kernel into the optimization problem to be solved. The choice of kernel is defined as a linear combination of kernels belonging to a potentially large family of different positive semidefinite kernels. The complexity of our algorithm does not increase significantly with respect to the number of kernels on the kernel family. Experiments on several benchmark datasets demonstrate that generalization performance of the proposed algorithm is not significantly different from that achieved by the standard KFD in which the kernel parameters have been tuned using cross validation. We also present results on a real-life colon cancer dataset that demonstrate the efficiency of the proposed method.

[1]  Gunnar Rätsch,et al.  A Mathematical Programming Approach to the Kernel Fisher Algorithm , 2000, NIPS.

[2]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[3]  J. Suykens,et al.  Ensemble Learning of Coupled Parmeterised Kernel Models , 2003 .

[4]  Jean Faivre.  Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults, P.J. Pickhardt, J.R. Choi, I. Hwang, J.A. Butler, M.L. Puckett, H.A. Hildebrandt, in: N Engl J Med, 349. (2003), 2191 , 2004 .

[5]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[6]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[7]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[8]  Glenn Fung,et al.  Finite Newton method for Lagrangian support vector machine classification , 2003, Neurocomputing.

[9]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[10]  Glenn Fung,et al.  Proximal support vector machine classifiers , 2001, KDD '01.

[11]  Robert A. Lordo,et al.  Learning from Data: Concepts, Theory, and Methods , 2001, Technometrics.

[12]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[13]  Kristin P. Bennett,et al.  MARK: a boosting algorithm for heterogeneous kernel models , 2002, KDD.

[14]  Olvi L. Mangasarian,et al.  Nonlinear Programming , 1969 .

[15]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[16]  James C. Bezdek,et al.  Convergence of Alternating Optimization , 2003, Neural Parallel Sci. Comput..

[17]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[18]  James C. Bezdek,et al.  Some Notes on Alternating Optimization , 2002, AFSS.

[19]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[20]  Xuegong Zhang,et al.  Kernel MSE algorithm: a unified framework for KFD, LS-SVM and KRR , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[21]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[22]  Bernhard Schölkopf,et al.  Regularization Networks and Support Vector Machines , 2000 .

[23]  Olvi L. Mangasarian,et al.  Generalized Support Vector Machines , 1998 .

[24]  P. Pickhardt,et al.  Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults. , 2003, The New England journal of medicine.

[25]  Michael I. Jordan,et al.  Fast Kernel Learning using Sequential Minimal Optimization , 2004 .

[26]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .