Kernel matching pursuit for large datasets

Kernel matching pursuit is a greedy algorithm for building an approximation of a discriminant function as a linear combination of some basis functions selected from a kernel-induced dictionary. Here we propose a modification of the kernel matching pursuit algorithm that aims at making the method practical for large datasets. Starting from an approximating algorithm, the weak greedy algorithm, we introduce a stochastic method for reducing the search space at each iteration. Then we study the implications of using an approximate algorithm and we show how one can control the trade-off between the accuracy and the need for resources. Finally, we present some experiments performed on a large dataset that support our approach and illustrate its applicability.

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[3]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[4]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[5]  M. Lipson,et al.  Tipping , 1968, Nursing mirror and midwives journal.

[6]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[7]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[8]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[9]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[10]  Pascal Vincent,et al.  Kernel Matching Pursuit , 2002, Machine Learning.

[11]  Charles Annis,et al.  Statistical Distributions in Engineering , 2001, Technometrics.

[12]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[13]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[14]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[15]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[16]  Rémi Gribonval,et al.  Approximate Weak Greedy Algorithms , 2001, Adv. Comput. Math..

[17]  Vladimir N. Temlyakov,et al.  Weak greedy algorithms[*]This research was supported by National Science Foundation Grant DMS 9970326 and by ONR Grant N00014‐96‐1‐1003. , 2000, Adv. Comput. Math..

[18]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .