Learning with Box Kernels

Supervised examples and prior knowledge on regions of the input space have been profitably integrated in kernel machines to improve the performance of classifiers in different real-world contexts. The proposed solutions, which rely on the unified supervision of points and sets, have been mostly based on specific optimization schemes in which, as usual, the kernel function operates on points only. In this paper, arguments from variational calculus are used to support the choice of a special class of kernels, referred to as box kernels, which emerges directly from the choice of the kernel function associated with a regularization operator. It is proven that there is no need to search for kernels to incorporate the structure deriving from the supervision of regions of the input space, because the optimal kernel arises as a consequence of the chosen regularization operator. Although most of the given results hold for sets, we focus attention on boxes, whose labeling is associated with their propositional description. Based on different assumptions, some representer theorems are given that dictate the structure of the solution in terms of box kernel expansion. Successful results are given for problems of medical diagnosis, image, and text categorization.

[1]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[2]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[3]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[4]  Thomas Gärtner,et al.  Simpler knowledge-based support vector machines , 2006, ICML.

[5]  David Haussler,et al.  Convolution kernels on discrete structures , 1999 .

[6]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[7]  Tomaso A. Poggio,et al.  Extensions of a Theory of Networks for Approximation and Learning , 1990, NIPS.

[8]  Simon Haykin,et al.  On Different Facets of Regularization Theory , 2002, Neural Computation.

[9]  Gérard Bloch,et al.  Incorporating prior knowledge in support vector machines for classification: A review , 2008, Neurocomputing.

[10]  Glenn Fung,et al.  Knowledge-Based Support Vector Machine Classifiers , 2002, NIPS.

[11]  Glenn Fung,et al.  Proximal Knowledge-based Classification , 2009 .

[12]  Marcello Sanguineti,et al.  Learning with Boundary Conditions , 2013, Neural Computation.

[13]  Jude W. Shavlik,et al.  Online Knowledge-Based Support Vector Machines , 2010, ECML/PKDD.

[14]  Glenn Fung,et al.  Knowledge-Based Nonlinear Kernel Classifiers , 2003, COLT.

[15]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[16]  Gregory E. Fasshauer,et al.  Green’s Functions: Taking Another Look at Kernel Approximation, RadialBasis Functions, and Splines , 2012 .

[17]  Le Song,et al.  A Hilbert Space Embedding for Distributions , 2007, Discovery Science.

[18]  Olvi L. Mangasarian,et al.  Nonlinear Knowledge-Based Classification , 2008, IEEE Transactions on Neural Networks.

[19]  Bernhard Schölkopf,et al.  From Regularization Operators to Support Vector Kernels , 1997, NIPS.

[20]  M. Giaquinta,et al.  Calculus of Variations I , 1995 .

[21]  Bernhard Schölkopf,et al.  Hilbert Space Embeddings and Metrics on Probability Measures , 2009, J. Mach. Learn. Res..

[22]  Thomas Gärtner,et al.  Multi-Instance Kernels , 2002, ICML.

[23]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[24]  Qi Ye,et al.  Reproducing kernels of generalized Sobolev spaces via a Green function approach with distributional operators , 2011, Numerische Mathematik.

[25]  Qi Ye,et al.  Reproducing kernels of Sobolev spaces via a green kernel approach with differential operators and boundary operators , 2011, Adv. Comput. Math..