Characterizing the Representer Theorem

The representer theorem assures that kernel methods retain optimality under penalized empirical risk minimization. While a sufficient condition on the form of the regularizer guaranteeing the representer theorem has been known since the initial development of kernel methods, necessary conditions have only been investigated recently. In this paper we completely characterize the necessary and sufficient conditions on the regularizer that ensure the representer theorem holds. The results are surprisingly simple yet broaden the conditions where the representer theorem is known to hold. Extension to the matrix domain is also addressed.

[1]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[2]  Laurent Schwartz,et al.  Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés (Noyaux reproduisants) , 1964 .

[3]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[4]  Grace Wahba,et al.  Spline Models for Observational Data , 1990 .

[5]  A. Kirsch.  An Introduction to the Mathematical Theory of Inverse Problems , 1996, Applied Mathematical Sciences.

[6]  Bernhard Schölkopf,et al.  A Generalized Representer Theorem , 2001, COLT/EuroCOLT.

[7]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2006 .

[8]  S. V. N. Vishwanathan,et al.  Leaving the Span , 2005, COLT.

[9]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[10]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[11]  C. Carmeli,et al.  VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM , 2006 .

[12]  Massimiliano Pontil,et al.  Convex multi-task feature learning , 2008, Machine Learning.

[13]  Francis R. Bach,et al.  A New Approach to Collaborative Filtering: Operator Estimation with Spectral Regularization , 2008, J. Mach. Learn. Res..

[14]  Charles A. Micchelli,et al.  When is there a representer theorem? Vector versus matrix regularizers , 2008, J. Mach. Learn. Res..

[15]  Bernhard Schölkopf,et al.  The representer theorem for Hilbert spaces: a necessary and sufficient condition , 2012, NIPS.

[16]  Wojciech Kotlowski,et al.  Kernelization of matrix updates, when and how? , 2014, Theor. Comput. Sci..