Refinement of Operator-valued Reproducing Kernels

This paper studies the construction of a refinement kernel for a given operator-valued reproducing kernel such that the vector-valued reproducing kernel Hilbert space of the refinement kernel contains that of the given kernel as a subspace. The study is motivated from the need of updating the current operator-valued reproducing kernel in multi-task learning when underfitting or overfitting occurs. Numerical simulations confirm that the established refinement kernel method is able to meet this need. Various characterizations are provided based on feature maps and vector-valued integral representations of operator-valued reproducing kernels. Concrete examples of refining translation invariant and finite Hilbert-Schmidt operator-valued reproducing kernels are provided. Other examples include refinement of Hessian of scalar-valued translation-invariant kernels and transformation kernels. Existence and properties of operator-valued reproducing kernels preserved during the refinement process are also investigated.

[1]  M. Solomjak,et al.  Spectral Theory of Self-Adjoint Operators in Hilbert Space , 1987 .

[2]  Yuesheng Xu,et al.  Refinement of Reproducing Kernels , 2009, J. Mach. Learn. Res..

[3]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[4]  S. Mallat.  Multiresolution approximations and wavelet orthonormal bases of L^2(R) , 1989 .

[5]  S. Bochner,et al.  Lectures on Fourier integrals : with an author's supplement on monotonic functions, Stieltjes integrals, and harmonic analysis , 1959 .

[6]  S. Berberian.  Notes on spectral theory , 1966 .

[7]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[8]  George Pedrick,et al.  Theory of reproducing kernels for Hilbert spaces of vector valued functions , 1957 .

[9]  Sayan Mukherjee,et al.  Learning Coordinate Covariances via Gradients , 2006, J. Mach. Learn. Res..

[10]  W. Rudin.  Real and complex analysis , 1968 .

[11]  N. SIAMJ.,et al.  DIVERGENCE-FREE KERNEL METHODS FOR APPROXIMATING THE STOKES PROBLEM , 2009 .

[12]  Colin Campbell,et al.  Learning Coordinate Gradients with Multi-Task Kernels , 2008, COLT.

[13]  Svenja Lowitzsch.  Approximation and interpolation employing divergence-free radial basis functions with applications , 2004 .

[14]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[15]  W. Rudin.  Real and complex analysis, 3rd ed. , 1987 .

[16]  Felipe Cucker,et al.  Learning Theory: An Approximation Theory Viewpoint: On the bias–variance problem , 2007 .

[17]  P. Fillmore.  Notes on operator theory , 1970 .

[18]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[19]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[20]  C. Carmeli,et al.  VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM , 2006 .

[21]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[22]  J. Burbea,et al.  Banach and Hilbert spaces of vector-valued functions: Their general theory and applications to holomorphy , 1984 .

[23]  J. Diestel,et al.  On vector measures , 1974 .

[24]  採編典藏組.  Society for Industrial and Applied Mathematics(SIAM) , 2008 .

[25]  Yuesheng Xu,et al.  Reproducing kernel Banach spaces for machine learning , 2009, 2009 International Joint Conference on Neural Networks.

[26]  Yuesheng Xu,et al.  Refinable Kernels , 2007, J. Mach. Learn. Res..

[27]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[28]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[29]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[30]  Sayan Mukherjee,et al.  Estimation of Gradients and Coordinate Covariation in Classification , 2006, J. Mach. Learn. Res..

[31]  Felipe Cucker,et al.  Learning Theory: An Approximation Theory Viewpoint (Cambridge Monographs on Applied & Computational Mathematics) , 2007 .

[32]  J. Retherford.  Review: J. Diestel and J. J. Uhl, Jr., Vector measures , 1978 .

[33]  Charles A. Micchelli,et al.  Universal Multi-Task Kernels , 2008, J. Mach. Learn. Res..

[34]  C. Carmeli,et al.  Vector valued reproducing kernel Hilbert spaces and universality , 2008, 0807.1659.

[35]  Yuesheng Xu,et al.  Universal Kernels , 2006, J. Mach. Learn. Res..

[36]  Liang Zhao,et al.  On the Inclusion Relation of Reproducing Kernel Hilbert Spaces , 2011, ArXiv.

[37]  Charles A. Micchelli,et al.  Learning Multiple Tasks with Kernel Methods , 2005, J. Mach. Learn. Res..

[38]  D. Whittaker,et al.  A Course in Functional Analysis , 1991, The Mathematical Gazette.