Approximation of high-dimensional kernel matrices by multilevel circulant matrices

Kernels are important in developing a variety of numerical methods, such as approximation, interpolation, neural networks, machine learning and meshless methods for solving engineering problems. A common problem of these kernel-based methods is to calculate inverses of kernel matrices generated by a kernel function and a set of points. Due to the denseness of these matrices, finding their inverses is computationally costly. To overcome this difficulty, we introduce in this paper an approximation of the kernel matrices by appropriate multilevel circulant matrices so that the fast Fourier transform can be applied to reduce the computational cost. Convergence analysis for the proposed approximation is established based on certain decay properties of the kernels.

[1]  R. V. Churchill,et al.  Lectures on Fourier Integrals , 1959 .

[2]  T. J. Rivlin,et al.  The optimal recovery of smooth functions , 1976 .

[3]  Ward Cheney,et al.  A course in approximation theory , 1999 .

[4]  Holger Wendland,et al.  Kernel techniques: From machine learning to meshless methods , 2006, Acta Numerica.

[5]  Yuesheng Xu,et al.  Reproducing kernel Banach spaces for machine learning , 2009, 2009 International Joint Conference on Neural Networks.

[6]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[7]  Charles A. Micchelli,et al.  A Survey of Optimal Recovery , 1977 .

[8]  C. R. Deboor,et al.  A practical guide to splines , 1978 .

[10]  Yuesheng Xu,et al.  Approximation of kernel matrices by circulant matrices and its application in kernel selection methods , 2010 .

[11]  Stéphane Jaffard.  Propriétés des matrices « bien localisées » près de leur diagonale et quelques applications , 1990 .

[12]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[13]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[14]  Yuesheng Xu,et al.  Refinable Kernels , 2007, J. Mach. Learn. Res..

[15]  Charles A. Micchelli,et al.  Learning the Kernel Function via Regularization , 2005, J. Mach. Learn. Res..

[16]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[17]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[18]  C. Gasquet,et al.  Fourier analysis and applications , 1998 .

[19]  Johan A. K. Suykens,et al.  Advances in learning theory : methods, models and applications , 2003 .

[20]  Ying Jiang,et al.  Fast discrete algorithms for sparse Fourier expansions of high dimensional functions , 2010, J. Complex..

[21]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[22]  Yuesheng Xu,et al.  Refinement of Reproducing Kernels , 2009, J. Mach. Learn. Res..

[23]  S. R. Simanca,et al.  On Circulant Matrices , 2012 .

[24]  Yuesheng Xu,et al.  Universal Kernels , 2006, J. Mach. Learn. Res..

[25]  E. E. Tyrtyshnikov.  A unifying approach to some old and new theorems on distribution and clustering , 1996 .

[26]  T. J. Rivlin,et al.  Optimal Estimation in Approximation Theory , 1977 .

[27]  Bernhard Schölkopf,et al.  On a Kernel-Based Method for Pattern Recognition, Regression, Approximation, and Operator Inversion , 1998, Algorithmica.

[28]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.