On Kernel Regression with Data-Dependent Kernels

The primary hyperparameter in kernel regression (KR) is the choice of kernel. In most theoretical studies of KR, one assumes the kernel is fixed before seeing the training data. Under this assumption, it is known that the optimal kernel is equal to the prior covariance of the target function. In this note, we consider KR in which the kernel may be updated after seeing the training data. We point out that an analogous choice of kernel using the posterior of the target function is optimal in this setting. Connections to the view of deep neural networks as data-dependent kernel learners are discussed.

[1]  Yamini Bansal,et al.  Limitations of the NTK for Understanding Generalization in Deep Learning , 2022, arXiv.org.

[2]  C. Pehlevan,et al.  Neural Networks as Kernel Learners: The Silent Alignment Effect , 2021, ICLR.

[3]  Philip M. Long.  Properties of the After Kernel , 2021, ArXiv.

[4]  Surya Ganguli,et al.  Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel , 2020, NeurIPS.

[5]  R Devon Hjelm,et al.  Implicit Regularization via Neural Feature Alignment , 2020, AISTATS.

[6]  Arthur Jacot,et al.  Kernel Alignment Risk Estimator: Risk Prediction from Training Data , 2020, NeurIPS.

[7]  Blake Bordelon,et al.  Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks , 2020, ICML.

[8]  Jaehoon Lee,et al.  Wide neural networks of any depth evolve as linear models under gradient descent , 2019, NeurIPS.

[9]  Arthur Jacot,et al.  Neural Tangent Kernel: Convergence and Generalization in Neural Networks , 2018, NeurIPS.

[10]  Richard E. Turner,et al.  Gaussian Process Behaviour in Wide Deep Neural Networks , 2018, ICLR.

[11]  Jeffrey Pennington,et al.  Deep Neural Networks as Gaussian Processes , 2017, ICLR.

[12]  Mehryar Mohri,et al.  Algorithms for Learning Kernels Based on Centered Alignment , 2012, J. Mach. Learn. Res..

[13]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[14]  N. Cristianini,et al.  On Kernel-Target Alignment , 2001, NIPS.

[15]  John C. Duchi,et al.  Learning Kernels with Random Features , 2016, NIPS.