BINNING IN GAUSSIAN KERNEL REGULARIZATION

Gaussian kernel regularization is widely used in the machine learning literature and has proved successful in many empirical experiments. The periodic version of Gaussian kernel regularization has been shown to be minimax rate optimal in estimating functions in any finite order Sobolev space. However, for a data set with n points, the computation complexity of the Gaussian kernel regularization method is of order O(n). In this paper we propose to use binning to reduce the computation of Gaussian kernel regularization in both regression and classification. For periodic Gaussian kernel regression, we show that the binned estimator achieves the same minimax rates as the unbinned estimator, but the computation is reduced to O(m) with m as the number of bins. To achieve the minimax rate in the kth order Sobolev space, m needs to be in the order of O(kn), which makes the binned estimator computation of order O(n) for k = 1, and even less for larger k. Our simulations show that the binned estimator (binning 120 data points into 20 bins in our simulation) provides almost the same accuracy with only 0.4% of computation time. For classification, binning with L2-loss Gaussian kernel regularization and Gaussian kernel Support Vector Machines is tested in a polar cloud detection problem.

[1]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[2]  Colin L. Mallows,et al.  Some Comments on Cp , 2000, Technometrics.

[3]  P. Bühlmann,et al.  Analyzing Bagging , 2001 .

[4]  Zhao Hong-hai.  Semi-Supervised Support Vector Machines for Data Classification , 2004 .

[5]  Hansong Zhang,et al.  Gacv for support vector machines , 2000 .

[6]  Berwin A. Turlach,et al.  A note on design transformation and binning in nonparametric curve estimation , 1998 .

[7]  Grace Wahba,et al.  Spline Models for Observational Data , 1990 .

[8]  L. Brown,et al.  Asymptotic equivalence of nonparametric regression and white noise , 1996 .

[9]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[10]  L. Brown,et al.  Statistical properties of the method of regularization with periodic Gaussian reproducing kernel , 2004, math/0410093.

[11]  F. Girosi,et al.  From regularization to radial, tensor and additive splines , 1993, Neural Networks for Signal Processing III - Proceedings of the 1993 IEEE-SP Workshop.

[12]  Bernhard Schölkopf,et al.  Generalization Performance of Regularization Networks and Support Vector Machines via Entropy Numbers of Compact Operators , 1998 .

[13]  O. Mangasarian,et al.  Semi-Supervised Support Vector Machines for Unlabeled Data Classification , 2001 .

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[15]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..