A New Generalized Error Path Algorithm for Model Selection

Model selection with cross validation (CV) is very popular in machine learning. However, CV with grid and other common search strategies cannot guarantee to find the model with minimum CV error, which is often the ultimate goal of model selection. Recently, various solution path algorithms have been proposed for several important learning algorithms including support vector classification, Lasso, and so on. However, they still do not guarantee to find the model with minimum CV error. In this paper, we first show that the solution paths produced by various algorithms have the property of piecewise linearity. Then, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter, based on the solution path. Finally, we propose a new generalized error path algorithm (GEP), and prove that it will find the model with minimum CV error for the entire range of the regularization parameter. The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search.

[1]  Gang Wang,et al.  A New Solution Path Algorithm in Support Vector Regression , 2008, IEEE Transactions on Neural Networks.

[2]  Tapio Pahikkala,et al.  Efficient cross-validation for kernelized least-squares regression with sparse basis expansions , 2012, Machine Learning.

[3]  Robert Koprowski,et al.  Machine learning, medical diagnosis, and biomedical engineering research - commentary , 2014, Biomedical engineering online.

[4]  Ronald L. Rivest,et al.  Introduction to Algorithms, third edition , 2009 .

[5]  M. Goddard,et al.  LASSO with cross-validation for genomic selection. , 2009, Genetics research.

[6]  S. Rosset,et al.  Piecewise linear regularized solution paths , 2007, 0708.2197.

[7]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[8]  Pierre Alquier,et al.  PAC-Bayesian AUC classification and scoring , 2014, NIPS.

[9]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[10]  Ichiro Takeuchi,et al.  Suboptimal Solution Path Algorithm for Support Vector Machine , 2011, ICML.

[11]  Robert Tibshirani,et al.  The Entire Regularization Path for the Support Vector Machine , 2004, J. Mach. Learn. Res..

[12]  Geoffrey E. Hinton.  A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[13]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[14]  Jianbo Yang,et al.  An Improved Algorithm for the Solution of the Regularization Path of Support Vector Machine , 2010, IEEE Transactions on Neural Networks.

[15]  Bin Gu,et al.  Regularization Path for $\nu$ -Support Vector Classification , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[17]  Michael Jahrer,et al.  Collaborative Filtering Ensemble , 2012, KDD Cup.

[18]  Eric Horvitz,et al.  Considering Cost Asymmetry in Learning Classifiers , 2006, J. Mach. Learn. Res..

[19]  Jian-Bo Yang,et al.  Determination of Global Minima of Some Common Validation Functions in Support Vector Machine , 2011, IEEE Transactions on Neural Networks.

[20]  Michael Izbicki,et al.  Algebraic classifiers: a generic approach to fast cross-validation, online training, and parallel training , 2013, ICML.

[21]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[22]  Ji Zhu,et al.  Efficient Computation and Model Selection for the Support Vector Regression , 2007, Neural Computation.

[23]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[24]  Sören Laue,et al.  Approximating Concavely Parameterized Optimization Problems , 2012, NIPS.

[25]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[26]  顾彬,et al.  Regularization Path for v-Support Vector Classification , 2012 .

[27]  Takafumi Kanamori,et al.  Nonparametric Conditional Density Estimation Using Piecewise-Linear Solution Path of Kernel Quantile Regression , 2009, Neural Computation.

[28]  J. Friedman.  Multivariate adaptive regression splines , 1990 .