Selection of Import Vectors via Binary Particle Swarm Optimization and Cross-Validation for Kernel Logistic Regression

Kernel logistic regression (KLR) is a powerful discriminative algorithm. It has similar loss function and algorithmic structure to the kernel support vector machine (SVM). Recently, Zhu and Hastie proposed the import vector machine (IVM) in which a subset of the input vectors of KLR are selected by minimizing the regularized negative log-likelihood to improve the generalization performance and to reduce computation cost. In this paper, two modifications of the original IVM are proposed. The cross-validation based criterion is used to select import vectors instead of the likelihood based criterion. Also binary particle swarm optimization is used to select good subset instead of the greedy stepwise algorithm of the original IVM. Through the comparison experiment, the improvement of the generalization performance of the proposed algorithm was confirmed.

[1]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[2]  D. Agrafiotis,et al.  Feature selection for structure-activity correlation using binary particle swarms. , 2002, Journal of medicinal chemistry.

[3]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[4]  Gavin C. Cawley,et al.  Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers , 2003, Pattern Recognit..

[5]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[6]  Erik D. Goodman,et al.  Swarmed feature selection , 2004, 33rd Applied Imagery Pattern Recognition Workshop (AIPR'04).

[7]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[8]  David Haussler,et al.  Probabilistic kernel regression models , 1999, AISTATS.

[9]  H. Akaike.  A new look at the statistical model identification , 1974 .

[10]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[11]  S. Sathiya Keerthi,et al.  A Fast Dual Algorithm for Kernel Logistic Regression , 2002, 2007 International Joint Conference on Neural Networks.

[12]  Kurita Takio,et al.  Tuning Hyperparameters of Support Vector Machines using Particle Swarm Optimization , 2007 .

[13]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[14]  Ji Zhu,et al.  Kernel Logistic Regression and the Import Vector Machine , 2001, NIPS.

[15]  Gavin C. Cawley,et al.  Efficient model selection for kernel logistic regression , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[16]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[17]  Russell C. Eberhart,et al.  A discrete binary version of the particle swarm algorithm , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.