A PSO and pattern search based memetic algorithm for SVMs parameters optimization

Abstract Addressing the issue of SVMs parameters optimization, this study proposes an efficient memetic algorithm based on particle swarm optimization algorithm (PSO) and pattern search (PS). In the proposed memetic algorithm, PSO is responsible for exploration of the search space and the detection of the potential regions with optimum solutions, while pattern search (PS) is used to produce an effective exploitation on the potential regions obtained by PSO. Moreover, a novel probabilistic selection strategy is proposed to select the appropriate individuals among the current population to undergo local refinement, keeping a well balance between exploration and exploitation. Experimental results confirm that the local refinement with PS and our proposed selection strategy are effective, and finally demonstrate the effectiveness and robustness of the proposed PSO-PS based MA for SVMs parameters optimization.

[1]  F. Imbault,et al.  A stochastic optimization approach for parameter tuning of support vector machines , 2004, ICPR 2004.

[2]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[3]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[4]  Bo Liu,et al.  An Effective PSO-Based Memetic Algorithm for Flow Shop Scheduling , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Jing Tang,et al.  Diversity-adaptive parallel memetic algorithm for solving large scale combinatorial optimization problems , 2006, Soft Comput..

[6]  Xin Yao,et al.  A Memetic Algorithm for VLSI Floorplanning , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[7]  Kristin P. Bennett,et al.  Model selection for primal SVM , 2011, Machine Learning.

[8]  Kay Chen Tan,et al.  A Multi-Facet Survey on Memetic Computation , 2011, IEEE Transactions on Evolutionary Computation.

[9]  Robert Sabourin,et al.  A dynamic model selection strategy for support vector machine classifiers , 2012, Appl. Soft Comput..

[10]  ZitzlerE.,et al.  Multiobjective evolutionary algorithms , 1999 .

[11]  M Reyes Sierra,et al.  Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art , 2006 .

[12]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[13]  Mohamed Cheriet,et al.  Optimizing resources in model selection for support vector machines , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[14]  Pablo Moscato,et al.  On Evolution, Search, Optimization, Genetic Algorithms and Martial Arts : Towards Memetic Algorithms , 1989 .

[15]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Combining meta-learning and search techniques to select parameters for support vector machines , 2012, Neurocomputing.

[16]  Carlos A. Coello Coello,et al.  HCS: A New Local Search Strategy for Memetic Multiobjective Evolutionary Algorithms , 2010, IEEE Transactions on Evolutionary Computation.

[17]  Shutao Li,et al.  Tuning SVM parameters by using a hybrid CLPSO-BFGS algorithm , 2010, Neurocomputing.

[18]  Xiaoli Zhang,et al.  An ACO-based algorithm for parameter optimization of support vector machines , 2010, Expert Syst. Appl..

[19]  Shawki Areibi,et al.  Strength Pareto Particle Swarm Optimization and Hybrid EA-PSO for Multi-Objective Optimization , 2010, Evolutionary Computation.

[20]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[21]  Wei-Chun Kao,et al.  Radius Margin Bounds for Support Vector . . . , 2003 .

[22]  Zhen Yang,et al.  Effective Memetic Algorithms for VLSI Design = Genetic Algorithms Local Search Multi-Level Clustering , 2004, Evolutionary Computation.

[23]  Man Ieee Systems,et al.  IEEE transactions on systems, man and cybernetics. Part B, Cybernetics , 1996 .

[24]  S. Sathiya Keerthi,et al.  Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms , 2002, IEEE Trans. Neural Networks.

[25]  Lei Wang,et al.  Two Criteria for Model Selection in Multiclass Support Vector Machines , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Robert Michael Lewis,et al.  On the Local Convergence of Pattern Search , 2003, SIAM J. Optim..

[27]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[28]  CherietMohamed,et al.  Optimizing resources in model selection for support vector machine , 2007 .

[29]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[30]  Xiaohui Hu,et al.  Engineering optimization with particle swarm , 2003, Proceedings of the 2003 IEEE Swarm Intelligence Symposium. SIS'03 (Cat. No.03EX706).

[31]  Hugo Jair Escalante,et al.  Particle Swarm Model Selection , 2009, J. Mach. Learn. Res..

[32]  Mohamed Cheriet,et al.  Model selection for the LS-SVM. Application to handwriting recognition , 2009, Pattern Recognit..

[33]  Mathias M. Adankon,et al.  Optimizing resources in model selection for support vector machines , 2005 .

[34]  Chih-Jen Lin,et al.  Radius Margin Bounds for Support Vector Machines with the RBF Kernel , 2002, Neural Computation.

[35]  Yew-Soon Ong,et al.  A Probabilistic Memetic Framework , 2009, IEEE Transactions on Evolutionary Computation.

[36]  Josef Kittler,et al.  Proceedings of the 4th International Conference on Pattern Recognition , 1988 .

[37]  S. Sathiya Keerthi,et al.  Evaluation of simple performance measures for tuning SVM hyperparameters , 2003, Neurocomputing.

[38]  Wu Meng,et al.  Application of Support Vector Machines in Financial Time Series Forecasting , 2007 .

[39]  X. C. Guo,et al.  A novel LS-SVMs hyper-parameter selection based on particle swarm optimization , 2008, Neurocomputing.

[40]  Francisco Herrera,et al.  Real-Coded Memetic Algorithms with Crossover Hill-Climbing , 2004, Evolutionary Computation.

[41]  Mauro Birattari,et al.  Swarm Intelligence , 2012, Lecture Notes in Computer Science.

[42]  Deyu Meng,et al.  Fast and Efficient Strategies for Model Selection of Gaussian Support Vector Machine , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[43]  Ching Y. Suen,et al.  Automatic model selection for the optimization of SVM kernels , 2005, Pattern Recognit..

[44]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[45]  Carlos Cotta,et al.  Memetic algorithms and memetic computing optimization: A literature review , 2012, Swarm Evol. Comput..