Recent advances on support vector machines research

Abstract Support vector machines (SVMs), with their roots in Statistical Learning Theory (SLT) and optimization methods, have become powerful tools for problem solution in machine learning. SVMs reduce most machine learning problems to optimization problems and optimization lies at the heart of SVMs. Lots of SVM algorithms involve solving not only convex problems, such as linear programming, quadratic programming, second order cone programming, semi-definite programming, but also non-convex and more general optimization problems, such as integer programming, semi-infinite programming, bi-level programming and so on. The purpose of this paper is to understand SVM from the optimization point of view, review several representative optimization models in SVMs, their applications in economics, in order to promote the research interests in both optimization-based SVMs theory and economics applications. This paper starts with summarizing and explaining the nature of SVMs. It then proceeds to discuss optimization...

[1]  Andreu Català,et al.  K-SVCR. A Multi-class Support Vector Machine , 2000, ECML.

[2]  V. Vapnik.  Estimation of Dependences Based on Empirical Data , 2006 .

[3]  Bart Baesens,et al.  Rule Extraction from Support Vector Machines: An Overview of Issues and Application in Credit Scoring , 2008, Rule Extraction from Support Vector Machines.

[4]  Jing Hu,et al.  Model Selection via Bilevel Optimization , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[5]  Chih-Jen Lin,et al.  Coordinate Descent Method for Large-scale L2-loss Linear Support Vector Machines , 2008, J. Mach. Learn. Res..

[6]  Ulf Brefeld,et al.  {AUC} maximizing support vector learning , 2005 .

[7]  Kin Keung Lai,et al.  Credit Scoring Models with AUC Maximization Based on Weighted SVM , 2009, Int. J. Inf. Technol. Decis. Mak..

[8]  M. A. López-Cerdá,et al.  Linear Semi-Infinite Optimization , 1998 .

[9]  Jing Hu,et al.  Bilevel Model Selection for Support Vector Machines , 2007 .

[10]  David C. Gibbon,et al.  Support vector machines: relevance feedback and information retrieval , 2002, Inf. Process. Manag..

[11]  Yingjie Tian,et al.  Unsupervised and Semi-Supervised Two-class Support Vector Machines , 2006, ICDM Workshops.

[12]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[13]  Krzysztof Kochanek,et al.  The environmental risk assessment for decision support system for water management in the vicinity of open cast mines (DS WMVOC) , 2010 .

[14]  J. Weston,et al.  Support vector density estimation , 1999 .

[15]  Grigorios Tsoumakas,et al.  Mining Multi-label Data , 2010, Data Mining and Knowledge Discovery Handbook.

[16]  Ingoo Han,et al.  Hybrid genetic algorithms and support vector machines for bankruptcy prediction , 2006, Expert Syst. Appl..

[17]  Stephen Kwek,et al.  Applying Support Vector Machines to Imbalanced Datasets , 2004, ECML.

[18]  Ulf Brefeld,et al.  Co-EM support vector learning , 2004, ICML.

[19]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[20]  Lean Yu,et al.  A Modified Least Squares Support Vector Machine Classifier with Application to Credit Risk Analysis , 2009, Int. J. Inf. Technol. Decis. Mak..

[21]  Kin Keung Lai,et al.  A new fuzzy support vector machine to evaluate credit risk , 2005, IEEE Transactions on Fuzzy Systems.

[22]  Yingjie Tian,et al.  Leave-one-out bounds for support vector ordinal regression machine , 2008, Neural Computing and Applications.

[23]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[24]  S. Nash,et al.  Linear and Nonlinear Programming , 1987 .

[25]  Bernhard Schölkopf,et al.  Bounds on Error Expectation for SVM , 2000 .

[26]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Koby Crammer,et al.  On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines , 2002, J. Mach. Learn. Res..

[28]  Robert J. Vanderbei,et al.  Linear Programming: Foundations and Extensions , 1998, Kluwer international series in operations research and management service.

[29]  Klaus Obermayer,et al.  Support vector learning for ordinal regression , 1999 .

[30]  Nicola Ancona,et al.  Goal detection in football by using support vector machines for classification , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[31]  Hsuan-Tien Lin,et al.  A note on Platt’s probabilistic outputs for support vector machines , 2007, Machine Learning.

[32]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[33]  Ying Jie Tian,et al.  Lp-norm proximal support vector machine and its applications , 2010, ICCS.

[35]  Francis Eng Hock Tay,et al.  Support vector machine with adaptive parameters in financial time series forecasting , 2003, IEEE Trans. Neural Networks.

[36]  Juwei Lu,et al.  Face recognition using feature optimization and /spl nu/-support vector learning , 2001, Neural Networks for Signal Processing XI: Proceedings of the 2001 IEEE Signal Processing Society Workshop (IEEE Cat. No.01TH8584).

[37]  John Shawe-Taylor,et al.  Two view learning: SVM-2K, Theory and Practice , 2005, NIPS.

[38]  Karsten M. Borgwardt,et al.  Kernel Methods in Bioinformatics , 2011, Handbook of Statistical Bioinformatics.

[39]  Thorsten Joachims,et al.  Training linear SVMs in linear time , 2006, KDD '06.

[40]  Harry Wechsler,et al.  Mixture of experts for classification of gender, ethnic origin, and pose of human faces , 2000, IEEE Trans. Neural Networks Learn. Syst..

[41]  Smaranda Stoenescu Cimpoeru,et al.  Neural networks and their application in credit risk assessment. Evidence from the Romanian market , 2011 .

[42]  M. Bajec,et al.  Method for selection of motor insurance fraud management system components based on business performance , 2011 .

[43]  Mohamed Cheriet,et al.  Model selection for the LS-SVM. Application to handwriting recognition , 2009, Pattern Recognit..

[44]  Ruoming Jin,et al.  Fast and exact out-of-core k-means clustering , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[45]  Andreu Català,et al.  Rule extraction from support vector machines , 2002, ESANN.

[46]  Donald Goldfarb,et al.  Robust convex quadratically constrained programs , 2003, Math. Program..

[47]  Kyoung-jae Kim,et al.  Financial time series forecasting using support vector machines , 2003, Neurocomputing.

[48]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[49]  M.R. Azimi-Sadjadi,et al.  Cloud classification using support vector machines , 2000, IGARSS 2000. IEEE 2000 International Geoscience and Remote Sensing Symposium. Taking the Pulse of the Planet: The Role of Remote Sensing in Managing the Environment. Proceedings (Cat. No.00CH37120).

[50]  Glenn Fung,et al.  Rule extraction from linear support vector machines , 2005, KDD '05.

[51]  Kin Keung Lai,et al.  Forecasting Foreign Exchange Rates With Artificial Neural Networks: A Review , 2004, Int. J. Inf. Technol. Decis. Mak..

[52]  Mark A. Randolph,et al.  A support vector machines-based rejection technique for speech recognition , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[53]  Marimuthu Palaniswami,et al.  Selecting bankruptcy predictors using a support vector machine approach , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[54]  Glenn Fung,et al.  Knowledge-Based Support Vector Machine Classifiers , 2002, NIPS.

[55]  Zhengxin Chen,et al.  Classifying Credit Card Accounts for Business Intelligence and Decision Making: a Multiple-criteria Quadratic Programming Approach , 2005, Int. J. Inf. Technol. Decis. Mak..

[56]  David Haussler,et al.  Probabilistic kernel regression models , 1999, AISTATS.

[57]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[59]  Arthur Gretton,et al.  Estimating the Leave-One-Out Error for Classification Learning with SVMs , 2001 .

[60]  Kristin P. Bennett,et al.  The Interplay of Optimization and Machine Learning Research , 2006, J. Mach. Learn. Res..

[61]  Jason Weston.  Leave-One-Out Support Vector Machines , 1999, IJCAI.

[62]  Paul S. Bradley,et al.  Feature Selection via Concave Minimization and Support Vector Machines , 1998, ICML.

[63]  Young-Chan Lee,et al.  Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters , 2005, Expert Syst. Appl..

[64]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[65]  Bernhard Schölkopf,et al.  Handbook of Statistical Bioinformatics , 2011 .

[66]  Chunhua Zhang,et al.  Rule Extraction from Support Vector Machines and Its Applications , 2011, 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[67]  Francis Eng Hock Tay,et al.  Financial Forecasting Using Support Vector Machines , 2001, Neural Computing & Applications.

[68]  Chih-Jen Lin,et al.  A sequential dual method for large scale multi-class linear svms , 2008, KDD.

[69]  Donald Goldfarb,et al.  Second-order cone programming , 2003, Math. Program..

[70]  Yingjie Tian,et al.  Leave-one-out Bounds for Support Vector Regression , 2005, International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC'06).

[71]  Andrew H. Sung,et al.  Intrusion detection using neural networks and support vector machines , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[72]  H. Zou,et al.  The F ∞ -norm support vector machine , 2008 .

[73]  Yuan-Hai Shao,et al.  Improvements on Twin Support Vector Machines , 2011, IEEE Transactions on Neural Networks.

[74]  Olvi L. Mangasarian,et al.  Nonlinear Knowledge-Based Classification , 2008, IEEE Transactions on Neural Networks.

[75]  Paul S. Bradley,et al.  Feature Selection via Mathematical Programming , 1997, INFORMS J. Comput..

[76]  Yuan Yao,et al.  Combining flat and structured representations for fingerprint classification with recursive neural networks and support vector machines , 2003, Pattern Recognit..

[77]  Qi Tian,et al.  Update relevant image weights for content-based image retrieval using support vector machines , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[78]  JIANPING LI,et al.  Feature Selection via Least Squares Support Feature Machine , 2007, Int. J. Inf. Technol. Decis. Mak..

[79]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[80]  Anastasios Tefas,et al.  Using Support Vector Machines to Enhance the Performance of Elastic Graph Matching for Frontal Face Authentication , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[81]  Masao Fukushima,et al.  Second-Order Cone Programming Formulations for Robust Multiclass Classification , 2007, Neural Computation.

[82]  Glenn Fung,et al.  Knowledge-Based Nonlinear Kernel Classifiers , 2003, COLT.

[83]  PerlovskyLeonid.  2009 Special Issue , 2009 .

[84]  Glenn Fung,et al.  Proximal support vector machine classifiers , 2001, KDD '01.

[85]  Zhengxin Chen,et al.  A Descriptive Framework for the Field of Data Mining and Knowledge Discovery , 2008, Int. J. Inf. Technol. Decis. Mak..

[86]  Ming-Wei Chang,et al.  Leave-One-Out Bounds for Support Vector Regression Model Selection , 2005, Neural Computation.

[87]  Nello Cristianini,et al.  Classification using String Kernels , 2000 .

[88]  Grigorios Tsoumakas,et al.  Multi-Label Classification: An Overview , 2007, Int. J. Data Warehous. Min..

[89]  Enrico Blanzieri,et al.  Fast Local Support Vector Machines for Large Datasets , 2009, MLDM.

[90]  Yingjie Tian,et al.  A Multi-class Classification Algorithm based on Ordinal Regression Machine , 2005 .

[91]  Jiri Matas,et al.  Support vector machines for face authentication , 2002, Image Vis. Comput..

[92]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[93]  Vladimir Vapnik,et al.  A new learning paradigm: Learning using privileged information , 2009, Neural Networks.

[94]  Thomas Hofmann,et al.  Large Margin Methods for Structured and Interdependent Output Variables , 2005, J. Mach. Learn. Res..

[95]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.

[96]  Joseph Picone,et al.  Applications of support vector machines to speech recognition , 2004, IEEE Transactions on Signal Processing.

[97]  Mu-Chen Chen,et al.  Credit scoring with a data mining approach based on support vector machines , 2007, Expert Syst. Appl..

[98]  Etienne de Klerk,et al.  Aspects of Semidefinite Programming , 2002 .

[99]  Eamonn J. Keogh,et al.  Curse of Dimensionality , 2010, Encyclopedia of Machine Learning.

[100]  Ralf Herbrich,et al.  Learning Kernel Classifiers: Theory and Algorithms , 2001 .

[101]  Xin Zhou,et al.  MSVM-RFE: extensions of SVM-RFE for multiclass gene selection on DNA microarray data , 2007, Bioinform..

[102]  Edward W. Wild,et al.  Multiple Instance Classification via Successive Linear Programming , 2008 .

[103]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[104]  Jun Yu,et al.  lp-norm support vector machine with CCCP , 2010, 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery.

[105]  Francis Eng Hock Tay,et al.  Modified support vector machines in financial time series forecasting , 2002, Neurocomputing.

[106]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[107]  B. Baesens,et al.  A support vector machine approach to credit scoring , 2003 .

[108]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[109]  F. Tay,et al.  Application of support vector machines in financial time series forecasting , 2001 .

[110]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[111]  Gunnar Rätsch,et al.  Large Scale Multiple Kernel Learning , 2006, J. Mach. Learn. Res..

[112]  Vladimir Vapnik,et al.  Estimation of Dependences Based on Empirical Data: Empirical Inference Science (Information Science and Statistics) , 2006 .

[113]  Cheng Soon Ong,et al.  mGene: accurate SVM-based gene finding with an application to nematode genomes. , 2009, Genome research.

[114]  Bart Baesens,et al.  Risk Management and Regulatory Compliance: A Data Mining Framework Based on Neural Network Rule Extraction , 2006, ICIS.

[115]  Sayan Mukherjee,et al.  Feature Selection for SVMs , 2000, NIPS.

[116]  Sheng-De Wang,et al.  Fuzzy support vector machines , 2002, IEEE Trans. Neural Networks.

[117]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[118]  Kyung-shik Shin,et al.  An application of support vector machines in bankruptcy prediction model , 2005, Expert Syst. Appl..

[119]  Cengiz Kahraman,et al.  A fuzzy approach to e-banking website quality assessment based on an integrated AHP-ELECTRE method , 2011 .

[120]  Chih-Jen Lin,et al.  A dual coordinate descent method for large-scale linear SVM , 2008, ICML '08.

[121]  Xindong Wu,et al.  10 Challenging Problems in Data Mining Research , 2006, Int. J. Inf. Technol. Decis. Mak..

[122]  Luca Zanni,et al.  A parallel solver for large quadratic programs in training support vector machines , 2003, Parallel Comput..

[123]  Francis Eng Hock Tay,et al.  Improved financial time series forecasting by combining Support Vector Machines with self-organizing feature map , 2001, Intell. Data Anal..

[124]  David Haussler,et al.  Exploiting Generative Models in Discriminative Classifiers , 1998, NIPS.

[125]  David J. Hand,et al.  A survey of the issues in consumer credit modelling research , 2005, J. Oper. Res. Soc..

[126]  Honggang Wang,et al.  Empirical Evaluation of Classifiers for Software Risk Management , 2009, Int. J. Inf. Technol. Decis. Mak..

[127]  Alexander J. Smola,et al.  Advances in Large Margin Classifiers , 2000 .

[128]  Dale Schuurmans,et al.  Unsupervised and Semi-Supervised Multi-Class Support Vector Machines , 2005, AAAI.

[129]  Nai-Yang Deng,et al.  Support Vector Machines: Optimization Based Theory, Algorithms, and Extensions , 2012 .

[130]  Yiming Ying,et al.  Multi-kernel regularized classifiers , 2007, J. Complex..

[131]  Kun Zhao,et al.  Unsupervised and Semi-supervised Lagrangian Support Vector Machines with Polyhedral Perturbations , 2007, 2009 Third International Symposium on Intelligent Information Technology Application.

[132]  Chunhua Zhang,et al.  The new interpretation of support vector machines on statistical learning theory , 2010 .