Robust generalized eigenvalue classifier with ellipsoidal uncertainty

Uncertainty is a concept associated with data acquisition and analysis, usually appearing in the form of noise or measure error, often due to some technological constraint. In supervised learning, uncertainty affects classification accuracy and yields low quality solutions. For this reason, it is essential to develop machine learning algorithms able to handle efficiently data with imprecision. In this paper we study this problem from a robust optimization perspective. We consider a supervised learning algorithm based on generalized eigenvalues and we provide a robust counterpart formulation and solution in case of ellipsoidal uncertainty sets. We demonstrate the performance of the proposed robust scheme on artificial and benchmark datasets from University of California Irvine (UCI) machine learning repository and we compare results against a robust implementation of Support Vector Machines.

[1]  Panos M. Pardalos,et al.  Robust Data Mining , 2012 .

[2]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[3]  Panos M. Pardalos,et al.  A classification method based on generalized eigenvalue problems , 2007, Optim. Methods Softw..

[4]  Shie Mannor,et al.  Robustness and Regularization of Support Vector Machines , 2008, J. Mach. Learn. Res..

[5]  E. B. Andersen,et al.  Information Science and Statistics , 1986 .

[6]  Laurent El Ghaoui,et al.  Robust Solutions to Least-Squares Problems with Uncertain Data , 1997, SIAM J. Matrix Anal. Appl..

[7]  Wenjie Hu,et al.  Robust support vector machine with bullet hole image classification , 2002 .

[8]  Constantine Caramanis,et al.  Theory and Applications of Robust Optimization , 2010, SIAM Rev..

[9]  Stephen J. Wright,et al.  Optimization for Machine Learning , 2013 .

[10]  Li Xiao-wen A Classification Method Based on Spatial Information , 2003 .

[11]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[12]  Stephen P. Boyd,et al.  Robust Fisher Discriminant Analysis , 2005, NIPS.

[13]  Zhang Liu,et al.  Interior-point methods for large-scale cone programming , 2011 .

[14]  Richard S. Johannes,et al.  Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus , 1988 .

[15]  Arkadi Nemirovski,et al.  Robust solutions of Linear Programming problems contaminated with uncertain data , 2000, Math. Program..

[16]  Mario Rosario Guarracino,et al.  Multiclass Generalized Eigenvalue Proximal Support Vector Machines , 2010, 2010 International Conference on Complex, Intelligent and Software Intensive Systems.

[17]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Laurent El Ghaoui,et al.  Robust Solutions to Uncertain Semidefinite Programs , 1998, SIAM J. Optim..

[19]  R. Fisher THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[20]  Panos M. Pardalos,et al.  Incremental Classification with Generalized Eigenvalues , 2007, J. Classif..

[21]  Arkadi Nemirovski,et al.  Robust solutions of uncertain linear programs , 1999, Oper. Res. Lett..

[22]  Alexander J. Smola,et al.  Second Order Cone Programming Approaches for Handling Missing and Uncertain Data , 2006, J. Mach. Learn. Res..

[23]  Theodore B. Trafalis,et al.  Robust support vector machines for classification and computational issues , 2007, Optim. Methods Softw..

[24]  Stephen P. Boyd,et al.  A minimax theorem with applications to machine learning, signal processing, and finance , 2007, CDC.

[25]  Arkadi Nemirovski,et al.  Robust optimization – methodology and applications , 2002, Math. Program..

[26]  Shie Mannor,et al.  Robust Regression and Lasso , 2008, IEEE Transactions on Information Theory.

[27]  Cédric Richard,et al.  Incorporating prior information into Support Vector Machines in the form of ellipsoidal knowledge sets , 2006, 2006 14th European Signal Processing Conference.

[28]  Melvyn Sim,et al.  The Price of Robustness , 2004, Oper. Res..

[29]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[30]  Mia Hubert,et al.  ROBPCA: A New Approach to Robust Principal Component Analysis , 2005, Technometrics.

[31]  Theodore B. Trafalis,et al.  Robust classification and regression using support vector machines , 2006, Eur. J. Oper. Res..

[32]  Zhi-Quan Luo,et al.  Robust adaptive beamforming using worst-case SINR optimization: a new diagonal loading-type solution for general-rank signal models , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[33]  A.B. Gershman,et al.  Robust adaptive beamforming using worst-case performance optimization , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[34]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[35]  Arkadi Nemirovski,et al.  Robust Convex Optimization , 1998, Math. Oper. Res..

[36]  H. Vincent On Minimax Robustness: A General Approach and Applications , 1984 .

[37]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[38]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.