Sparse Universum Quadratic Surface Support Vector Machine Models for Binary Classification

In binary classification, kernel-free linear or quadratic support vector machines are proposed to avoid dealing with difficulties such as finding appropriate kernel functions or tuning their hyper-parameters. Furthermore, Universum data points, which do not belong to any class, can be exploited to embed prior knowledge into the corresponding models so that the generalization performance is improved. In this paper, we design novel kernel-free Universum quadratic surface support vector machine models. Further, we propose the L1 norm regularized version that is beneficial for detecting potential sparsity patterns in the Hessian of the quadratic surface and reducing to the standard linear models if the data points are (almost) linearly separable. The proposed models are convex such that standard numerical solvers can be utilized for solving them. Nonetheless, we formulate a least squares version of the L1 norm regularized model and next, design an effective tailored algorithm that only requires solving one linear system. Several theoretical properties of these models are then reported/proved as well. We finally conduct numerical experiments on both artificial and public benchmark data sets to demonstrate the feasibility and effectiveness of the proposed models. ∗ Corresponding Author Hossein Moosaei Department of Mathematics, Faculty of Science, University of Bojnord, Bojnord, Iran Department of Applied Mathematics, School of Computer Science, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic E-mail: hmoosaei@gmail.com, moosaei@ub.ac.ir, hmoosaei@kam.mff.cuni.cz Ahmad Mousavi Institute for Mathematics and its Applications, University of Minnesota, Minneapolis, MN 55455, USA E-mail: amousavi@umn.edu Milan Hlad́ık Department of Applied Mathematics, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic E-mail: hladik@kam.mff.cuni.cz Zheming Gao College of Information Science and Engineering, Northeastern University, Shenyang, Liaoning 110819, China E-mail: tonygaobasketball@hotmail.com ar X iv :2 10 4. 01 33 1v 1 [ cs .L G ] 3 A pr 2 02 1 2 Hossein Moosaei et al.

[1]  Ramin Ayanzadeh,et al.  A Survey on Compressive Sensing: Classical Results and Recent Advancements , 2019, ArXiv.

[2]  Robert A. Lordo,et al.  Learning from Data: Concepts, Theory, and Methods , 2001, Technometrics.

[3]  Ting Wang,et al.  Color image segmentation using pixel wise support vector machine classification , 2011, Pattern Recognit..

[4]  Yanshan Xiao,et al.  A new multi-task learning method with universum data , 2020 .

[5]  Panos M. Pardalos,et al.  An improvement on parametric $$\nu $$ν-support vector algorithm for classification , 2019, Ann. Oper. Res..

[6]  Yong Shi,et al.  Twin support vector machine with Universum data , 2012, Neural Networks.

[7]  Erdinc Akyildirim,et al.  Prediction of cryptocurrency returns using machine learning , 2020, Annals of Operations Research.

[8]  Jinglai Shen,et al.  Least Sparsity of p-Norm Based Optimization Problems with p>1 , 2017, SIAM J. Optim..

[9]  Fatemeh Bazikar,et al.  DC programming and DCA for parametric-margin ν-support vector machine , 2020, Applied Intelligence.

[10]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[11]  Panos M. Pardalos,et al.  A survey of data mining techniques applied to agriculture , 2009, Oper. Res..

[12]  Marvin M. Doyley,et al.  Finite Element Reconstruction of Stiffness Images in MR Elastography Using Statistical Physical Forward Modeling and Proximal Optimization Methods , 2021, 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI).

[13]  Domenico Ciuonzo,et al.  Learning Wireless Sensor Networks for Source Localization , 2019, Sensors.

[14]  Francis Ferraro,et al.  Event Representation with Sequential, Semi-Supervised Discrete Variables , 2020, NAACL.

[15]  Jinglai Shen,et al.  Solution uniqueness of convex piecewise affine functions based optimization with applications to constrained ℓ1 minimization , 2017, ESAIM: Control, Optimisation and Calculus of Variations.

[16]  Jan R. Magnus,et al.  The Elimination Matrix: Some Lemmas and Applications , 1980, SIAM J. Algebraic Discret. Methods.

[17]  Panos M. Pardalos,et al.  Robust chance-constrained support vector machines with second-order moment information , 2018, Ann. Oper. Res..

[18]  Bernhard Schölkopf,et al.  An Analysis of Inference with the Universum , 2007, NIPS.

[19]  Marvin M. Doyley,et al.  Ultrasound Elasticity Imaging Using Physics-Based Models and Learning-Based Plug-and-Play Priors , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  Francis Ferraro,et al.  A Discrete Variational Recurrent Topic Model without the Reparametrization Trick , 2020, NeurIPS.

[21]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[22]  Alvin Lim,et al.  Quadratic Surface Support Vector Machine with L1 Norm Regularization , 2019, Journal of Industrial & Management Optimization.

[23]  Yan-Qin Bai,et al.  Quadratic Kernel-Free Least Square Twin Support Vector Machine for Binary Classification Problems , 2019, Journal of the Operations Research Society of China.

[24]  Jian Luo,et al.  Soft Quadratic Surface Support Vector Machine for Binary Classification , 2016, Asia Pac. J. Oper. Res..

[25]  Jason Weston,et al.  Inference with the Universum , 2006, ICML.

[26]  Hua Yu,et al.  Quadratic kernel-free least squares support vector machine for target diseases classification , 2015, Journal of Combinatorial Optimization.

[27]  Marvin M. Doyley,et al.  A statistical framework for model-based inverse problems in ultrasound elastography , 2020, ArXiv.

[28]  Roohallah Alizadehsani,et al.  Computer aided decision making for heart disease detection using hybrid neural network-Genetic algorithm , 2017, Comput. Methods Programs Biomed..

[29]  Jinglai Shen,et al.  Exact Support and Vector Recovery of Constrained Sparse Vectors via Constrained Matching Pursuit , 2019, 1903.07236.

[30]  William Stafiord Noble,et al.  Support vector machine applications in computational biology , 2004 .

[31]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[32]  K. Chou,et al.  Application of SVM to predict membrane protein types. , 2004, Journal of theoretical biology.

[33]  Jian Luo,et al.  A kernel-free double well potential support vector machine with applications , 2021, Eur. J. Oper. Res..

[34]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[35]  Jian Luo,et al.  Benchmarking robustness of load forecasting models under data integrity attacks , 2018 .

[36]  Issam Dagher,et al.  Quadratic kernel-free non-linear support vector machine , 2008, J. Glob. Optim..