Abstract — Among neural models the Support Vector Machine (SVM) solutions are attracting increasing attention, mostly because they eliminate certain crucial questions involved by neural network construction. The main drawback of standard SVM is its high computational complexity, therefore recently a new technique, the Least Squares SVM (LS–SVM) has been introduced. In this paper we present an extended view of the Least Squares Support Vector Regression (LS–SVR), which enables us to develop new formulations and algorithms to this regression technique. Based on manipulating the linear equation set -which embodies all information about the regression in the learning process- some new methods are introduced to simplify the formulations, speed up the calculations and/or provide better results. Keywords — Function estimation, Least–Squares Support Vector Machines, Regression, System Modeling I. I NTRODUCTION HIS paper focuses on the Least Squares version of SVM [1], the LS–SVM [2], whose main advantage is that it is computationally more efficient than the standard SVM method. In this case training requires the solution of a linear equation set instead of the long and computationally hard quadratic programming problem involved by the standard SVM. It must also be emphasized that LS–SVM is closely related to Gaussian processes and regularization networks in that the obtained linear systems are equivalent [2]. While the least squares version incorporates all training data in the network to produce the result, the traditional SVM selects some of them (the support vectors) that are important in the regression. The use of only a subset of all vectors is a desirable property of SVMs, because it provides additional information regarding the training data and concludes in a more effective solution formulating a smaller network. Similarly to the sparseness of a traditional SVM, sparse LS-SVM solutions can also be reached by applying pruning methods [3][4]. Unfortunately if this LS–SVM pruning is applied, the performance declines proportionally to the eliminated training samples, since the information (input-output relation) they described is lost. Another problem is that sparseness can be reached by iterative methods, which
[1]
R. Shah,et al.
Least Squares Support Vector Machines
,
2022
.
[2]
Anthony Widjaja,et al.
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
,
2003,
IEEE Transactions on Neural Networks.
[3]
Yuh-Jye Lee,et al.
RSVM: Reduced Support Vector Machines
,
2001,
SDM.
[4]
Gunnar Rätsch,et al.
Input space versus feature space in kernel-based methods
,
1999,
IEEE Trans. Neural Networks.
[5]
William H. Press,et al.
Numerical recipes in C
,
2002
.
[6]
G. Baudat,et al.
Kernel-based methods and function approximation
,
2001,
IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).
[7]
P. Holland,et al.
Robust regression using iteratively reweighted least-squares
,
1977
.
[8]
Vladimir N. Vapnik,et al.
The Nature of Statistical Learning Theory
,
2000,
Statistics for Engineering and Information Science.
[9]
Johan A. K. Suykens,et al.
Sparse approximation using least squares support vector machines
,
2000,
2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).
[10]
G. Horvath,et al.
A sparse least squares support vector machine classifier
,
2004,
2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).
[11]
Gene H. Golub,et al.
Matrix computations
,
1983
.
[12]
G. Horváth,et al.
A generalised LS-SVM
,
2003
.
[13]
B. Ripley,et al.
Robust Statistics
,
2018,
Encyclopedia of Mathematical Geosciences.
[14]
Johan A. K. Suykens,et al.
Weighted least squares support vector machines: robustness and sparse approximation
,
2002,
Neurocomputing.
[15]
Johan A. K. Suykens,et al.
Sparse least squares Support Vector Machine classifiers
,
2000,
ESANN.
[16]
Vladimir Cherkassky,et al.
The Nature Of Statistical Learning Theory
,
1997,
IEEE Trans. Neural Networks.