Model optimizing and feature selecting for support vector regression in time series forecasting

In this paper, the problem of optimizing SVR automatically for time series forecasting is considered, which involves introducing auto-adaptive parameters C"i and @e"i to depict the non-uniform distribution of the information offered by the training data, developing multiple kernel function K"@s to rescale different attributes of input space, optimizing all the parameters involved simultaneously with genetic algorithm and performing feature selection to reduce the redundant information. Experimental results assess the feasibility of our approach (called Model-optimizing SVR or briefly MO-SVR) and demonstrate that our method is a promising alternative for time series forecasting.

[1]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[2]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[3]  Bernhard Schölkopf,et al.  From Regularization Operators to Support Vector Kernels , 1997, NIPS.

[4]  J.T. Kwok,et al.  Linear Dependency betweenand the Input Noise in -Support Vector Regression , 2001 .

[5]  Zhizhong Wang,et al.  Optimized Local Kernel Machines for Fast Time Series Forecasting , 2007, Third International Conference on Natural Computation (ICNC 2007).

[6]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[7]  Klaus Obermayer,et al.  Support vector learning for ordinal regression , 1999 .

[8]  Francis Eng Hock Tay,et al.  ε-Descending Support Vector Machines for Financial Time Series Forecasting , 2002, Neural Processing Letters.

[9]  Alessandro Sperduti,et al.  Multiclass Classification with Multi-Prototype Support Vector Machines , 2005, J. Mach. Learn. Res..

[10]  Korris Fu-Lai Chung,et al.  Theoretically Optimal Parameter Choices for Support Vector Regression Machines with Noisy Input , 2005, Soft Comput..

[11]  K. Chellapilla,et al.  Optimization of bilinear time series models using fast evolutionary programming , 1998, IEEE Signal Processing Letters.

[12]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[13]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[14]  F. Tay,et al.  Application of support vector machines in financial time series forecasting , 2001 .

[15]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[16]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[17]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[18]  Gunnar Rätsch,et al.  Predicting Time Series with Support Vector Machines , 1997, ICANN.

[19]  Kyoung-jae Kim,et al.  Financial time series forecasting using support vector machines , 2003, Neurocomputing.

[20]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[21]  Lutgarde M. C. Buydens,et al.  Using support vector machines for time series prediction , 2003 .

[22]  Léon Bottou,et al.  Local Learning Algorithms , 1992, Neural Computation.

[23]  Huan Liu,et al.  Efficient Feature Selection via Analysis of Relevance and Redundancy , 2004, J. Mach. Learn. Res..

[24]  Andrew R. Webb,et al.  Statistical Pattern Recognition , 1999 .

[25]  Wei Chu,et al.  Bayesian support vector regression using a unified loss function , 2004, IEEE Transactions on Neural Networks.

[26]  Francis X. Diebold,et al.  Elements of Forecasting , 1997 .

[27]  L. Buydens,et al.  Determination of optimal support vector regression parameters by genetic algorithms and simplex optimization , 2005 .

[28]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[29]  Shie Mannor,et al.  Sparse Online Greedy Support Vector Regression , 2002, ECML.

[30]  Shun-ichi Amari,et al.  Network information criterion-determining the number of hidden units for an artificial neural network model , 1994, IEEE Trans. Neural Networks.

[31]  H. Akaike.  A new look at the statistical model identification , 1974 .

[32]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[33]  Matthias W. Seeger,et al.  Bayesian Model Selection for Support Vector Machines, Gaussian Processes and Other Kernel Classifiers , 1999, NIPS.

[34]  Tapio Elomaa,et al.  Proceedings of the 13th European Conference on Machine Learning , 2002 .

[35]  Francis Eng Hock Tay,et al.  Modified support vector machines in financial time series forecasting , 2002, Neurocomputing.

[36]  Tao Xiong,et al.  A combined SVM and LDA approach for classification , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[37]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[38]  G. Cawley.  Model Selection for Support Vector Machines via Adaptive Step-Size Tabu Search , 2001 .

[39]  Wenwu He,et al.  Fast Forecasting with Simplified Kernel Regression Machines , 2007 .

[40]  Dan A. Simovici,et al.  On feature selection through clustering , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[41]  Carlos Soares,et al.  A Meta-Learning Method to Select the Kernel Width in Support Vector Regression , 2004, Machine Learning.

[42]  V. Vapnik,et al.  Bounds on Error Expectation for Support Vector Machines , 2000, Neural Computation.

[43]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[44]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[45]  D. E. Goldberg,et al.  Genetic Algorithm in Search , 1989 .

[46]  Ju-Jang Lee,et al.  Heterogeneous local model networks for time series prediction , 2005, Appl. Math. Comput..

[47]  Chih-Hung Wu,et al.  Dynamically Optimizing Parameters in Support Vector Regression: An Application of Electricity Load Forecasting , 2006, Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06).

[48]  Thomas P. Trappenberg,et al.  A Heuristic for Free Parameter Optimization with Support Vector Machines , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[49]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.