Least squares kernel ensemble regression in Reproducing Kernel Hilbert Space

Abstract Ensemble regression method shows better performance than single regression since ensemble regression method can combine several single regression methods together to improve accuracy and stability of a single regressor. In this paper, we propose a novel kernel ensemble regression method by minimizing total least square loss in multiple Reproducing Kernel Hilbert Spaces (RKHSs). Base kernel regressors are co-optimized and weighted to form an ensemble regressor. In this way, the problem of finding suitable kernel types and their parameters in base kernel regressor is solved in the ensemble regression framework. Experimental results on several datasets, such as artificial datasets, UCI regression and classification datasets, show that our proposed approach achieves the lowest regression loss among comparative regression methods such as ridge regression, support vector regression (SVR), gradient boosting, decision tree regression and random forest.

[1]  Evgeny Burnaev,et al.  Conformalized Kernel Ridge Regression , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[2]  Jeffrey A. Fessler,et al.  Dictionary-free MRI parameter estimation via kernel ridge regression , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[3]  Olivier Sigaud,et al.  Many regression algorithms, one unified model: A review , 2015, Neural Networks.

[4]  Xuelong Li,et al.  Similarity Constraints-Based Structured Output Regression Machine: An Approach to Image Super-Resolution , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Karthikeyan Natesan Ramamurthy,et al.  Multiple kernel interpolation for inverting non-linear dimensionality reduction and dimension estimation , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Yufeng Liu,et al.  Kernel continuum regression , 2013, Comput. Stat. Data Anal..

[7]  Chao Li,et al.  Active multi-kernel domain adaptation for hyperspectral image classification , 2017, Pattern Recognit..

[8]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[9]  Chiou-Shann Fuh,et al.  Multiple Kernel Learning for Dimensionality Reduction , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[11]  Kamalika Das,et al.  Sparse inverse kernel Gaussian Process regression , 2013, Stat. Anal. Data Min..

[12]  Masahiro Yukawa,et al.  Adaptive Learning in Cartesian Product of Reproducing Kernel Hilbert Spaces , 2014, IEEE Transactions on Signal Processing.

[13]  Lorenzo Bruzzone,et al.  Multiple Kernel Learning for Remote Sensing Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[15]  Jiang Liu,et al.  Fault Prediction for Power Plant Equipment Based on Support Vector Regression , 2015, 2015 8th International Symposium on Computational Intelligence and Design (ISCID).

[16]  Qiang Zhang,et al.  Risk prediction of type II diabetes based on random forest model , 2017, 2017 Third International Conference on Advances in Electrical, Electronics, Information, Communication and Bio-Informatics (AEEICB).

[17]  Weifeng Liu,et al.  Online Laplacian-Regularized Support Vector Regression , 2017, 2017 3rd IEEE International Conference on Cybernetics (CYBCON).

[18]  Don R. Hush,et al.  An Explicit Description of the Reproducing Kernel Hilbert Spaces of Gaussian RBF Kernels , 2006, IEEE Transactions on Information Theory.

[19]  Yuh-Jyh Hu,et al.  Prediction of Patient-Controlled Analgesic Consumption: A Multimodel Regression Tree Approach , 2018, IEEE Journal of Biomedical and Health Informatics.

[20]  Tie Zhou,et al.  Ordered-Subset Ridge Regression in Image Reconstruction , 2009, 2009 2nd International Congress on Image and Signal Processing.

[21]  Junhui Wang,et al.  A unified penalized method for sparse additive quantile models: an RKHS approach , 2017 .

[22]  Oliver Kramer,et al.  Precise Wind Power Prediction with SVM Ensemble Regression , 2014, ICANN.

[23]  Vimal Bhatia,et al.  Finite dictionary techniques for MSER equalization in RKHS , 2017, Signal Image Video Process..

[24]  Joel J. P. C. Rodrigues,et al.  Predicting hypertensive disorders in high-risk pregnancy using the random forest approach , 2017, 2017 IEEE International Conference on Communications (ICC).

[25]  Jiawei Jiang,et al.  TencentBoost: A Gradient Boosting Tree System with Parameter Server , 2017, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[26]  Qiang Liu,et al.  Coupled Multiple Kernel Learning for Supervised Classification , 2017, Comput. Informatics.

[27]  Alexander J. Smola,et al.  Support Vector Regression Machines , 1996, NIPS.

[28]  Zhi-feng Wu,et al.  A regression tree approach to investigate the nonlinear relationship between land surface temperature and vegetation abundance , 2016, 2016 4th International Workshop on Earth Observation and Remote Sensing Applications (EORSA).

[29]  Yong Xu,et al.  Lasso logistic regression based approach for extracting plants coregenes responding to abiotic stresses , 2011, The Fourth International Workshop on Advanced Computational Intelligence.

[30]  Minping Qian,et al.  Pathway Detection Based on Hierarchical LASSO Regression Model , 2009, 2009 2nd International Conference on Biomedical Engineering and Informatics.

[31]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[32]  Zhao Kang,et al.  Image Projection Ridge Regression for Subspace Clustering , 2017, IEEE Signal Processing Letters.

[33]  Marion Gilson,et al.  An RKHS approach to systematic kernel selection in nonlinear system identification , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[34]  Yang Li,et al.  Radar HRRP target recognition based on Gradient Boosting Decision Tree , 2016, 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).