Self-adaptive robust nonlinear regression for unknown noise via mixture of Gaussians

For most regression problems, the optimal regression model can be obtained by minimizing a loss function, and the selection of loss functions has great effect on the performance of the derived regression model. Squared loss is widely used in regression. It is theoretically optimal for Gaussian noise. However, real data are usually polluted by complex and unknown noise, especially in the era of big data, the noise may not be fitted well by any single distribution. To address the above problem, two novel nonlinear regression models for single-task and multi-task problems are developed in this work, where the noise is fitted by Mixture of Gaussians. It was proved that any continuous distributions can be approximated by Mixture of Gaussians. To obtain the optimal parameters in the proposed models, an iterative algorithm based on Expectation Maximization is designed. The proposed models turn to be a self-adaptive robust nonlinear regression models. The experimental results on synthetic and real-world benchmark datasets show that the proposed models produce good performance compared with current regression algorithms and provide superior robustness.

[1]  Marina Thottan,et al.  Anomaly detection in IP networks , 2003, IEEE Trans. Signal Process..

[2]  Ping-Feng Pai,et al.  Solar power output forecasting using evolutionary seasonal decomposition least-square support vector regression , 2016 .

[3]  R. Brereton,et al.  Crucial problems in regression modelling and their solutions. , 2002, The Analyst.

[4]  Xindong Wu,et al.  The Top Ten Algorithms in Data Mining , 2009 .

[5]  Qinghua Hu,et al.  Short-Term Wind Speed or Power Forecasting With Heteroscedastic Support Vector Regression , 2016, IEEE Transactions on Sustainable Energy.

[6]  Hwa Jen Yap,et al.  A Constrained Optimization based Extreme Learning Machine for noisy data regression , 2016, Neurocomputing.

[7]  Jie Hu,et al.  A novel support vector regression for data set with outliers , 2015, Appl. Soft Comput..

[8]  G CanchoVicente,et al.  Bayesian nonlinear regression models with scale mixtures of skew-normal distributions , 2011 .

[9]  Johan A. K. Suykens,et al.  Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes , 2009, ICANN.

[10]  Yuichiro Anzai,et al.  Pattern Recognition and Machine Learning , 1992, Springer US.

[11]  Sergios Theodoridis,et al.  Robust Linear Regression Analysis— A Greedy Approach , 2014, IEEE Transactions on Signal Processing.

[12]  V. Maz'ya,et al.  On approximate approximations using Gaussian kernels , 1996 .

[13]  Xiaowei Yang,et al.  A heuristic weight-setting strategy and iteratively updating algorithm for weighted least-squares support vector regression , 2008, Neurocomputing.

[14]  Zhongguang Fu,et al.  Outliers detection method of multiple measuring points of parameters in power plant units , 2015 .

[15]  Girijesh Prasad,et al.  Gaussian Mixture Model-based noise reduction in resting state fMRI data , 2013, Journal of Neuroscience Methods.

[16]  Aldo M. Garay,et al.  Heteroscedastic nonlinear regression models based on scale mixtures of skew-normal distributions. , 2011, Statistics & probability letters.

[17]  J. Militký,et al.  Detection of single influential points in OLS regression model building , 2001 .

[18]  P. J. García Nieto,et al.  A hybrid PSO optimized SVM-based model for predicting a successful growth cycle of the Spirulina platensis from raceway experiments data , 2016, J. Comput. Appl. Math..

[19]  V. H. Lachos,et al.  Robust mixture regression modeling based on scale mixtures of skew-normal distributions , 2016 .

[20]  Giuliano Galimberti,et al.  A multivariate linear regression analysis using finite mixtures of t distributions , 2014, Comput. Stat. Data Anal..

[21]  Michael Georgiopoulos,et al.  A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes , 2010, Data Mining and Knowledge Discovery.

[22]  Yan-Lin He,et al.  A novel robust regression model based on functional link least square (FLLS) and its application to modeling complex chemical processes , 2016 .

[23]  Azuraliza Abu Bakar,et al.  Outlier detection based on rough sets theory , 2009, Intell. Data Anal..

[24]  Henry Leung,et al.  A variational Bayesian approach to robust sensor fusion based on Student-t distribution , 2013, Inf. Sci..

[25]  X. Zhuang,et al.  Gaussian mixture density modeling of non-Gaussian source for autoregressive process , 1995, IEEE Trans. Signal Process..

[26]  Ping Zhong,et al.  Robust non-convex least squares loss function for regression with outliers , 2014, Knowl. Based Syst..

[27]  Jian Yang,et al.  Recursive robust least squares support vector regression based on maximum correntropy criterion , 2012, Neurocomputing.

[28]  Jugal K. Kalita,et al.  A multi-step outlier-based anomaly detection approach to network-wide traffic , 2016, Inf. Sci..

[29]  Yifei Wang,et al.  A normal least squares support vector machine (NLS-SVM) and its learning algorithm , 2009, Neurocomputing.

[30]  Heteroscedastic nonlinear regression models based on scale mixtures of skew-normal distributions. , 2011 .

[31]  Milan Meloun,et al.  Statistical Data Analysis: A Practical Guide , 2011 .

[32]  Christian Pierdzioch,et al.  A real-time quantile-regression approach to forecasting gold returns under asymmetric loss , 2015 .

[33]  Junsheng Zhao,et al.  Adaptive natural gradient learning algorithms for Mackey-Glass chaotic time prediction , 2015, Neurocomputing.

[34]  Dipak K. Dey,et al.  Bayesian nonlinear regression models with scale mixtures of skew-normal distributions: Estimation and case influence diagnostics , 2011, Comput. Stat. Data Anal..

[35]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[36]  Deyu Meng,et al.  Robust Matrix Factorization with Unknown Noise , 2013, 2013 IEEE International Conference on Computer Vision.

[37]  Lan Bai,et al.  Weighted Lagrange ε-twin support vector regression , 2016, Neurocomputing.

[38]  Qinghua Hu,et al.  Noise model based v-support vector regression with its application to short-term wind speed forecasting , 2014, Neural Networks.

[39]  Chuanfa Chen,et al.  A robust weighted least squares support vector regression based on least trimmed squares , 2015, Neurocomputing.

[40]  Raymond T. Ng,et al.  Distance-based outliers: algorithms and applications , 2000, The VLDB Journal.

[41]  Jian Yang,et al.  Robust nuclear norm regularized regression for face recognition with occlusion , 2015, Pattern Recognit..

[42]  Xiaowei Yang,et al.  A robust least squares support vector machine for regression and classification with noise , 2014, Neurocomputing.

[43]  Geoffrey J. McLachlan,et al.  Mixture models : inference and applications to clustering , 1989 .

[44]  Luis Alonso,et al.  Multioutput Support Vector Regression for Remote Sensing Biophysical Parameter Estimation , 2011, IEEE Geoscience and Remote Sensing Letters.

[45]  Lin Li,et al.  Multi-output least-squares support vector regression machines , 2013, Pattern Recognit. Lett..