Non-flat Function Estimation Using Orthogonal Least Squares Regression with Multi-scale Wavelet Kernel

Estimating the non-flat function which comprises both the steep variations and the smooth variations is a hard problem. The existing kernel methods with a single common variance for all the regressors can not achieve satisfying results. In this paper, a novel multi-scale model is constructed to tackle the problem by orthogonal least squares regression (OLSR) with wavelet kernel. The scheme tunes the dilation and translation of each wavelet kernel regressor by incrementally minimizing the training mean square error using a guided random search algorithm. In order to prevent the possible over-fitting, a practical method to select termination threshold is used. The experimental results show that, for non-flat function estimation problem, OLSR outperforms traditional methods in terms of precision and sparseness. And OLSR with wavelet kernel has a faster convergence rate as compared to that with conventional Gaussian kernel.

[1]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[2]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[3]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[4]  Sheng Chen,et al.  Orthogonal least squares regression with tunable kernels , 2005 .

[5]  Sheng Chen,et al.  Experiments with repeating weighted boosting search for optimization signal processing applications , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[6]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[7]  Li Zhang,et al.  Wavelet support vector machine , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[8]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[9]  B. Schölkopf,et al.  Linear programs for automatic accuracy control in regression. , 1999 .

[10]  Stéphane Canu,et al.  Kernel Basis Pursuit , 2005, ECML.

[11]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[12]  D. N. Zheng,et al.  Training sparse MS-SVR with an expectation-maximization algorithm , 2006, Neurocomputing.

[13]  Jiaxin Wang,et al.  Non-flat function estimation with a multi-scale support vector regression , 2006, Neurocomputing.

[14]  Sheng Chen,et al.  Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks , 1999, IEEE Trans. Neural Networks.