Enhanced Nadaraya-Watson Kernel Regression: Surface Approximation for Extremely Small Samples

The function approximation problem is to find the appropriate relationship between a dependent and independent variable(s). Function approximation algorithms generally require sufficient samples to approximate a function. Insufficient samples may cause any approximation algorithm to result in unsatisfactory predictions. To solve this problem, a function approximation algorithm called Weighted Kernel Regression (WKR), which is based on Nadaraya-Watson kernel regression, is proposed. In the proposed framework, the original Nadaraya-Watson kernel regression algorithm is enhanced by expressing the observed samples in a square kernel matrix. The WKR is trained to estimate the weight for the testing phase. The weight is estimated iteratively and is governed by the error function to find a good approximation model. Two experiments are conducted to show the capability of the WKR. The results show that the proposed WKR model is effective in cases where the target surface function is non-linear and the given training sample is small. The performance of the WKR is also compared with other existing function approximation algorithms, such as artificial neural networks (ANN).

[1]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[2]  E. Nadaraya.  On Estimating Regression , 1964 .

[3]  Marzuki Khalid,et al.  A Non-linear Function Approximation from Small Samples Based on Nadaraya-Watson Kernel Regression , 2010, 2010 2nd International Conference on Computational Intelligence, Communication Systems and Networks.

[4]  Claudio Moraga,et al.  A diffusion-neural-network for learning from small samples , 2004, Int. J. Approx. Reason..

[5]  Gérard Bloch,et al.  Support vector regression from simulation data and few experimental samples , 2008, Inf. Sci..

[6]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[7]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[8]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[9]  Bart Kosko,et al.  Fuzzy Systems as Universal Approximators , 1994, IEEE Trans. Computers.

[10]  Soon-Chuan Ong,et al.  Learning from small data sets to improve assembly semiconductor manufacturing processes , 2010, 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE).

[11]  A. Isaksson,et al.  Cross-validation and bootstrapping are unreliable in small sample classification , 2008, Pattern Recognit. Lett..

[12]  J. C. Suh,et al.  Function Approximations by Superimposing Genetic Programming Trees: With Applications to Engineering Problems , 2000, Inf. Sci..

[13]  B. Yegnanarayana,et al.  Artificial Neural Networks , 2004 .

[14]  Jules Thibault,et al.  Process modeling with neural networks using small experimental datasets , 1999 .

[15]  吳柏林,et al.  ON NONPARAMETRIC ESTIMATION FOR THE GROWTH OF TOTAL FACTOR PRODUCTIVITY: A STUDY ON CHINA AND ITS FOUR EASTERN PROVINCES , 2007 .

[16]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[18]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[19]  Sungzoon Cho,et al.  Observational Learning Algorithm for an Ensemble of Neural Networks , 2002, Pattern Analysis & Applications.

[20]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[21]  Chenghu Zhou,et al.  An improved kernel regression method based on Taylor expansion , 2007, Appl. Math. Comput..

[22]  Der-Chiang Li,et al.  Approximate modeling for high order non-linear functions using small sample sets , 2008, Expert Syst. Appl..

[23]  Zbigniew Telec,et al.  Nonparametric Statistical Analysis of Machine Learning Algorithms for Regression Problems , 2010, KES.

[24]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[25]  Yong Shi,et al.  Towards Efficient Fuzzy Information Processing - Using the Principle of Information Diffusion , 2002, Studies in Fuzziness and Soft Computing.

[26]  Tomaso Poggio,et al.  Incorporating prior information in machine learning by creating virtual examples , 1998, Proc. IEEE.

[27]  Katya Rodríguez-Vázquez,et al.  Multi-branches Genetic Programming as a Tool for Function Approximation , 2004, GECCO.