Function and surface approximation based on enhanced Kernel Regression for small sample sets

The function approximation problem is to find the appropriate relationship between a dependent and independent variable(s). Function approximation algorithms generally require sufficient samples to approximate a function. Insufficient samples may cause any function approximation algorithm to result in unsatisfactory predictions. To solve this problem, a function approximation algorithm called Weighted Kernel Regression (WKR), which is based on Nadaraya-Watson kernel regression (NWKR), is proposed. In the proposed framework, the original NWKR algorithm is enhanced by expressing the observed samples in a square kernel matrix. The WKR is trained to estimate the weight for the testing phase. The weight is estimated iteratively and governed by the error function to find a good approximation model. Four experiments are conducted to show the capability of the WKR. The results show that the proposed WKR model is effective in cases where the target function is non-linear and the given training sample is small. The performance of the WKR is also compared with other existing function approximation algorithms, such as artificial neural networks (ANN).

[1]  B. Yegnanarayana,et al.  Artificial Neural Networks , 2004 .

[2]  Jules Thibault,et al.  Process modeling with neural networks using small experimental datasets , 1999 .

[3]  吳柏林,et al.  ON NONPARAMETRIC ESTIMATION FOR THE GROWTH OF TOTAL FACTOR PRODUCTIVITY: A STUDY ON CHINA AND ITS FOUR EASTERN PROVINCES , 2007 .

[4]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[5]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[6]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[7]  Tomaso Poggio,et al.  Incorporating prior information in machine learning by creating virtual examples , 1998, Proc. IEEE.

[8]  Yong Shi,et al.  Towards Efficient Fuzzy Information Processing - Using the Principle of Information Diffusion , 2002, Studies in Fuzziness and Soft Computing.

[9]  A. Roli.  Artificial Neural Networks , 2012, Lecture Notes in Computer Science.

[10]  Marzuki Khalid,et al.  A Non-linear Function Approximation from Small Samples Based on Nadaraya-Watson Kernel Regression , 2010, 2010 2nd International Conference on Computational Intelligence, Communication Systems and Networks.

[11]  Chenghu Zhou,et al.  An improved kernel regression method based on Taylor expansion , 2007, Appl. Math. Comput..

[12]  Katya Rodríguez-Vázquez,et al.  Multi-branches Genetic Programming as a Tool for Function Approximation , 2004, GECCO.

[13]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[14]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[15]  E. Nadaraya.  On Estimating Regression , 1964 .

[16]  Claudio Moraga,et al.  A diffusion-neural-network for learning from small samples , 2004, Int. J. Approx. Reason..

[17]  Bart Kosko,et al.  Fuzzy Systems as Universal Approximators , 1994, IEEE Trans. Computers.

[18]  Soon-Chuan Ong,et al.  Learning from small data sets to improve assembly semiconductor manufacturing processes , 2010, 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE).

[20]  Gérard Bloch,et al.  Support vector regression from simulation data and few experimental samples , 2008, Inf. Sci..

[21]  J. C. Suh,et al.  Function Approximations by Superimposing Genetic Programming Trees: With Applications to Engineering Problems , 2000, Inf. Sci..

[22]  Zbigniew Telec,et al.  Nonparametric Statistical Analysis of Machine Learning Algorithms for Regression Problems , 2010, KES.

[23]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[24]  Sungzoon Cho,et al.  Observational Learning Algorithm for an Ensemble of Neural Networks , 2002, Pattern Analysis & Applications.

[25]  A. Isaksson,et al.  Cross-validation and bootstrapping are unreliable in small sample classification , 2008, Pattern Recognit. Lett..

[26]  Der-Chiang Li,et al.  Approximate modeling for high order non-linear functions using small sample sets , 2008, Expert Syst. Appl..

[27]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[28]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..