Generalized regression neural networks in time-varying environment

The current state of knowledge regarding nonstationary processes is significantly poorer then in the case of stationary signals. In many applications, signals are treated as stationary only because in this way it is easier to analyze them; in fact, they are nonstationary. Nonstationary processes are undoubtedly more difficult to analyze and their diversity makes application of universal tools impossible. In this paper we propose a new class of generalized regression neural networks working in nonstationary environment. The generalized regession neural networks (GRNN) studied in this paper are able to follow changes of the best model, i.e., time-varying regression functions. The novelty is summarized as follows: 1) We present adaptive GRNN tracking time-varying regression functions. 2) We prove convergence of the GRNN based on general learning theorems presented in Section IV. 3) We design in detail special GRNN based on the Parzen and orthogonal series kernels. In each case we precise conditions ensuring convergence of the GRNN to the best models described by regression function. 4) We investigate speed of convergence of the GRNN and compare performance of specific structures based on the Parzen kernel and orthogonal series kernel. 5) We study various nonstationarities (multiplicative, additive, "scale change," "movable argument") and design in each case the GRNN based on the Parzen kernel and orthogonal series kernel.

[1]  Masafumi Watanabe.  ON CONVERGENCES OF ASYMPTOTICALLY OPTIMAL DISCRIMINANT FUNCTIONS FOR PATTERN CLASSIFICATION PROBLEMS , 1974 .

[2]  Lianfen Qian,et al.  Regularized Radial Basis Function Networks: Theory and Applications , 2002, Technometrics.

[3]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[4]  L. Carleson.  On convergence and growth of partial sums of Fourier series , 1966 .

[5]  A. Krzyżak,et al.  Asymptotic properties of kernel estimates of a regression function , 1980 .

[6]  G. Walter.  Properties of Hermite Series Estimation of Probability Density , 1977 .

[7]  P. Sjölin.  Convergence almost everywhere of certain singular integrals and multiple Fourier series , 1971 .

[8]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[9]  Pietro Burrascano,et al.  Learning vector quantization for the probabilistic neural network , 1991, IEEE Trans. Neural Networks.

[10]  A. Krzyżak,et al.  Distribution-Free Pointwise Consistency of Kernel Regression Estimate , 1984 .

[11]  Leszek Rutkowski,et al.  Identification of MISO nonlinear regressions in the presence of a wide class of disturbances , 1991, IEEE Trans. Inf. Theory.

[12]  Anthony Zaknich.  General Regression Neural Network , 2003 .

[13]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[14]  L. Rutkowski,et al.  Sequential Estimates of Probability Densities by Orthogonal Series and Their Application in Pattern Classification LESZEK , 2005 .

[15]  R. Tapia,et al.  Nonparametric Function Estimation, Modeling, and Simulation , 1987 .

[16]  L. Rutkowski.  Sequential Estimates of a Regression Function by Orthogonal Series with Applications in Discrimination , 1981 .

[17]  Christopher J. BISHOPAbstra,et al.  Orthogonal Functions , 2022 .

[18]  D. Meeter.  Stochastic Approximation and Nonlinear Regression , 1969 .

[19]  H. Davies.  STRONG CONSISTENCY OF A SEQUENTIAL ESTIMATOR OF A PROBABILITY DENSITY FUNCTION , 1973 .

[20]  L. Devroye.  On the Almost Everywhere Convergence of Nonparametric Regression Function Estimates , 1981 .

[21]  Leszek Rutkowski,et al.  Adaptive probabilistic neural networks for pattern classification in time-varying environment , 2004, IEEE Transactions on Neural Networks.

[23]  Leszek Rutkowski,et al.  New Soft Computing Techniques for System Modeling, Pattern Classification and Image Processing , 2004 .

[24]  Rene F. Swarttouw,et al.  Orthogonal polynomials , 2020, NIST Handbook of Mathematical Functions.

[25]  D. F. Specht,et al.  Enhancements to probabilistic neural networks , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[26]  Dan W. Patterson,et al.  Artificial Neural Networks: Theory and Applications , 1998 .

[27]  A. Ullah,et al.  Nonparametric Econometrics , 1999 .

[28]  E. Nadaraya.  On Estimating Regression , 1964 .

[29]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[30]  V. Dupac.  A DYNAMIC STOCHASTIC APPROXIMATION METHOD , 1965 .

[31]  B. Yandell.  Spline smoothing and nonparametric regression , 1989 .

[32]  Wee Ser,et al.  Probabilistic neural-network structure determination for pattern classification , 2000, IEEE Trans. Neural Networks Learn. Syst..

[33]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[34]  Donald F. Specht,et al.  Probabilistic neural networks and the polynomial Adaline as complementary techniques for classification , 1990, IEEE Trans. Neural Networks.

[35]  Leszek Rutkowski.  Multiple Fourier series procedures for extraction of nonlinear regressions from noisy data , 1993, IEEE Trans. Signal Process..

[36]  Joseph P. LaSalle,et al.  A Course in Mathematical Analysis , 1906, Nature.

[37]  Leszek Rutkowski,et al.  Flexible neuro-fuzzy systems , 2003, IEEE Trans. Neural Networks.