A Gaussian mixture framework for incremental nonparametric regression with topology learning neural networks

Incremental learning is important for memory critical systems, especially when the growth of information technology has pushed the memory and storage costs to limits. Despite the great amount of effort researching into incremental classification paradigms and algorithms, the regression is given far less attention. In this paper, an incremental regression framework that is able to model the linear and nonlinear relationships between response variables and explanatory variables is proposed. A three layer feed-forward neural network structure is devised where the weights of the hidden layer are trained by topology learning neural networks. A Gaussian mixture weighted integrator is used to synthesize from the output of the hidden layer to give smoothed predictions. Two hidden layer parameters learning strategies whether by Growing Neural Gas (GNG) or the single layered Self-Organizing Incremental Neural Network (SOINN) are explored. The GNG strategy is more robust and flexible, and single layered SOINN strategy is less sensitive to parameter settings. Experiments are carried out on an artificial dataset and 6 UCI datasets. The artificial dataset experiments show that the proposed method is able to give predictions more smoothed than K-nearest-neighbor (KNN) and the regression tree. Comparing to the parametric method Support Vector Regression (SVR), the proposed method has significant advantage when learning on data with multi-models. Incremental methods including Passive and Aggressive regression, Online Sequential Extreme Learning Machine, Self-Organizing Maps and Incremental K-means are compared with the proposed method on the UCI datasets, and the results show that the proposed method outperforms them on most datasets.