Novel direct and self-regulating approaches to determine optimum growing multi-experts network structure

This work presents two novel approaches to determine optimum growing multi-experts network (GMN) structure. The first method called direct method deals with expertise domain and levels in connection with local experts. The growing neural gas (GNG) algorithm is used to cluster the local experts. The concept of error distribution is used to apportion error among the local experts. After reaching the specified size of the network, redundant experts removal algorithm is invoked to prune the size of the network based on the ranking of the experts. However, GMN is not ergonomic due to too many network control parameters. Therefore, a self-regulating GMN (SGMN) algorithm is proposed. SGMN adopts self-adaptive learning rates for gradient-descent learning rules. In addition, SGMN adopts a more rigorous clustering method called fully self-organized simplified adaptive resonance theory in a modified form. Experimental results show SGMN obtains comparative or even better performance than GMN in four benchmark examples, with reduced sensitivity to learning parameters setting. Moreover, both GMN and SGMN outperform the other neural networks and statistical models. The efficacy of SGMN is further justified in three industrial applications and a control problem. It provides consistent results besides holding out a profound potential and promise for building a novel type of nonlinear model consisting of several local linear models.

[1]  G. Golub,et al.  Good Ridge Parameter , 1979 .

[2]  S. Omohundro.  The Delaunay Triangulation and Function Learning , 1990 .

[3]  Kazuo Tanaka,et al.  Successive identification of a fuzzy model and its applications to prediction of a complex system , 1991 .

[4]  Shyh Hwang,et al.  An identification algorithm in fuzzy relational systems , 1996, Soft Computing in Intelligent Systems and Information Processing. Proceedings of the 1996 Asian Fuzzy Systems Symposium.

[5]  Bernd Fritzke,et al.  A Growing Neural Gas Network Learns Topologies , 1994, NIPS.

[6]  Vladimir Cherkassky,et al.  Learning from Data: Concepts, Theory, and Methods , 1998 .

[7]  Bernd Fritzke,et al.  Fast learning with incremental RBF networks , 1994, Neural Processing Letters.

[8]  Stephen A. Billings,et al.  Radial basis function network configuration using genetic algorithms , 1995, Neural Networks.

[9]  Thomas Martinetz,et al.  Topology representing networks , 1994, Neural Networks.

[10]  Yong-Zai Lu,et al.  Fuzzy Model Identification and Self-Learning for Dynamic Systems , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  Hans Henrik Thodberg,et al.  A review of Bayesian neural networks with an application to near infrared spectroscopy , 1996, IEEE Trans. Neural Networks.

[12]  Giulio Sandini,et al.  An incremental growing neural network and its application to robot control , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[13]  T. Sejnowski,et al.  Irresistible environment meets immovable neurons , 1997, Behavioral and Brain Sciences.

[14]  David E. Rumelhart,et al.  Generalization by Weight-Elimination with Application to Forecasting , 1990, NIPS.

[15]  George W. Irwin,et al.  Nonlinear control structures based on embedded neural system models , 1997, IEEE Trans. Neural Networks.

[16]  Ujjwal Bhattacharya,et al.  Self-adaptive learning rates in backpropagation algorithm improve its function approximation performance , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[17]  Roderick Murray-Smith,et al.  Local model networks and local learning , 1994 .

[18]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[19]  Paolo Frasconi,et al.  Learning without local minima in radial basis function networks , 1995, IEEE Trans. Neural Networks.

[20]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[21]  Dušan Petrovački,et al.  Evolutional development of a multilevel neural network , 1993, Neural Networks.

[22]  Stefan Schaal,et al.  From Isolation to Cooperation: An Alternative View of a System of Experts , 1995, NIPS.

[23]  T. J. McAvoy,et al.  Dynamics of pH in Controlled Stirred Tank Reactor , 1972 .

[24]  Michio Sugeno,et al.  A fuzzy-logic-based approach to qualitative modeling , 1993, IEEE Trans. Fuzzy Syst..

[25]  Sigeru Omatu,et al.  Process control by on-line trained neural controllers , 1992, IEEE Trans. Ind. Electron..

[26]  J. Stephen Judd,et al.  On the complexity of loading shallow neural networks , 1988, J. Complex..

[27]  Euntai Kim,et al.  A new approach to fuzzy modeling , 1997, IEEE Trans. Fuzzy Syst..

[28]  Reza Langari,et al.  Building Sugeno-type models using fuzzy discretization and orthogonal parameter estimation techniques , 1995, IEEE Trans. Fuzzy Syst..

[29]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[30]  Kurt Hornik,et al.  Some new results on neural network approximation , 1993, Neural Networks.

[31]  H. W. Werntges.  Partitions of unity improve neural function approximators , 1993, IEEE International Conference on Neural Networks.

[32]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[33]  Ethem Alpaydin,et al.  Constructive feedforward ART clustering networks. I , 2002, IEEE Trans. Neural Networks.

[34]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[35]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[36]  Yves Chauvin,et al.  A Back-Propagation Algorithm with Optimal Use of Hidden Units , 1988, NIPS.

[37]  Andrea Baraldi,et al.  Constructive Feedforward ART Clustering Networks — Part II , 2001 .

[38]  R. Tong.  The evaluation of fuzzy models derived from experimental data , 1980 .

[39]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[40]  P. Kumar,et al.  Theory and practice of recursive identification , 1985, IEEE Transactions on Automatic Control.

[41]  M. Agarwal.  A systematic classification of neural-network-based control , 1997 .

[42]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[43]  Yinghua Lin,et al.  A new approach to fuzzy-neural system modeling , 1995, IEEE Trans. Fuzzy Syst..

[44]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[45]  Petter Krus,et al.  Fluid Power Control of a Flexible Mechanical Structure , 1997 .

[46]  Lennart Ljung,et al.  Theory and Practice of Recursive Identification , 1983 .

[47]  Witold Pedrycz.  Identification in fuzzy systems , 1984, IEEE Transactions on Systems, Man, and Cybernetics.

[48]  M. Sugeno,et al.  Derivation of Fuzzy Control Rules from Human Operator's Control Actions , 1983 .

[49]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.