Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation

This paper presents a multiobjective evolutionary algorithm to optimize radial basis function neural networks (RBFNNs) in order to approach target functions from a set of input-output pairs. The procedure allows the application of heuristics to improve the solution of the problem at hand by including some new genetic operators in the evolutionary process. These new operators are based on two well-known matrix transformations: singular value decomposition (SVD) and orthogonal least squares (OLS), which have been used to define new mutation operators that produce local or global modifications in the radial basis functions (RBFs) of the networks (the individuals in the population in the evolutionary procedure). After analyzing the efficiency of the different operators, we have shown that the global mutation operators yield an improved procedure to adjust the parameters of the RBFNNs.

[1]  Peter J. Fleming,et al.  Genetic Algorithms for Multiobjective Optimization: FormulationDiscussion and Generalization , 1993, ICGA.

[2]  Julio Ortega Lopera,et al.  Improved RAN sequential prediction using orthogonal techniques , 2001, Neurocomputing.

[3]  Jerry M. Mendel,et al.  Generating fuzzy rules by learning from examples , 1992, IEEE Trans. Syst. Man Cybern..

[4]  Manuel Valenzuela-Rendón,et al.  A Non-Generational Genetic Algorithm for Multiobjective Optimization , 1997, ICGA.

[5]  Héctor Pomares,et al.  A systematic approach to a self-generating fuzzy rule-table for function approximation , 2000, IEEE Trans. Syst. Man Cybern. Part B.

[6]  Héctor Pomares,et al.  A new radial basis function networks structure: application to time series prediction , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[7]  Héctor Pomares,et al.  Self-organized fuzzy system generation from training examples , 2000, IEEE Trans. Fuzzy Syst..

[8]  Roman Rosipal,et al.  Prediction of Chaotic Time-Series with a Resource-Allocating RBF Network , 1998, Neural Processing Letters.

[9]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[10]  H. Bersini,et al.  Using incremental learning algorithms in the search for minimal and effective fuzzy models , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[11]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[12]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[13]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[14]  Robert J. Hammell,et al.  Interpolation, Completion, and Learning Fuzzy Rules , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[15]  Yacov Y. Haimes,et al.  Multiobjective Decision Making: Theory and Methodology , 1983 .

[16]  Jerome H. Friedman.  Multivariate adaptive regression splines (with discussion) , 1991 .

[17]  John Yen,et al.  Simplifying fuzzy rule-based models using orthogonal transformation methods , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[18]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[19]  Peter J. Fleming,et al.  Multiobjective optimization and multiple constraint handling with evolutionary algorithms. II. Application example , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[20]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[21]  E. G. Kogbetliantz.  Solution of linear equations by diagonalization of coefficients matrix , 1955 .

[22]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[23]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[24]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[25]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[26]  Peter J. Fleming,et al.  Multiobjective optimization and multiple constraint handling with evolutionary algorithms. I. A unified formulation , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[27]  Ludmila I. Kuncheva,et al.  Initializing of an RBF network by a genetic algorithm , 1997, Neurocomputing.

[28]  C. Fonseca,et al.  GENETIC ALGORITHMS FOR MULTI-OBJECTIVE OPTIMIZATION: FORMULATION, DISCUSSION, AND GENERALIZATION , 1993 .

[29]  Didier Guériot,et al.  RBF neural network, basis functions and genetic algorithm , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[30]  Nicolaos B. Karayiannis,et al.  Reformulated radial basis neural networks trained by gradient descent , 1999, IEEE Trans. Neural Networks.

[31]  R. Weiner.  Lecture Notes in Economics and Mathematical Systems , 1985 .

[32]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[33]  Giuseppe Patanè,et al.  The enhanced LBG algorithm , 2001, Neural Networks.

[34]  Ignacio Rojas,et al.  A New Clustering Technique for Function Approximation , 2005 .

[35]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[36]  C. Hwang.  Multiple Objective Decision Making - Methods and Applications: A State-of-the-Art Survey , 1979 .

[37]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[38]  Andrew R. Webb,et al.  Shape-adaptive radial basis functions , 1998, IEEE Trans. Neural Networks.

[39]  Héctor Pomares,et al.  Structure identification in complete rule-based fuzzy systems , 2002, IEEE Trans. Fuzzy Syst..

[40]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[41]  Chuen-Tsai Sun,et al.  Functional equivalence between radial basis function networks and fuzzy inference systems , 1993, IEEE Trans. Neural Networks.

[42]  Ernesto Tarantino,et al.  Optimizing Neural Networks for Time Series Prediction , 1999 .

[43]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[44]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[45]  Hisao Ishibuchi,et al.  Multi-objective genetic local search algorithm , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[46]  C. Hwang,et al.  Fuzzy Multiple Objective Decision Making: Methods And Applications , 1996 .

[47]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[48]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[49]  R. Brent,et al.  The Solution of Singular-Value and Symmetric Eigenvalue Problems on Multiprocessor Arrays , 1985 .

[50]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[51]  Robert M. Farber,et al.  How Neural Nets Work , 1987, NIPS.

[52]  Mohamad T. Musavi,et al.  On the training of radial basis function classifiers , 1992, Neural Networks.

[53]  Héctor Pomares,et al.  Analysis of the Functional Block Involved in the Design of Radial Basis Function Networks , 2000, Neural Processing Letters.

[54]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[55]  G. C. Mouzouris,et al.  Designing fuzzy logic systems for uncertain environments using a singular-value-QR decomposition method , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[56]  Lorenzo Bruzzone,et al.  A technique for the selection of kernel-function parameters in RBF neural networks for classification of remote-sensing images , 1999, IEEE Trans. Geosci. Remote. Sens..

[57]  Partha Pratim Kanjilal,et al.  On the application of orthogonal transformation for the design and analysis of feedforward networks , 1995, IEEE Trans. Neural Networks.

[58]  Peter Grant,et al.  Orthogonal least squares algorithms for training multi-output radial basis function networks , 1991 .

[59]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[60]  Bart Kosko,et al.  Fuzzy function approximation with ellipsoidal rules , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[61]  Vladimir Cherkassky,et al.  Constrained topological mapping for nonparametric regression analysis , 1991, Neural Networks.

[62]  Nicolaos B. Karayiannis,et al.  Growing radial basis neural networks: merging supervised and unsupervised learning with network growth techniques , 1997, IEEE Trans. Neural Networks.

[63]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[64]  Vladimir Cherkassky,et al.  Comparison of adaptive methods for function estimation from samples , 1996, IEEE Trans. Neural Networks.

[65]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[66]  António Gaspar-Cunha,et al.  Use of Genetic Algorithms in Multicriteria Optimization to Solve Industrial Problems , 1997, ICGA.

[67]  Bruce A. Whitehead,et al.  Genetic evolution of radial basis function coverage using orthogonal niches , 1996, IEEE Trans. Neural Networks.

[68]  Kalyanmoy Deb,et al.  An Investigation of Niche and Species Formation in Genetic Function Optimization , 1989, ICGA.

[69]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (3rd ed.) , 1996 .

[70]  J. Yen,et al.  An SVD-based fuzzy model reduction strategy , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[71]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[72]  John Yen,et al.  Radial basis function networks, regression weights, and the expectation-maximization algorithm , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[73]  Bruce A. Whitehead,et al.  Evolving space-filling curves to distribute radial basis functions over an input space , 1994, IEEE Trans. Neural Networks.

[74]  Julio Ortega,et al.  On-Line Optimization of Radial Basis Function Networks with Orthogonal Techniques , 1999, IWANN.

[75]  Jesús González Peñalver.  Identificación y optimización de redes de funciones base radiales para aproximación funcional , 2001 .

[76]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[77]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[78]  Ignacio Rojas,et al.  Short-Term Prediction of Chaotic Time Series by Using RBF Network with Regression Weights , 2000, Int. J. Neural Syst..

[79]  R. Brent,et al.  Computation of the Singular Value Decomposition Using Mesh-Connected Processors , 1983 .

[80]  Paul Van Dooren,et al.  On efficient implementations of Kogbetliantz's algorithm for computing the singular value decomposition , 1987 .