An adaptive multiobjective evolutionary approach to optimize artmap neural networks

This dissertation deals with the evolutionary optimization of ART neural network architectures. ART (adaptive resonance theory) was introduced by a Grossberg in 1976. In the last 20 years (1987-2007) a number of ART neural network architectures were introduced into the literature (Fuzzy ARTMAP (1992), Gaussian ARTMAP (1996 and 1997) and Ellipsoidal ARTMAP (2001)). In this dissertation, we focus on the evolutionary optimization of ART neural network architectures with the intent of optimizing the size and the generalization performance of the ART neural network. A number of researchers have focused on the evolutionary optimization of neural networks, but no research has been performed on the evolutionary optimization of ART neural networks, prior to 2006, when Daraiseh has used evolutionary techniques for the optimization of ART structures. This dissertation extends in many ways and expands in different directions the evolution of ART architectures, such as: (a) uses a multi-objective optimization of ART structures, thus providing to the user multiple solutions (ART networks) with varying degrees of merit, instead of a single solution (b) uses GA parameters that are adaptively determined throughout the ART evolution, (c) identifies a proper size of the validation set used to calculate the fitness function needed for ART's evolution, thus speeding up the evolutionary process, (d) produces experimental results that demonstrate the evolved ART's effectiveness (good accuracy and small size) and efficiency (speed) compared with other competitive ART structures, as well as other classifiers (CART (Classification and Regression Trees) and SVM (Support Vector Machines)). The overall methodology to evolve ART using a multi-objective approach, the chromosome representation of an ART neural network, the genetic operators used in ART's evolution, and the automatic adaptation of some of the GA parameters in ART's evolution could also be applied in the evolution of other exemplar based neural network classifiers such as the probabilistic neural network and the radial basis function neural network.

[1]  Richard J. Enbody,et al.  Further Research on Feature Selection and Classification Using Genetic Algorithms , 1993, ICGA.

[2]  Stephen Grossberg,et al.  Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system , 1991, Neural Networks.

[3]  Yannis A. Dimitriadis,et al.  Study of distributed learning as a solution to category proliferation in Fuzzy ARTMAP based neural systems , 2003, Neural Networks.

[4]  A. R. Burton,et al.  Utilisation of an adaptive resonance theory neural network as a genetic algorithm fitness evaluator , 1997, Proceedings of IEEE International Symposium on Information Theory.

[5]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[6]  David E. Goldberg,et al.  A niched Pareto genetic algorithm for multiobjective optimization , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[7]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[8]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[9]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[10]  Dirk Thierens,et al.  Convergence Models of Genetic Algorithm Selection Schemes , 1994, PPSN.

[11]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[12]  Jerzy W. Bala,et al.  Using Learning to Facilitate the Evolution of Features for Recognizing Visual Concepts , 1996, Evolutionary Computation.

[13]  Jeffrey Horn,et al.  The Niched Pareto Genetic Algorithm 2 Applied to the Design of Groundwater Remediation Systems , 2001, EMO.

[14]  Terence C. Fogarty,et al.  Varying the Probability of Mutation in the Genetic Algorithm , 1989, ICGA.

[15]  Zbigniew Michalewicz,et al.  Adaptation in evolutionary computation: a survey , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[16]  Ahmad Al-Daraiseh,et al.  Genetically Engineered Adaptive Resonance Theory (art) Neural Network Architectures , 2006 .

[17]  Ron Kohavi,et al.  Improving simple Bayes , 1997 .

[18]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[19]  James R. Williamson,et al.  A Constructive, Incremental-Learning Network for Mixture Modeling and Classification , 1997, Neural Computation.

[20]  Lothar Thiele,et al.  A Comparison of Selection Schemes used in Genetic Algorithms , 1995 .

[21]  Patrick K. Simpson,et al.  Dynamic Feature Set Training of Neural Nets for Classification , 1995, Evolutionary Programming.

[22]  Jack Sklansky,et al.  A note on genetic algorithms for large-scale feature selection , 1989, Pattern Recognition Letters.

[23]  Gregory L. Heileman,et al.  Rademacher penalization applied to fuzzy ARTMAP and boosted ARTMAP , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[24]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[25]  Shiro Usui,et al.  Mutation-based genetic neural network , 2005, IEEE Transactions on Neural Networks.

[26]  Hisao Ishibuchi,et al.  Multi-objective genetic local search algorithm , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[27]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[28]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[29]  Lipo Wang,et al.  A GA-based RBF classifier with class-dependent features , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[30]  Milos Manic,et al.  Robust algorithm for neural network training , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[31]  C. A. Coello Coello,et al.  Evolutionary multi-objective optimization: a historical view of the field , 2006, IEEE Computational Intelligence Magazine.

[32]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[33]  S. Grossberg,et al.  ART 2: self-organization of stable category recognition codes for analog input patterns. , 1987, Applied optics.

[34]  Kalyanmoy Deb,et al.  Genetic Algorithms, Noise, and the Sizing of Populations , 1992, Complex Syst..

[35]  Kalyanmoy Deb,et al.  An Investigation of Niche and Species Formation in Genetic Function Optimization , 1989, ICGA.

[36]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[37]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[38]  W. Mendenhall,et al.  Statistics for engineering and the sciences , 1984 .

[39]  Ah-Hwee Tan,et al.  Rule Extraction: From Neural Architecture to Symbolic Representation , 1995 .

[40]  Peter J. Fleming,et al.  An Overview of Evolutionary Algorithms in Multiobjective Optimization , 1995, Evolutionary Computation.

[41]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[42]  David E. Goldberg,et al.  Genetic Algorithms, Tournament Selection, and the Effects of Noise , 1995, Complex Syst..

[43]  D. B. Fogel,et al.  Using evolutionary programing to create neural networks that are capable of playing tic-tac-toe , 1993, IEEE International Conference on Neural Networks.

[44]  C. Fonseca,et al.  GENETIC ALGORITHMS FOR MULTI-OBJECTIVE OPTIMIZATION: FORMULATION, DISCUSSION, AND GENERALIZATION , 1993 .

[45]  Robert F. Harrison,et al.  A modified fuzzy ARTMAP architecture for the approximation of noisy mappings , 1995, Neural Networks.

[46]  Jihoon Yang,et al.  Feature Subset Selection Using a Genetic Algorithm , 1998, IEEE Intell. Syst..

[47]  David E. Goldberg,et al.  Evaluation relaxation using substructural information and linear estimation , 2006, GECCO '06.

[48]  David B. Fogel,et al.  An Evolutionary Programming Approach to Self-Adaptation on Finite State Machines , 1995, Evolutionary Programming.

[49]  James E. Baker,et al.  Adaptive Selection Methods for Genetic Algorithms , 1985, International Conference on Genetic Algorithms.

[50]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[51]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[52]  David E. Goldberg,et al.  Genetic Algorithms, Selection Schemes, and the Varying Effects of Noise , 1996, Evolutionary Computation.

[53]  Yaochu Jin,et al.  A comprehensive survey of fitness approximation in evolutionary computation , 2005, Soft Comput..

[54]  Georgios C. Anagnostopoulos,et al.  Exemplar-based pattern recognition via semi-supervised learning , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[55]  Michael Georgiopoulos,et al.  Properties of learning in ARTMAP , 1994, Neural Networks.

[56]  Stephen Grossberg,et al.  ARTMAP: supervised real-time learning and classification of nonstationary data by a self-organizing neural network , 1991, [1991 Proceedings] IEEE Conference on Neural Networks for Ocean Engineering.

[57]  Georgios C. Anagnostopoulos,et al.  Novel approaches in adaptive resonance theory for machine learning , 2001 .

[58]  J. Dennis,et al.  A closer look at drawbacks of minimizing weighted sums of objectives for Pareto set generation in multicriteria optimization problems , 1997 .

[59]  Georgios C. Anagnostopoulos,et al.  Category regions as new geometrical concepts in Fuzzy-ART and Fuzzy-ARTMAP , 2002, Neural Networks.

[60]  Georgios C. Anagnostopoulos,et al.  Cross-validation in Fuzzy ARTMAP for large databases , 2001, Neural Networks.

[61]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[62]  Ethem Alpaydin,et al.  Omnivariate decision trees , 2001, IEEE Trans. Neural Networks.

[63]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[64]  Francisco Sandoval Hernández,et al.  Genetic Synthesis of Discrete-Time Recurrent Neural Network , 1993, IWANN.

[65]  James R. Williamson,et al.  Gaussian ARTMAP: A Neural Network for Fast Incremental Learning of Noisy Multidimensional Maps , 1996, Neural Networks.

[66]  Gail A. Carpenter,et al.  Distributed ARTMAP: a neural network for fast distributed supervised learning , 1998, Neural Networks.

[67]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[68]  Erick Cantú-Paz,et al.  Feature Subset Selection by Estimation of Distribution Algorithms , 2002, GECCO.

[69]  Lawrence Davis,et al.  Hybridizing the Genetic Algorithm and the K Nearest Neighbors Classification Algorithm , 1991, ICGA.

[70]  Robert E. Smith,et al.  Fitness inheritance in genetic algorithms , 1995, SAC '95.

[71]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[72]  Shing Chiang Tan,et al.  On the reduction of complexity in the architecture of fuzzy ARTMAP with dynamic decay adjustment , 2006, Neurocomputing.

[73]  Marco Laumanns,et al.  Combining Convergence and Diversity in Evolutionary Multiobjective Optimization , 2002, Evolutionary Computation.

[74]  Prabhat Hajela,et al.  Genetic search strategies in multicriterion optimal design , 1991 .

[75]  Randall S. Sexton,et al.  Toward global optimization of neural networks: A comparison of the genetic algorithm and backpropagation , 1998, Decis. Support Syst..

[76]  Brad L. Miller,et al.  Noise, sampling, and efficient genetic algorthms , 1997 .

[77]  Bofeng Zhang,et al.  Impulse force based ART network with GA optimization , 2003, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003.

[78]  Stephen Grossberg,et al.  Adaptive pattern classification and universal recoding: II. Feedback, expectation, olfaction, illusions , 1976, Biological Cybernetics.

[79]  Konstantinos P. Ferentinos,et al.  Biological engineering applications of feedforward neural networks designed and parameterized by genetic algorithms , 2005, Neural Networks.

[80]  Carlos A. Coello Coello,et al.  An updated survey of GA-based multiobjective optimization techniques , 2000, CSUR.

[81]  Jonathan E. Fieldsend,et al.  Pareto evolutionary neural networks , 2005, IEEE Transactions on Neural Networks.

[82]  Yannis A. Dimitriadis,et al.  MicroARTMAP: use of mutual information for category reduction in fuzzy ARTMAP , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[83]  J. Fitzpatrick,et al.  Genetic Algorithms in Noisy Environments , 2005, Machine Learning.

[84]  Marco Laumanns,et al.  SPEA2: Improving the strength pareto evolutionary algorithm , 2001 .

[85]  Georgios C. Anagnostopoulos,et al.  Reducing generalization error and category proliferation in ellipsoid ARTMAP via tunable misclassification error tolerance: boosted ellipsoid ARTMAP , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[86]  Lalit M. Patnaik,et al.  Adaptive probabilities of crossover and mutation in genetic algorithms , 1994, IEEE Trans. Syst. Man Cybern..

[87]  Jonathan E. Fieldsend,et al.  Using unconstrained elite archives for multiobjective optimization , 2003, IEEE Trans. Evol. Comput..

[88]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[89]  Thomas Bck,et al.  Self-adaptation in genetic algorithms , 1991 .