Multi-network evolutionary systems and automatic decomposition of complex problems

Multi-network systems, i.e. multiple neural network systems, can often solve complex problems more effectively than their monolithic counterparts. Modular neural networks (MNNs) tackle a complex problem by decomposing it into simpler subproblems and then solving them. Unlike the decomposition in MNNs, a neural network ensemble usually includes redundant component nets and is often inspired by statistical theories. This paper presents different types of problem decompositions and discusses the suitability of various multi-network systems for different decompositions. A classification of various multi-network systems, in the context of problem decomposition, is obtained by exploiting these differences. Then a specific type of problem decomposition, which gives no information about the subproblems and is often ignored in literature, is discussed in detail and a novel MNN architecture for problem decomposition is presented. Finally, a co-evolutionary model is presented, which is used to design and optimize such MNNs with subtask specific modules. The model consists of two populations. The first population consists of a pool of modules and the second population synthesizes complete systems by drawing elements from the pool of modules. Modules represent a part of the solution, which co-operate with each other to form a complete solution. Using two artificial supervised learning tasks, constructed from smaller subtasks, it can be shown that if a particular task decomposition is better than others, in terms of performance on the overall task, it can be evolved using the co-evolutionary model.

[1]  César Hervás-Martínez,et al.  Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks) , 2002, Neural Networks.

[2]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[3]  Marco Zaffalon,et al.  Robust Feature Selection by Mutual Information Distributions , 2002, UAI.

[4]  J. Moody,et al.  Feature Selection Based on Joint Mutual Information , 1999 .

[5]  Christian Igel,et al.  Empirical evaluation of the improved Rprop learning algorithms , 2003, Neurocomputing.

[6]  David W. Opitz,et al.  Feature Selection for Ensembles , 1999, AAAI/IAAI.

[7]  M. Husken,et al.  Optimization for problem classes-neural networks that learn to learn , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[8]  J. Bullinaria.  To Modularize or Not To Modularize ? , 2002 .

[9]  Henrik Gollee,et al.  Modular Neural Networks and Self-Decomposition , 1997 .

[10]  Gavin Brown,et al.  Diversity in neural network ensembles , 2004 .

[11]  D. Thieffry,et al.  Modularity in development and evolution. , 2000, BioEssays : news and reviews in molecular, cellular and developmental biology.

[12]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[13]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[14]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[15]  S. Kosslyn,et al.  Why are What and Where Processed by Separate Cortical Visual Systems? A Computational Investigation , 1989, Journal of Cognitive Neuroscience.

[16]  Lior Rokach,et al.  Improving Supervised Learning by Feature Decomposition , 2002, FoIKS.

[17]  G. Lendaris,et al.  On Prestructuring ANNs Using A Priori Knowledge , 1994 .

[18]  Robert E. Jenkins,et al.  A Simplified Neural-Network Solution through Problem Decomposition: The Case of the Truck Backer-Upper , 1992, Neural Computation.

[19]  Noel E. Sharkey,et al.  Artificial neural networks for coordination and control: The portability of experiential representations , 1997, Robotics Auton. Syst..

[20]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[21]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[22]  G. Wagner,et al.  A case study of the evolution of modularity: towards a bridge between evolutionary biology, artificial life, neuro- and cognitive science , 1998 .

[23]  Yuansong Liao,et al.  Constructing Heterogeneous Committees Using Input Feature Grouping: Application to Economic Forecasting , 1999, NIPS.

[24]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[25]  Michael I. Jordan,et al.  Modular and hierarchical learning systems , 1998 .

[26]  Marc Toussaint,et al.  Task-dependent evolution of modularity in neural networks , 2002 .

[27]  Tomas Hrycej.  Modular learning in neural networks - a modularized approach to neural network classification , 1992, Sixth-Generation computer technology series.

[28]  César Hervás-Martínez,et al.  Cooperative coevolution of artificial neural network ensembles for pattern classification , 2005, IEEE Transactions on Evolutionary Computation.

[29]  Bernhard Sendhoff,et al.  Variable encoding of modular neural networks for time series prediction , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[30]  Deniz Erdogmus,et al.  Feature selection in MLPs and SVMs based on maximum output information , 2004, IEEE Transactions on Neural Networks.

[31]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[32]  Risto Miikkulainen,et al.  COOPERATIVE COEVOLUTION OF MULTI-AGENT SYSTEMS , 2001 .

[33]  Michael I. Jordan,et al.  Task Decomposition Through Competition in a Modular Connectionist Architecture: The What and Where Vision Tasks , 1990, Cogn. Sci..

[34]  Domenico Parisi,et al.  Evolving Modular Architectures for Neural Networks , 2000, NCPW.

[35]  Masami Ito,et al.  Task decomposition and module combination based on class relations: a modular neural network for pattern classification , 1999, IEEE Trans. Neural Networks.

[36]  Xin Yao,et al.  Credit Assignment Among Neurons in Co-evolving Populations , 2004, PPSN.

[37]  Xin Yao,et al.  Co-evolutionary modular neural networks for automatic problem decomposition , 2005, 2005 IEEE Congress on Evolutionary Computation.

[38]  Yaser S. Abu-Mostafa,et al.  A Method for Learning From Hints , 1992, NIPS.

[39]  Amanda J. C. Sharkey,et al.  Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems , 1999 .