Evolution of functional link networks

This paper addresses the genetic design of functional link networks (FLN). FLN are high-order perceptrons (HOP) without hidden units. Despite their linear nature, FLN can capture nonlinear input-output relationships, provided that they are fed with an adequate set of polynomial inputs, which are constructed out of the original input attributes. Given this set, it turns out to be very simple to train the network, as compared with a multilayer perceptron (MLP). However finding the optimal subset of units is a difficult problem because of its nongradient nature and the large number of available units, especially for high degrees. Some constructive growing methods have been proposed to address this issue, Here, we rely on the global search capabilities of a genetic algorithm to scan the space of subsets of polynomial units, which is plagued by a host of local minima. By contrast, the quadratic error function of each individual FLN has only one minimum, which makes fitness evaluation practically noiseless. We find that surprisingly simple FLN compare favorably with other more complex architectures derived by means of constructive and evolutionary algorithms on some UCI benchmark data sets. Moreover, our models are especially amenable to interpretation, due to an incremental approach that penalizes complex architectures and starts with a pool of single-attribute FLN.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  George S. Sebestyen,et al.  Decision-making processes in pattern recognition , 1962 .

[3]  Nikhil R. Pal,et al.  Soft computing for feature analysis , 1999, Fuzzy Sets Syst..

[4]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[5]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[6]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[7]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[8]  Dejan J. Sobajic,et al.  Learning and generalization characteristics of the random vector Functional-link net , 1994, Neurocomputing.

[9]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[10]  Jihoon Yang,et al.  Feature Subset Selection Using a Genetic Algorithm , 1998, IEEE Intell. Syst..

[11]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[12]  Jürgen Schürmann,et al.  Pattern classification , 2008 .

[13]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[14]  Paulo J. G. Lisboa,et al.  Translation, rotation, and scale invariant pattern recognition by high-order neural networks and moment classifiers , 1992, IEEE Trans. Neural Networks.

[15]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[16]  Sung-Bae Cho,et al.  Evolutionary modular neural networks for intelligent systems , 1998, Int. J. Intell. Syst..

[17]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[18]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[19]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[20]  Richard S. Sutton,et al.  Iterative Construction of Sparse Polynomial Approximations , 1991, NIPS.

[21]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[22]  Stefano Nolfi,et al.  Econets: Neural networks that learn in an environment , 1990 .

[23]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[24]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[25]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[26]  Joydeep Ghosh,et al.  Efficient Higher-Order Neural Networks for Classification and Function Approximation , 1992, Int. J. Neural Syst..

[27]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[28]  Manoel Fernando Tenorio,et al.  Self-organizing network for optimum supervised learning , 1990, IEEE Trans. Neural Networks.

[29]  Georg Thimm,et al.  Optimization of high order perceptrons , 1997 .

[30]  Eric B. Baum,et al.  A Proposal for More Powerful Learning Algorithms , 1989, Neural Computation.

[31]  P. GALLINARI,et al.  On the relations between discriminant analysis and multilayer perceptrons , 1991, Neural Networks.

[32]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[33]  Hans-Paul Schwefel,et al.  Evolution and optimum seeking , 1995, Sixth-generation computer technology series.

[34]  David W. Aha,et al.  Noise-Tolerant Instance-Based Learning Algorithms , 1989, IJCAI.

[35]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[36]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[37]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[38]  Ralf Salomon,et al.  Evolutionary algorithms and gradient search: similarities and differences , 1998, IEEE Trans. Evol. Comput..

[39]  Juan Julián Merelo Guervós,et al.  G-lvq, a Combination of Genetic Algorithms and Lvq , 1995, ICANNGA.

[40]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[41]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .