Simultaneous optimization of neural network weights and active nodes using metaheuristics

Optimization of neural network (NN) is significantly influenced by the transfer function used in its active nodes. It has been observed that the homogeneity in the activation nodes does not provide the best solution. Therefore, the customizable transfer functions whose underlying parameters are subjected to optimization were used to provide heterogeneity to NN. For experimental purposes, a meta-heuristic framework using a combined genotype representation of connection weights and transfer function parameter was used. The performance of adaptive Logistic, Tangent-hyperbolic, Gaussian and Beta functions were analyzed. Concise comparisons between different transfer function and between the NN optimization algorithms are presented. The comprehensive analysis of the results obtained over the benchmark dataset suggests that the Artificial Bee Colony with adaptive transfer function provides the best results in terms of classification accuracy over the particle swarm optimization and differential evolution algorithms.

[1]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[2]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[3]  Ilaria Castelli.  Neural Networks with Adaptive Activation Functions , 2009 .

[4]  Dervis Karaboga,et al.  AN IDEA BASED ON HONEY BEE SWARM FOR NUMERICAL OPTIMIZATION , 2005 .

[5]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[6]  R. Storn,et al.  Differential Evolution - A simple and efficient adaptive scheme for global optimization over continuous spaces , 2004 .

[7]  Adel M. Alimi,et al.  Designing Beta Basis Function Neural Network for optimization using Artificial Bee Colony (ABC) , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[8]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[9]  Adam Slowik,et al.  Application of an Adaptive Differential Evolution Algorithm With Multiple Trial Vectors to Artificial Neural Network Training , 2011, IEEE Transactions on Industrial Electronics.

[10]  Andries P. Engelbrecht,et al.  Introduction to Computational Intelligence , 2007 .

[11]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[12]  F.J. Von Zuben,et al.  Hierarchical evolution of heterogeneous neural networks , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[13]  Teresa Bernarda Ludermir,et al.  Optimization of the weights and asymmetric activation function family of neural network for time series forecasting , 2013, Expert Syst. Appl..

[14]  David White,et al.  GANNet: A Genetic Algorithm for Optimizing Topology and Weights in Neural Network Design , 1993, IWANN.

[15]  Xin Yao,et al.  Evolutionary design of artificial neural networks with different nodes , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[16]  Enrique Alba,et al.  Parallel Metaheuristics: A New Class of Algorithms , 2005 .

[17]  Andries P. Engelbrecht,et al.  Computational Intelligence: An Introduction , 2002 .

[18]  Wang Wei,et al.  An ANN model of optimizing activation functions based on constructive algorithm and GP , 2010, 2010 International Conference on Computer Application and System Modeling (ICCASM 2010).

[19]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[20]  P. N. Suganthan,et al.  Differential Evolution Algorithm With Strategy Adaptation for Global Numerical Optimization , 2009, IEEE Transactions on Evolutionary Computation.

[21]  Adel M. Alimi,et al.  Beta fuzzy logic systems approximation properties in the mimo case , 2003 .

[22]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[23]  Panos A. Ligomenides,et al.  GANNET: a genetic algorithm for searching topology and weight spaces in neural network design. The first step in finding a neural network solution , 1993 .

[24]  Andries Petrus Engelbrecht,et al.  Lambda-gamma learning with feedforward neural networks using particle swarm optimization , 2011, 2011 IEEE Symposium on Swarm Intelligence.

[25]  Edmondo Trentin,et al.  Supervised and Unsupervised Co-training of Adaptive Activation Functions in Neural Nets , 2011, PSL.

[26]  D. Karaboga,et al.  On the performance of artificial bee colony (ABC) algorithm , 2008, Appl. Soft Comput..

[27]  Norbert Jankowski,et al.  Survey of Neural Transfer Functions , 1999 .