Evolutionary Spiking Neural Networks for Solving Supervised Classification Problems

This paper presents a grammatical evolution (GE)-based methodology to automatically design third generation artificial neural networks (ANNs), also known as spiking neural networks (SNNs), for solving supervised classification problems. The proposal performs the SNN design by exploring the search space of three-layered feedforward topologies with configured synaptic connections (weights and delays) so that no explicit training is carried out. Besides, the designed SNNs have partial connections between input and hidden layers which may contribute to avoid redundancies and reduce the dimensionality of input feature vectors. The proposal was tested on several well-known benchmark datasets from the UCI repository and statistically compared against a similar design methodology for second generation ANNs and an adapted version of that methodology for SNNs; also, the results of the two methodologies and the proposed one were improved by changing the fitness function in the design process. The proposed methodology shows competitive and consistent results, and the statistical tests support the conclusion that the designs produced by the proposal perform better than those produced by other methodologies.

[1]  Hui Li,et al.  Evolutionary artificial neural networks: a review , 2011, Artificial Intelligence Review.

[2]  Ángel Fernando Kuri Morales.  Training Neural Networks Using Non-standard Norms - Preliminary Results , 2000, MICAI.

[3]  Juan Humberto Sossa Azuela,et al.  Design of artificial neural networks using a modified Particle Swarm Optimization algorithm , 2009, 2009 International Joint Conference on Neural Networks.

[4]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[5]  Ammar Belatreche,et al.  Advances in Design and Application of Spiking Neural Networks , 2006, Soft Comput..

[6]  Qingxiang Wu,et al.  An Evolutionary Strategy for Supervised Training of Biologically Plausible Neural Networks , 2003 .

[7]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[8]  Roberto Antonio Vázquez,et al.  Training spiking neural models using cuckoo search algorithm , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[9]  Wang Qian,et al.  A cooperative method for supervised learning in Spiking neural networks , 2010, The 2010 14th International Conference on Computer Supported Cooperative Work in Design.

[10]  Juan Martín Carpio Valadez,et al.  Comparing Metaheuristic Algorithms on the Training Process of Spiking Neural Networks , 2014, Recent Advances on Hybrid Approaches for Designing Intelligent Systems.

[11]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[12]  Ammar Belatreche.  Biologically Inspired Neural Networks: Models, Learning, and Applications , 2010 .

[13]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[14]  Edoardo Amaldi,et al.  A Review of Combinatorial Problems Arising in Feedforward Neural Network Design , 1994, Discret. Appl. Math..

[15]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[16]  F. J. Anscombe,et al.  The Validity of Comparative Experiments , 1948 .

[17]  Bruno Apolloni,et al.  Special Issue: Contemporary development of neural computation and applications , 2012, Neural Computing and Applications.

[18]  Ron Meir,et al.  Evolving a learning algorithm for the binary perceptron , 1991 .

[19]  Alfonso Ortega,et al.  Christiansen Grammar Evolution: Grammatical Evolution With Semantics , 2007, IEEE Transactions on Evolutionary Computation.

[20]  Stefan Schliebs.  Optimisation and Modelling of Spiking Neural Networks: Enhancing Neural Information Processing Systems through the Power of Evolution , 2010 .

[21]  Michael Schmitt,et al.  On the Complexity of Learning for Spiking Neurons with Temporal Coding , 1999, Inf. Comput..

[22]  James Kennedy,et al.  Particle swarm optimization , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[23]  Amir Hossein Gandomi,et al.  Cuckoo search algorithm: a metaheuristic approach to solve structural optimization problems , 2011, Engineering with Computers.

[24]  Jianwei Zhang,et al.  A Survey on CPG-Inspired Control Models and System Implementation , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[26]  Ah Chung Tsoi,et al.  Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results , 1998, Neural Networks.

[27]  M. O'Neill,et al.  Grammatical evolution , 2001, GECCO '09.

[28]  Sameer Singh,et al.  Novelty detection: a review - part 2: : neural network based approaches , 2003, Signal Process..

[29]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[30]  A. K. Morales,et al.  Non-standard norms in genetically trained neural networks , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[31]  Beatriz A. Garro,et al.  Designing Artificial Neural Networks Using Particle Swarm Optimization Algorithms , 2015, Comput. Intell. Neurosci..

[32]  Rosario Baltazar,et al.  Comparison of PSO and DE for Training Neural Networks , 2011, 2011 10th Mexican International Conference on Artificial Intelligence.

[33]  Daniel Rivero,et al.  Generation and simplification of Artificial Neural Networks by means of Genetic Programming , 2010, Neurocomputing.

[34]  J. Stephen Judd,et al.  On the complexity of loading shallow neural networks , 1988, J. Complex..

[35]  J. Stephen Judd,et al.  Neural network design and the complexity of learning , 1990, Neural network modeling and connectionism.

[36]  Hojjat Adeli,et al.  Spiking Neural Networks , 2009, Int. J. Neural Syst..

[37]  Andres Espinal,et al.  A FPGA-Based Neuromorphic Locomotion System for Multi-Legged Robots , 2017, IEEE Access.

[38]  Roberto Antonio Vázquez,et al.  Tuning the parameters of an integrate and fire neuron via a genetic algorithm for solving pattern recognition problems , 2015, Neurocomputing.

[39]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[40]  Weiguo Sheng,et al.  An Adaptive Memetic Algorithm With Rank-Based Mutation for Artificial Neural Network Architecture Optimization , 2017, IEEE Access.

[41]  Roberto Antonio Vázquez,et al.  Integrate and Fire neurons and their application in pattern recognition , 2010, 2010 7th International Conference on Electrical Engineering Computing Science and Automatic Control.

[42]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[43]  K. Chung,et al.  Limit Distributions for Sums of Independent Random Variables. , 1955 .

[44]  Hédi Soula,et al.  Spontaneous Dynamics of Asymmetric Random Recurrent Spiking Neural Networks , 2004, Neural Computation.

[45]  Martín Carpio,et al.  Partially-Connected Artificial Neural Networks Developed by Grammatical Evolution for Pattern Recognition Problems , 2018, Fuzzy Logic Augmentation of Neural and Optimization Algorithms.

[46]  Juan Martín Carpio Valadez,et al.  Comparing Evolutionary Strategy Algorithms for Training Spiking Neural Networks , 2015, Res. Comput. Sci..

[47]  Auke Jan Ijspeert,et al.  Central pattern generators for locomotion control in animals and robots: A review , 2008, Neural Networks.

[48]  Juan Martín Carpio Valadez,et al.  Quadrupedal Robot Locomotion: A Biologically Inspired Approach and Its Hardware Implementation , 2016, Comput. Intell. Neurosci..

[49]  Roberto Antonio Vázquez.  Pattern Recognition Using Spiking Neurons and Firing Rates , 2010, IBERAMIA.

[50]  Caro Lucas,et al.  Evolving artificial neural network structure using grammar encoding and colonial competitive algorithm , 2012, Neural Computing and Applications.

[51]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[52]  Beatriz A. Garro,et al.  Training Spiking Neurons by Means of Particle Swarm Optimization , 2011, ICSI.

[53]  Václav Snásel,et al.  Metaheuristic design of feedforward neural networks: A review of two decades of research , 2017, Eng. Appl. Artif. Intell..

[54]  Euripidis Glavas,et al.  Neural network construction and training using grammatical evolution , 2008, Neurocomputing.

[55]  Hava T. Siegelmann,et al.  On the Intractability of Loading Neural Networks , 1994 .

[56]  Luis Fernando de Mingo López,et al.  The optimal combination: Grammatical swarm, particle swarm optimization and neural networks , 2012, J. Comput. Sci..

[57]  Kyu Ho Park,et al.  Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates , 1996, Neurocomputing.

[58]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[59]  David A. Elizondo,et al.  A Survey of Partially Connected Neural Networks , 1997, Int. J. Neural Syst..

[60]  Juan Martín Carpio Valadez,et al.  Developing Architectures of Spiking Neural Networks by Using Grammatical Evolution Based on Evolutionary Strategy , 2014, MCPR.

[61]  Fardin Ahmadizar,et al.  Artificial neural network development by means of a novel combination of grammatical evolution and genetic algorithm , 2015, Eng. Appl. Artif. Intell..

[62]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1990 .

[63]  Michael O'Neill,et al.  Grammatical Evolution: Evolving Programs for an Arbitrary Language , 1998, EuroGP.

[64]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[65]  Sander M. Bohte,et al.  Error-backpropagation in temporally encoded networks of spiking neurons , 2000, Neurocomputing.

[66]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[67]  Gabriela Ochoa,et al.  Evolvability metrics in adaptive operator selection , 2014, GECCO.

[68]  Kenji Doya,et al.  Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning , 2017, Neural Networks.

[69]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[70]  Guoqiang Peter Zhang,et al.  Neural networks for classification: a survey , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[71]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[72]  Enrique Alba,et al.  Full Automatic ANN Design: A Genetic Approach , 1993, IWANN.

[73]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .