Dendritic Neuron Model With Effective Learning Algorithms for Classification, Approximation, and Prediction

An artificial neural network (ANN) that mimics the information processing mechanisms and procedures of neurons in human brains has achieved a great success in many fields, e.g., classification, prediction, and control. However, traditional ANNs suffer from many problems, such as the hard understanding problem, the slow and difficult training problems, and the difficulty to scale them up. These problems motivate us to develop a new dendritic neuron model (DNM) by considering the nonlinearity of synapses, not only for a better understanding of a biological neuronal system, but also for providing a more useful method for solving practical problems. To achieve its better performance for solving problems, six learning algorithms including biogeography-based optimization, particle swarm optimization, genetic algorithm, ant colony optimization, evolutionary strategy, and population-based incremental learning are for the first time used to train it. The best combination of its user-defined parameters has been systemically investigated by using the Taguchi’s experimental design method. The experiments on 14 different problems involving classification, approximation, and prediction are conducted by using a multilayer perceptron and the proposed DNM. The results suggest that the proposed learning algorithms are effective and promising for training DNM and thus make DNM more powerful in solving classification, approximation, and prediction problems.

[1]  Jatinder N. D. Gupta,et al.  Comparative evaluation of genetic algorithm and backpropagation for training neural networks , 2000, Inf. Sci..

[2]  Zheng Tang,et al.  A Breast Cancer Classifier Using a Neuron Model with Dendritic Nonlinearity , 2015, IEICE Trans. Inf. Syst..

[3]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[4]  Yuanqing Li,et al.  Neural-Dynamic-Method-Based Dual-Arm CMG Scheme With Time-Varying Constraints Applied to Humanoid Robots , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[6]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[7]  Surya Ganguli,et al.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization , 2014, NIPS.

[8]  B. S. Lim,et al.  Optimal design of neural networks using the Taguchi method , 1995, Neurocomputing.

[9]  Gisbert Schneider,et al.  Optimized Particle Swarm Optimization (OPSO) and its application to artificial neural network training , 2006, BMC Bioinformatics.

[10]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[11]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[12]  Guang-Bin Huang,et al.  Extreme Learning Machine for Multilayer Perceptron , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Daniel J. Simon,et al.  Evolutionary optimization algorithms : biologically-Inspired and population-based approaches to computer intelligence , 2013 .

[14]  Zheng Tang,et al.  Unsupervised learnable neuron model with nonlinear interaction on dendrites , 2014, Neural Networks.

[15]  Naiqi Wu,et al.  SVM-DT-based adaptive and collaborative intrusion detection , 2018, IEEE/CAA Journal of Automatica Sinica.

[16]  Jiahai Wang,et al.  Financial time series prediction using a dendritic neuron model , 2016, Knowl. Based Syst..

[17]  T. Poggio,et al.  Nonlinear interactions in a dendritic tree: localization, timing, and role in information processing. , 1983, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Wolfgang Maass,et al.  Branch-Specific Plasticity Enables Self-Organization of Nonlinear Computation in Single Neurons , 2011, The Journal of Neuroscience.

[19]  Marco Dorigo,et al.  Ant colony optimization for continuous domains , 2008, Eur. J. Oper. Res..

[20]  Ping Wang,et al.  Cuckoo Search and Particle Filter-Based Inversing Approach to Estimating Defects via Magnetic Flux Leakage Signals , 2016, IEEE Transactions on Magnetics.

[21]  Nicolas Brunel,et al.  Single neuron dynamics and computation , 2014, Current Opinion in Neurobiology.

[22]  Y. Jan,et al.  Branching out: mechanisms of dendritic arborization , 2010, Nature Reviews Neuroscience.

[23]  Chia-Feng Juang,et al.  A hybrid of genetic algorithm and particle swarm optimization for recurrent network design , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[24]  Yanqing Zhang,et al.  Granular Neural Networks With Evolutionary Interval Learning , 2008, IEEE Transactions on Fuzzy Systems.

[25]  Burkhard Lenze,et al.  How to make sigma-pi neural networks perform perfectly on regular training sets , 1994, Neural Networks.

[26]  Oscar Castillo,et al.  Optimization of type-2 fuzzy weights in backpropagation learning for neural networks using GAs and PSO , 2016, Appl. Soft Comput..

[27]  Marzuki Khalid,et al.  Evolutionary Fuzzy ARTMAP Neural Networks for Classification of Semiconductor Defects , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Bartlett W. Mel,et al.  An Augmented Two-Layer Model Captures Nonlinear Analog Spatial Integration Effects in Pyramidal Neuron Dendrites , 2014, Proceedings of the IEEE.

[29]  L. Abbott,et al.  Synaptic computation , 2004, Nature.

[30]  Zhijun Zhang,et al.  Three Recurrent Neural Networks and Three Numerical Methods for Solving a Repetitive Motion Planning Scheme of Redundant Robot Manipulators , 2017, IEEE/ASME Transactions on Mechatronics.

[31]  Michael R. Lyu,et al.  A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training , 2007, Appl. Math. Comput..

[32]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[33]  Le Zhang,et al.  A survey of randomized algorithms for training neural networks , 2016, Inf. Sci..

[34]  MengChu Zhou,et al.  Dynamic Behavior of Artificial Hodgkin–Huxley Neuron Model Subject to Additive Noise , 2016, IEEE Transactions on Cybernetics.

[35]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1972 .

[36]  Nikhil R. Pal,et al.  Feature Selection Using a Neural Framework With Controlled Redundancy , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Moncef Gabbouj,et al.  Evolutionary artificial neural networks by multi-dimensional particle swarm optimization , 2009, Neural Networks.

[38]  Liang Zhao,et al.  PSO-based single multiplicative neuron model for time series prediction , 2009, Expert Syst. Appl..

[39]  Wei Wu,et al.  Uniform Approximation Capabilities of Sum-of-Product and Sigma-Pi-Sigma Neural Networks , 2007, ISNN.

[40]  P. Jesper Sjöström,et al.  One Cell to Rule Them All, and in the Dendrites Bind Them , 2011, Front. Syn. Neurosci..

[41]  Concha Bielza,et al.  A review on probabilistic graphical models in evolutionary computation , 2012, Journal of Heuristics.

[42]  Prem Kumar Kalra,et al.  Time series prediction with single multiplicative neuron model , 2007, Appl. Soft Comput..

[43]  Christian Igel,et al.  Empirical evaluation of the improved Rprop learning algorithms , 2003, Neurocomputing.

[44]  Jiujun Cheng,et al.  An approximate logic neuron model with a dendritic structure , 2016, Neurocomputing.

[45]  Richard Stuart Neville,et al.  Transformations of sigma-pi nets: obtaining reflected functions by reflecting weight matrices , 2002, Neural Networks.

[46]  Jiujun Cheng,et al.  Using a Single Dendritic Neuron to Forecast Tourist Arrivals to Japan , 2017, IEICE Trans. Inf. Syst..

[47]  Wei Wu,et al.  Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks , 2007, Neural Processing Letters.

[48]  Li Wang,et al.  Functional-type single-input-rule-modules connected neural fuzzy system for wind speed prediction , 2017, IEEE/CAA Journal of Automatica Sinica.

[49]  Pedro Larrañaga,et al.  Analyzing the Population Based Incremental Learning Algorithm by Means of Discrete Dynamical Systems , 2000, Complex Syst..

[50]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[51]  Geoff Holmes,et al.  A Modified Quickprop Algorithm , 1991, Neural Computation.

[52]  Nader Fathianpour,et al.  A hybrid simultaneous perturbation artificial bee colony and back-propagation algorithm for training a local linear radial basis neural network on ore grade estimation , 2017, Neurocomputing.

[53]  Luigi Grippo,et al.  Decomposition Techniques for Multilayer Perceptron Training , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[54]  Randall S. Sexton,et al.  Toward global optimization of neural networks: A comparison of the genetic algorithm and backpropagation , 1998, Decis. Support Syst..

[55]  Francisco Herrera,et al.  A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization , 2009, J. Heuristics.

[56]  Walter Senn,et al.  Somato-dendritic Synaptic Plasticity and Error-backpropagation in Active Dendrites , 2016, PLoS Comput. Biol..

[57]  Romain D. Cazé,et al.  Dendrites enable a robust mechanism for neuronal stimulus selectivity , 2016, bioRxiv.

[58]  Claudia Clopath,et al.  Modelling plasticity in dendrites: from single cells to networks , 2017, Current Opinion in Neurobiology.

[59]  C. Koch,et al.  Multiplicative computation in a visual neuron sensitive to looming , 2002, Nature.

[60]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[61]  Kevin N. Gurney,et al.  Training nets of hardware realizable sigma-pi units , 1992, Neural Networks.

[62]  Pedro Larrañaga,et al.  A Review on Estimation of Distribution Algorithms , 2002, Estimation of Distribution Algorithms.

[63]  Bartlett W. Mel,et al.  Sigma-Pi Learning: On Radial Basis Functions and Cortical Associative Learning , 1989, NIPS.

[64]  Adam P. Piotrowski,et al.  Differential Evolution algorithms applied to Neural Network training suffer from stagnation , 2014, Appl. Soft Comput..

[65]  Hussein A. Abbass,et al.  Speeding Up Backpropagation Using Multiobjective Evolutionary Algorithms , 2003, Neural Computation.

[66]  MengChu Zhou,et al.  Generating Highly Accurate Predictions for Missing QoS Data via Aggregating Nonnegative Latent Factor Models , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[67]  Dan Simon,et al.  Biogeography-Based Optimization , 2022 .

[68]  MengChu Zhou,et al.  Colored Traveling Salesman Problem , 2015, IEEE Transactions on Cybernetics.

[69]  Andrew Lewis,et al.  Let a biogeography-based optimizer train your Multi-Layer Perceptron , 2014, Inf. Sci..

[70]  Tao Jiang,et al.  A neuron model with synaptic nonlinearities in a dendritic tree for liver disorders , 2017 .

[71]  Robert C. Creese,et al.  Supplier Selection Based on a Neural Network Model Using Genetic Algorithm , 2009, IEEE Transactions on Neural Networks.

[72]  Andries P. Engelbrecht,et al.  Computational Intelligence: An Introduction , 2002 .

[73]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[74]  David Shan-Hill Wong,et al.  Design and Application of a Variable Selection Method for Multilayer Perceptron Neural Network With LASSO , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[75]  Genichi Taguchi Computer-Based Robust Engineering : Essentials for DFSS , 2010 .

[76]  Christof Koch,et al.  The role of single neurons in information processing , 2000, Nature Neuroscience.

[77]  Stefan Wermter,et al.  A self-organizing map of sigma-pi units , 2007, Neurocomputing.

[78]  E. Marder,et al.  Plasticity in single neuron and circuit computations , 2004, Nature.

[79]  Malcolm I. Heywood,et al.  A framework for improved training of Sigma-Pi networks , 1995, IEEE Trans. Neural Networks.

[80]  Thomas Stützle,et al.  Ant colony optimization: artificial ants as a computational intelligence technique , 2006 .

[81]  M. Zhou,et al.  Gaussian Classifier-Based Evolutionary Strategy for Multimodal Optimization , 2014, IEEE Transactions on Neural Networks and Learning Systems.