Bio-inspired Stochastic Growth and Initialization for Artificial Neural Networks

Current initialization methods for artificial neural networks (ANNs) assume full connectivity between network layers. We propose that a bio-inspired initialization method for establishing connections between neurons in an artificial neural network will produce more accurate results relative to a fully connected network. We demonstrate four implementations of a novel, stochastic method for generating sparse connections in spatial, growth-based connectivity (GBC) maps. Connections in GBC maps are used to generate initial weights for neural networks in a deep learning compatible framework. These networks, designated as Growth-Initialized Neural Networks (GrINNs), have sparse connections between the input layer and the hidden layer. GrINNs were tested with user-specified nominal connectivity percentages ranging from 5–45%, resulting in unique connectivity percentages ranging from 4–28%. For reference, fully connected networks are defined as having 100% unique connectivity within this context. GrINNs with nominal connectivity percentages \(\ge \)20% produced better accuracy than fully connected ANNs when trained and tested on the MNIST dataset.

[1]  Roger D. Quinn,et al.  Neuromechanical Model of Rat Hind Limb Walking with Two Layer CPGs and Muscle Synergies , 2018, Living Machines.

[2]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[3]  Roger D. Quinn,et al.  Leg-local neural mechanisms for searching and learning enhance robotic locomotion , 2018, Biological Cybernetics.

[4]  Nikola Kasabov,et al.  Dynamic evolving spiking neural networks for on-line spatio- and spectro-temporal pattern recognition. , 2013, Neural networks : the official journal of the International Neural Network Society.

[5]  Vittorio Maniezzo,et al.  Genetic evolution of the topology and weight distribution of neural networks , 1994, IEEE Trans. Neural Networks.

[6]  Huaguang Zhang,et al.  A Comprehensive Review of Stability Analysis of Continuous-Time Recurrent Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[7]  O. J. Dunn.  Multiple Comparisons among Means , 1961 .

[8]  Karl J. Friston,et al.  Active Inference: A Process Theory , 2017, Neural Computation.

[9]  Xuanjing Huang,et al.  Recurrent Neural Network for Text Classification with Multi-Task Learning , 2016, IJCAI.

[10]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[11]  Timothée Masquelier,et al.  Bio-inspired unsupervised learning of visual features leads to robust invariant object recognition , 2015, Neurocomputing.

[12]  D. P. Mital,et al.  Application of neural networks in robotic control , 1991, 1991., IEEE International Sympoisum on Circuits and Systems.

[13]  T. W. Anderson,et al.  Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes , 1952 .

[14]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[15]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[16]  Thomas Martinetz,et al.  A neural network for robot control: cooperation between neural units as a requirement for learning , 1993 .

[17]  Shiuh-Jer Huang,et al.  Neural network controller for robotic motion control , 1996 .

[18]  Bruce P. Graham,et al.  Continuum model for tubulin-driven neurite elongation , 2004, Neurocomputing.

[19]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[21]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[22]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[23]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[24]  Jun Zhao,et al.  Recurrent Convolutional Neural Networks for Text Classification , 2015, AAAI.

[25]  Bruce Graham,et al.  Biologically plausible models of neurite outgrowth. , 2005, Progress in brain research.

[26]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[27]  Wei Li,et al.  A Neural Network with Central Pattern Generators Entrained by Sensory Feedback Controls Walking of a Bipedal Model , 2016, Living Machines.

[28]  Thomas Nowotny,et al.  Classifying continuous, real-time e-nose sensor data using a bio-inspired spiking network modelled on the insect olfactory system , 2016, Bioinspiration & biomimetics.

[29]  Simei Gomes Wysoski,et al.  Evolving spiking neural networks for audiovisual information processing , 2010, Neural Networks.

[30]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[31]  Zenghui Wang,et al.  Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review , 2017, Neural Computation.

[32]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[33]  Roger D. Quinn,et al.  A Functional Subnetwork Approach to Designing Synthetic Nervous Systems That Control Legged Robot Locomotion , 2017, Front. Neurorobot..

[34]  Arpan Kumar Kar,et al.  Bio inspired computing - A review of algorithms and scope of applications , 2016, Expert Syst. Appl..

[35]  Arjen van Ooyen,et al.  Mathematical modelling and numerical simulation of the morphological development of neurons , 2006, BMC Neuroscience.