Analysis of Liquid Ensembles for Enhancing the Performance and Accuracy of Liquid State Machines

Liquid state machine (LSM), a bio-inspired computing model consisting of the input sparsely connected to a randomly interlinked reservoir (or liquid) of spiking neurons followed by a readout layer, finds utility in a range of applications varying from robot control and sequence generation to action, speech, and image recognition. LSMs stand out among other Recurrent Neural Network (RNN) architectures due to their simplistic structure and lower training complexity. Plethora of recent efforts have been focused toward mimicking certain characteristics of biological systems to enhance the performance of modern artificial neural networks. It has been shown that biological neurons are more likely to be connected to other neurons in the close proximity, and tend to be disconnected as the neurons are spatially far apart. Inspired by this, we propose a group of locally connected neuron reservoirs, or an ensemble of liquids approach, for LSMs. We analyze how the segmentation of a single large liquid to create an ensemble of multiple smaller liquids affects the latency and accuracy of an LSM. In our analysis, we quantify the ability of the proposed ensemble approach to provide an improved representation of the input using the Separation Property (SP) and Approximation Property (AP). Our results illustrate that the ensemble approach enhances class discrimination (quantified as the ratio between the SP and AP), leading to better accuracy in speech and image recognition tasks, when compared to a single large liquid. Furthermore, we obtain performance benefits in terms of improved inference time and reduced memory requirements, due to lowered number of connections and the freedom to parallelize the liquid evaluation process.

[1]  Kaushik Roy,et al.  Fast, low power evaluation of elementary functions using radial basis function networks , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[2]  M. Mitchell Waldrop,et al.  Computer modelling: Brain in a box , 2012, Nature.

[3]  Andrea Montanari,et al.  A mean field view of the landscape of two-layer neural networks , 2018, Proceedings of the National Academy of Sciences.

[4]  Wenrui Zhang,et al.  Information-Theoretic Intrinsic Plasticity for Online Unsupervised Learning in Spiking Neural Networks , 2019, Front. Neurosci..

[5]  K. Fukunaga,et al.  Nonparametric Discriminant Analysis , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Dan Ventura,et al.  Preparing More Effective Liquid State Machines Using Hebbian Learning , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[7]  Bozhkov Lachezar,et al.  Echo State Network , 2017, Encyclopedia of Machine Learning and Data Mining.

[8]  Rüdiger Dillmann,et al.  Scaling up liquid state machines to predict over address events from dynamic vision sensors , 2017, Bioinspiration & biomimetics.

[9]  Dan Ventura,et al.  Spatiotemporal Pattern Recognition via Liquid State Machines , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[10]  A. Zador,et al.  Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex , 2003, Nature.

[11]  W. Singer,et al.  Distributed Fading Memory for Stimulus Properties in the Primary Visual Cortex , 2009, PLoS biology.

[12]  Fangzheng Xue,et al.  Improving liquid state machine with hybrid plasticity , 2016, 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC).

[13]  Christof Koch,et al.  Ephaptic coupling of cortical neurons , 2011, Nature Neuroscience.

[14]  Qian Wang,et al.  Energy efficient parallel neuromorphic architectures with approximate arithmetic on FPGA , 2017, Neurocomputing.

[15]  Benjamin Schrauwen,et al.  Isolated word recognition using a Liquid State Machine , 2005, ESANN.

[16]  Henry Markram,et al.  A Model for Real-Time Computation in Generic Neural Microcircuits , 2002, NIPS.

[17]  Peng Li,et al.  Architectural design exploration for neuromorphic processors with memristive synapses , 2014, 14th IEEE International Conference on Nanotechnology.

[18]  Ziang Xie,et al.  Neural Text Generation: A Practical Guide , 2017, ArXiv.

[19]  B. Schrauwen,et al.  Isolated word recognition with the Liquid State Machine: a case study , 2005, Inf. Process. Lett..

[20]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[21]  Yong Zhang,et al.  A Digital Liquid State Machine With Biologically Inspired Learning and Its Application to Speech Recognition , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Kaushik Roy,et al.  Image segmentation with stochastic magnetic tunnel junctions and spiking neurons , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[23]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[24]  Robert A. Legenstein,et al.  Methods for Estimating the Computational Power and Generalization Capability of Neural Microcircuits , 2004, NIPS.

[25]  Romain Brette,et al.  Neuroinformatics Original Research Article Brian: a Simulator for Spiking Neural Networks in Python , 2022 .

[26]  Henry Markram,et al.  Computational models for generic cortical microcircuits , 2004 .

[27]  Eris Chinellato,et al.  Facial expression recognition based on Liquid State Machines built of alternative neuron models , 2009, 2009 International Joint Conference on Neural Networks.

[28]  Abigail Morrison,et al.  Liquid computing on and off the edge of chaos with a striatal microcircuit , 2014, Front. Comput. Neurosci..

[29]  Kun Il Park,et al.  Fundamentals of Probability and Stochastic Processes with Applications to Communications , 2017 .

[30]  Andrew Philippides,et al.  Unsupervised Learning in an Ensemble of Spiking Neural Networks Mediated by ITDP , 2016, PLoS Comput. Biol..

[31]  L. F Abbott,et al.  Lapicque’s introduction of the integrate-and-fire model neuron (1907) , 1999, Brain Research Bulletin.

[32]  Gopalakrishnan Srinivasan,et al.  SpiLinC: Spiking Liquid-Ensemble Computing for Unsupervised Speech and Image Recognition , 2018, Front. Neurosci..

[33]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[34]  D. M. Hutton,et al.  The Art of Multiprocessor Programming , 2008 .

[35]  Rolf Kötter,et al.  Neuroscience databases : a practical guide , 2003 .

[36]  Zhigang Zeng,et al.  Ensembles of echo state networks for time series prediction , 2013, 2013 Sixth International Conference on Advanced Computational Intelligence (ICACI).

[37]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[38]  Priyadarshini Panda,et al.  Learning to Generate Sequences with Combination of Hebbian and Non-hebbian Plasticity in Recurrent Spiking Neural Networks , 2017, Front. Neurosci..

[39]  Haizhou Li,et al.  A Spiking Neural Network Framework for Robust Sound Classification , 2018, Front. Neurosci..

[40]  Qian Wang,et al.  D-LSM: Deep Liquid State Machine with unsupervised recurrent reservoir tuning , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[41]  Ben Jones,et al.  Is there a Liquid State Machine in the Bacterium Escherichia Coli? , 2007, 2007 IEEE Symposium on Artificial Life.

[42]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[43]  Parami Wijesinghe,et al.  An All-Memristor Deep Spiking Neural Computing System: A Step Toward Realizing the Low-Power Stochastic Brain , 2017, IEEE Transactions on Emerging Topics in Computational Intelligence.

[44]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[45]  Siddharth Joshi,et al.  Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines , 2015, Front. Neurosci..

[46]  Robert A. Legenstein,et al.  Long short-term memory and Learning-to-learn in networks of spiking neurons , 2018, NeurIPS.

[47]  Jieping Ye,et al.  Generalized Linear Discriminant Analysis: A Unified Framework and Efficient Model Selection , 2008, IEEE Transactions on Neural Networks.

[48]  Martin Krzywinski,et al.  Points of Significance: Multiple linear regression , 2015, Nature Methods.

[49]  Subhrajit Roy,et al.  An Online Unsupervised Structural Plasticity Algorithm for Spiking Neural Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Narayan Srinivasa,et al.  Learning to Recognize Actions From Limited Training Examples Using a Recurrent Spiking Neural Model , 2017, Front. Neurosci..

[51]  Qian Wang,et al.  General-purpose LSM learning processor architecture and theoretically guided design space exploration , 2015, 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[52]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[53]  Richard F. Lyon,et al.  A computational model of filtering, detection, and compression in the cochlea , 1982, ICASSP.

[54]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[55]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[56]  Seung Hwan Lee,et al.  Reservoir computing using dynamic memristors for temporal information processing , 2017, Nature Communications.

[57]  Panos E. Trahanias,et al.  Use of the separation property to derive Liquid State Machines with enhanced classification performance , 2013, Neurocomputing.

[58]  Subhrajit Roy,et al.  An Online Structural Plasticity Rule for Generating Better Reservoirs , 2016, Neural Computation.

[59]  Bahadir Kasap,et al.  Dynamic parallelism for synaptic updating in GPU-accelerated spiking neural network simulations , 2018, Neurocomputing.

[60]  Shaista Hussain,et al.  Hardware efficient, neuromorphic dendritically enhanced readout for liquid state machines , 2013, 2013 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[61]  Narayan Srinivasa,et al.  Energy-Efficient Neuron, Synapse and STDP Integrated Circuits , 2012, IEEE Transactions on Biomedical Circuits and Systems.

[62]  Jonas Degrave,et al.  Morphological Properties of Mass–Spring Networks for Optimal Locomotion Learning , 2017, Front. Neurorobot..