Computing with Spikes: The Advantage of Fine-Grained Timing

Neural-inspired spike-based computing machines often claim to achieve considerable advantages in terms of energy and time efficiency by using spikes for computation and communication. However, fundamental questions about spike-based computation remain unanswered. For instance, how much advantage do spike-based approaches have over conventional methods, and under what circumstances does spike-based computing provide a comparative advantage? Simply implementing existing algorithms using spikes as the medium of computation and communication is not guaranteed to yield an advantage. Here, we demonstrate that spike-based communication and computation within algorithms can increase throughput, and they can decrease energy cost in some cases. We present several spiking algorithms, including sorting a set of numbers in ascending/descending order, as well as finding the maximum or minimum or median of a set of numbers. We also provide an example application: a spiking median-filtering approach for image processing providing a low-energy, parallel implementation. The algorithms and analyses presented here demonstrate that spiking algorithms can provide performance advantages and offer efficient computation of fundamental operations useful in more complex algorithms.

[1]  S. Amari,et al.  Competition and Cooperation in Neural Nets , 1982 .

[2]  Andrew D. Back,et al.  A spiking neural network architecture for nonlinear function approximation , 2001, Neural Networks.

[3]  Leslie G. Valiant,et al.  Parallelism in Comparison Problems , 1975, SIAM J. Comput..

[4]  J Gautrais,et al.  Rate coding versus temporal order coding: a theoretical approach. , 1998, Bio Systems.

[5]  Marius Usher,et al.  Dynamics of decision-making: from evidence accumulation to preference and belief , 2013, Front. Psychol..

[6]  Shih-Chii Liu,et al.  Computation with Spikes in a Winner-Take-All Network , 2009, Neural Computation.

[7]  B. Cessac A discrete time neural network model with spiking neurons , 2007, Journal of mathematical biology.

[8]  Neil Immerman,et al.  Expressibility and Parallel Complexity , 1989, SIAM J. Comput..

[9]  Ojas Parekh,et al.  Spiking network algorithms for scientific computing , 2016, 2016 IEEE International Conference on Rebooting Computing (ICRC).

[10]  Selim G. Akl,et al.  On the Power of some Pram Models , 1999, Parallel Algorithms Appl..

[11]  Bingni W. Brunton,et al.  Distinct relationships of parietal and prefrontal cortices to evidence accumulation , 2014, Nature.

[12]  H. Wu,et al.  Adaptive impulse detection using center-weighted median filters , 2001, IEEE Signal Processing Letters.

[13]  Conrado Martínez Partial Quicksort , 2003 .

[14]  Ojas Parekh,et al.  Constant-Depth and Subcubic-Size Threshold Circuits for Matrix Multiplication , 2018, SPAA.

[15]  William Softky Elastic Nanocomputation in an Ideal Brain (1p abstract + 36 pages + 49 endnotes) , 2014 .

[16]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[17]  Pekka Orponen,et al.  General-Purpose Computation with Neural Networks: A Survey of Complexity Theoretic Results , 2003, Neural Computation.

[18]  M. R. Mehta,et al.  Role of experience and oscillations in transforming a rate code into a temporal code , 2002, Nature.

[19]  Stanislas Dehaene,et al.  Accumulation of Evidence during Sequential Decision Making: The Importance of Top–Down Factors , 2010, The Journal of Neuroscience.

[20]  Charles K. Chui,et al.  A universal noise removal algorithm with an impulse detector , 2005, IEEE Transactions on Image Processing.

[21]  Chris Eliasmith,et al.  Developing and applying a toolkit from a general neurocomputational framework , 1999, Neurocomputing.

[22]  Cypher Valiant's maximum algorithm with sequential memory accesses. Technical report , 1988 .

[23]  Simon J. Thorpe,et al.  Spike arrival times: A highly efficient coding scheme for neural networks , 1990 .

[24]  William R. Softky Fine analog coding minimizes information transmission , 1996, Neural Networks.

[25]  Andrew S. Cassidy,et al.  Convolutional networks for fast, energy-efficient neuromorphic computing , 2016, Proceedings of the National Academy of Sciences.

[26]  Wolfgang Maass,et al.  Belief Propagation in Networks of Spiking Neurons , 2009, Neural Computation.

[27]  J. O’Keefe,et al.  Using hippocampal 'place cells' for navigation , 1993, Neural Information Processing Systems.

[28]  Jean-Pierre Rospars,et al.  REVIEW ARTICLE: Neuronal coding and spiking randomness , 2007, The European journal of neuroscience.

[29]  Sophie Denève,et al.  Spike-Based Population Coding and Working Memory , 2011, PLoS Comput. Biol..

[30]  Humberto Bustince,et al.  The median and its extensions , 2011, Fuzzy Sets Syst..

[31]  Richard Cole,et al.  A Parallel Median Algorithm , 1985, Inf. Process. Lett..

[32]  Robert A. Legenstein,et al.  Neuromorphic hardware in the loop: Training a deep spiking network on the BrainScaleS wafer-scale system , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[33]  L. F Abbott,et al.  Lapicque’s introduction of the integrate-and-fire model neuron (1907) , 1999, Brain Research Bulletin.

[34]  S. Grossberg Contour Enhancement , Short Term Memory , and Constancies in Reverberating Neural Networks , 1973 .

[35]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[36]  Youlian Zhu,et al.  An Improved Median Filtering Algorithm for Image Noise Reduction , 2012 .

[37]  Richard Cole,et al.  Parallel merge sort , 1988, 27th Annual Symposium on Foundations of Computer Science (sfcs 1986).

[38]  Richard M. Karp,et al.  Parallel Algorithms for Shared-Memory Machines , 1991, Handbook of Theoretical Computer Science, Volume A: Algorithms and Complexity.

[39]  Ronald L. Rivest,et al.  Introduction to Algorithms, 3rd Edition , 2009 .

[40]  John H. Reif,et al.  An optimal parallel algorithm for integer sorting , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[41]  Chris Eliasmith,et al.  Improving Spiking Dynamical Networks: Accurate Delays, Higher-Order Synapses, and Time Cells , 2018, Neural Computation.

[42]  Ryad Benosman,et al.  STICK: Spike Time Interval Computational Kernel, a Framework for General Purpose Computation Using Neurons, Precise Timing, Delays, and Synchrony , 2015, Neural Computation.

[43]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[44]  Wulfram Gerstner,et al.  Generalized integrate-and-fire models of neuronal activity approximate spike trains of a detailed model to a high degree of accuracy. , 2004, Journal of neurophysiology.

[45]  Sophie Denève,et al.  Computational Account of Spontaneous Activity as a Signature of Predictive Coding , 2017, PLoS Comput. Biol..

[46]  Selim G. Akl An Optimal Algorithm for Parallel Selection , 1984, Inf. Process. Lett..

[47]  Christian K. Machens,et al.  Predictive Coding of Dynamical Variables in Balanced Spiking Networks , 2013, PLoS Comput. Biol..

[48]  Chris Eliasmith,et al.  Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems , 2004, IEEE Transactions on Neural Networks.

[49]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[50]  J. J. Hopfield,et al.  Pattern recognition computation using action potential timing for stimulus representation , 1995, Nature.

[51]  Sophie Denève,et al.  Bayesian Spiking Neurons I: Inference , 2008, Neural Computation.

[52]  C. Anderson,et al.  Local limit theorems for the maxima of discrete random variables , 1980, Mathematical Proceedings of the Cambridge Philosophical Society.

[53]  Wolfgang Maass,et al.  On the Computational Power of Winner-Take-All , 2000, Neural Computation.

[54]  Pieter R. Roelfsema,et al.  How Precise is Neuronal Synchronization? , 1995, Neural Computation.

[55]  Eisenstat C. Eisenstat O(log*n) algorithms on a Sum-CRCW PRAM , 2006, Computing.

[56]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[57]  A V Herz,et al.  Neural codes: firing rates and beyond. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[58]  Sander M. Bohte,et al.  Computing with Spiking Neuron Networks , 2012, Handbook of Natural Computing.

[59]  S. Thorpe,et al.  Spike times make sense , 2005, Trends in Neurosciences.

[60]  Wolfgang Maass,et al.  Computing with spiking neurons , 1999 .

[61]  Christian K. Machens,et al.  Efficient codes and balanced networks , 2016, Nature Neuroscience.

[62]  François W. Primeau,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014 .

[63]  Wolfgang Maass,et al.  Networks of Spiking Neurons: The Third Generation of Neural Network Models , 1996, Electron. Colloquium Comput. Complex..

[64]  Allan Borodin,et al.  Routing, Merging, and Sorting on Parallel Models of Computation , 1985, J. Comput. Syst. Sci..

[65]  T. Kailath,et al.  Discrete Neural Computation: A Theoretical Foundation , 1995 .

[66]  Alexander Tiskin,et al.  Parallel Selection by Regular Sampling , 2010, Euro-Par.

[67]  Conrad D. James,et al.  Optimization-based computation with spiking neurons , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[68]  Uzi Vishkin,et al.  Finding the Maximum, Merging, and Sorting in a Parallel Computation Model , 1981, J. Algorithms.

[69]  Arnaud Delorme,et al.  Spike-based strategies for rapid processing , 2001, Neural Networks.

[70]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[71]  Eugene M. Izhikevich,et al.  Polychronization: Computation with Spikes , 2006, Neural Computation.

[72]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[73]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1990 .

[74]  James Christopher Wyllie,et al.  The Complexity of Parallel Computations , 1979 .

[75]  Fred Rothganger,et al.  A Spike-Timing Neuromorphic Architecture , 2017, 2017 IEEE International Conference on Rebooting Computing (ICRC).

[76]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).