Stochastic Computing Can Improve Upon Digital Spiking Neural Networks

With the surge in popularity of machine learning algorithms, research has turned towards exploring novel computing architectures in order to increase performance while limiting power consumption. Inspired by their biological counterparts, digital spiking neural networks have emerged as energy efficient alternatives to conventional hardware implementations, yet remain largely incompatible with cutting edge learning methods. Representing information with single-bit binary pulse trains, the behaviour of spiking neural networks exhibit many interesting analogues to the existing field of stochastic computing. In this paper, we not only illustrate the parallels between digital spiking neural networks and stochastic computing, but we also demonstrate that many computing elements in modern spiking hardware are, in fact, implementations of stochastic circuits. In addition, we show that stochastic computing design techniques can be leveraged in order to address shortcomings in current spiking neural network architectures.

[1]  Andrew S. Cassidy,et al.  Building block of a programmable neuromorphic substrate: A digital neurosynaptic core , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[2]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[3]  Kia Bazargan,et al.  The synthesis of complex arithmetic computation on stochastic bit streams using sequential logic , 2012, 2012 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[4]  Mikko H. Lipasti,et al.  Bridging the semantic gap: Emulating biological neuronal behaviors with simple digital neurons , 2013, 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA).

[5]  W. J. Poppelbaum,et al.  Stochastic computing elements and systems , 1967, AFIPS '67 (Fall).

[6]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[7]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[8]  Brian R. Gaines,et al.  Stochastic computing , 1967, AFIPS '67 (Spring).

[9]  Andrew S. Cassidy,et al.  Real-Time Scalable Cortical Computing at 46 Giga-Synaptic OPS/Watt with ~100× Speedup in Time-to-Solution and ~100,000× Reduction in Energy-to-Solution , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[10]  Dharmendra S. Modha,et al.  A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[11]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  Shih-Chii Liu,et al.  Minitaur, an Event-Driven FPGA-Based Spiking Network Accelerator , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[13]  Kia Bazargan,et al.  The synthesis of linear Finite State Machine-based Stochastic Computational Elements , 2012, 17th Asia and South Pacific Design Automation Conference.

[14]  Eugene M. Izhikevich,et al.  Which model to use for cortical spiking neurons? , 2004, IEEE Transactions on Neural Networks.

[15]  Howard C. Card,et al.  Stochastic Neural Computation I: Computational Elements , 2001, IEEE Trans. Computers.

[16]  Andrew S. Cassidy,et al.  Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[17]  Yiran Chen,et al.  A new learning method for inference accuracy, core occupation, and performance co-optimization on TrueNorth chip , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[18]  Andrew S. Cassidy,et al.  Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware , 2016, 2016 IEEE International Conference on Rebooting Computing (ICRC).

[19]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[20]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[21]  John P. Hayes,et al.  STRAUSS: Spectral Transform Use in Stochastic Circuit Synthesis , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[22]  Andrew S. Cassidy,et al.  Cognitive computing systems: Algorithms and applications for networks of neurosynaptic cores , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[23]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.