Resilience and Robustness of Spiking Neural Networks for Neuromorphic Systems

Though robustness and resilience are commonly quoted as features of neuromorphic computing systems, the expected performance of neuromorphic systems in the face of hardware failures is not clear. In this work, we study the effect of failures on the performance of four different training algo-rithms for spiking neural networks on neuromorphic systems: two back-propagation-based training approaches (Whetstone and SLAYER), a liquid state machine or reservoir computing approach, and an evolutionary optimization-based approach (EONS). We show that these four different approaches have very different resilience characteristics with respect to simulated hardware failures. We then analyze an approach for training more resilient spiking neural networks using the evolutionary optimization approach. We show how this approach produces more resilient networks and discuss how it can be extended to other spiking neural network training approaches as well.

[1]  Gerd Ascheid,et al.  Accurate neuron resilience prediction for a flexible reliability management in neural network accelerators , 2018, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[2]  Krishnendu Chakrabarty,et al.  Design of fault-tolerant neuromorphic computing systems , 2018, 2018 IEEE 23rd European Test Symposium (ETS).

[3]  Gerd Ascheid,et al.  Automated design of error-resilient and hardware-efficient deep neural networks , 2019, Neural Computing and Applications.

[4]  Catherine D. Schuman,et al.  Intelligent Reservoir Generation for Liquid State Machines using Evolutionary Optimization , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[5]  Kishan G. Mehrotra,et al.  Training techniques to obtain fault-tolerant neural networks , 1994, Proceedings of IEEE 24th International Symposium on Fault- Tolerant Computing.

[6]  Krishnendu Chakrabarty,et al.  Fault Tolerance in Neuromorphic Computing Systems , 2019, 2019 24th Asia and South Pacific Design Automation Conference (ASP-DAC).

[7]  Xu Sun,et al.  meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting , 2017, ICML.

[8]  Yogesh Singh,et al.  Fault tolerance of feedforward artificial neural networks- a framework of study , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[9]  Catherine D. Schuman,et al.  Non-Traditional Input Encoding Schemes for Spiking Neuromorphic Systems , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[10]  Frank Hutter,et al.  Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution , 2018, ICLR.

[11]  Catherine D. Schuman,et al.  Design of Superconducting Optoelectronic Networks for Neuromorphic Computing , 2018, 2018 IEEE International Conference on Rebooting Computing (ICRC).

[12]  Catherine D. Schuman,et al.  A Soft-Matter Biomolecular Memristor Synapse for Neuromorphic Systems , 2018, 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[13]  Chenchen Liu,et al.  Rescuing memristor-based neuromorphic design with high defects , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[14]  Garrick Orchard,et al.  SLAYER: Spike Layer Error Reassignment in Time , 2018, NeurIPS.

[15]  Catherine D. Schuman,et al.  167-PFlops Deep Learning for Electron Microscopy: From Learning Physics to Atomic Manipulation , 2018, SC18: International Conference for High Performance Computing, Networking, Storage and Analysis.

[16]  Gerd Ascheid,et al.  An Efficient Bit-Flip Resilience Optimization Method for Deep Neural Networks , 2019, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[17]  Pritish Narayanan,et al.  Neuromorphic computing using non-volatile memory , 2017 .

[18]  Gu-Yeon Wei,et al.  Ares: A framework for quantifying the resilience of deep neural networks , 2018, 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC).

[19]  Catherine D. Schuman,et al.  The TENNLab Suite of LIDAR-Based Control Applications for Recurrent , Spiking , Neuromorphic Systems , 2019 .

[20]  Catherine D. Schuman,et al.  An evolutionary optimization framework for neural networks and neuromorphic architectures , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[21]  Guanpeng Li,et al.  Understanding Error Propagation in Deep Learning Neural Network (DNN) Accelerators and Applications , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[22]  Minjae Lee,et al.  Fault tolerance analysis of digital feed-forward deep neural networks , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Chris Yakopcic,et al.  Tolerance to defective memristors in a neuromorphic learning circuit , 2014, NAECON 2014 - IEEE National Aerospace and Electronics Conference.

[24]  Catherine D. Schuman,et al.  A Survey of Neuromorphic Computing and Neural Networks in Hardware , 2017, ArXiv.

[25]  B. E. Segee,et al.  Fault tolerance of pruned multilayer networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[26]  Craig M. Vineyard,et al.  Training deep neural networks for binary communication with the Whetstone method , 2019 .

[27]  Catherine D. Schuman,et al.  Memristive Mixed-Signal Neuromorphic Systems: Energy-Efficient Learning at the Circuit-Level , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[28]  Catherine D. Schuman,et al.  Multi-Objective Optimization for Size and Resilience of Spiking Neural Networks , 2019, 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON).

[29]  A Learning Algorithm for Fault Tolerant Feedforward Neural Networks , 1996 .

[30]  Zhi-Hua Zhou,et al.  Evolving Fault-Tolerant Neural Networks , 2003, Neural Computing & Applications.