Reliability-Driven Neural Network Training for Memristive Crossbar-Based Neuromorphic Computing Systems

In recent years, memristive crossbar-based neuromorphic computing systems (NCS) have provided a promising solution to the acceleration of neural networks. However, stuck-at faults (SAFs) in the memristor devices significantly degrade the computing accuracy of NCS. Besides, the memristor suffers from the process variations, causing deviation of the actual programming resistance from its target resistance. In this paper, we propose a reliability-driven network training framework for a memristive crossbar-based NCS, with taking account of both SAFs and device variations challenges. A dropout-inspired approach is first developed to alleviate the impact of SAFs. A new weighted error function, including cross-entropy error (CEE), the l2-norm of weights, and the sum of squares of first-order derivatives of CEE with respect to weights, is further proposed to obtain a smooth error curve, where the effects of variations are suppressed. Experimental results show that the proposed method can boost the computation accuracy of NCS and improve the NCS robustness.

[1]  Yu Wang,et al.  Computation-oriented fault-tolerance schemes for RRAM computing systems , 2017, 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC).

[2]  Chenchen Liu,et al.  Rescuing memristor-based neuromorphic design with high defects , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[3]  Yiran Chen,et al.  Neuromorphic computing's yesterday, today, and tomorrow - an evolutional view , 2018, Integr..

[4]  Yu Wang,et al.  Stuck-at Fault Tolerance in RRAM Computing Systems , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[5]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[6]  Xuefei Ning,et al.  Fault-tolerant training with on-line fault detection for RRAM-based neural computing systems , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[7]  Yiran Chen,et al.  A quantization-aware regularized learning method in multilevel memristor-based neuromorphic computing system , 2017, 2017 IEEE 6th Non-Volatile Memory Systems and Applications Symposium (NVMSA).

[8]  Yiran Chen,et al.  Vortex: Variation-aware training for memristor X-bar , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[9]  A. Asenov,et al.  Intrinsic parameter fluctuations in decananometer MOSFETs introduced by gate line edge roughness , 2003 .

[10]  Feng Wu,et al.  Memristive Crossbar Mapping for Neuromorphic Computing Systems on 3D IC , 2018, ACM Great Lakes Symposium on VLSI.

[11]  U-In Chung,et al.  Multi-level switching of triple-layered TaOx RRAM with excellent reliability for storage class memory , 2012, 2012 Symposium on VLSI Technology (VLSIT).

[12]  Yiran Chen,et al.  An EDA framework for large scale hybrid neuromorphic computing systems , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[13]  Yervant Zoriaii.  Effective March Algorithms for Testing Single-Order Addressed Memories , 1993 .

[14]  Yiran Chen,et al.  Accelerator-friendly neural-network training: Learning variations and defects in RRAM crossbar , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[15]  Chilukuri K. Mohan,et al.  Modifying training algorithms for improved fault tolerance , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[16]  Xuefei Ning,et al.  Fault-Tolerant Training Enabled by On-Line Fault Detection for RRAM-Based Neural Computing Systems , 2019, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[17]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[20]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[21]  Gian Carlo Cardarilli,et al.  Hardware design of LIF with Latency neuron model with memristive STDP synapses , 2017, Integr..

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[24]  Frederick T. Chen,et al.  RRAM Defect Modeling and Failure Analysis Based on March Test and a Novel Squeeze-Search Scheme , 2015, IEEE Transactions on Computers.

[25]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.