VCAM: Variation Compensation through Activation Matching for Analog Binarized Neural Networks

We propose an energy-efficient analog implementation of binarized neural network with a novel technique called VCAM, variation compensation through activation matching. The architecture consists of 1T1R ReRAM arrays and differential amplifiers for implementing synapses and neurons, respectively. To restore classification test accuracy degraded by process variation, we adjust the biases of the neurons to match their average output activations with those of ideal neurons. Experimental results show that the proposed approach recovers the accuracy to 98.55% on MNIST and 89.63% on CIFAR-10 even in the presence of 50% threshold voltage and 15% resistance variations at 3-sigma point. This result corresponds to the accuracy degradation of only 0.05% and 1.35%, respectively, compared to the ideal case.

[1]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[2]  Sparsh Mittal,et al.  A Survey of ReRAM-Based Architectures for Processing-In-Memory and Neural Networks , 2018, Mach. Learn. Knowl. Extr..

[3]  Jagan Singh Meena,et al.  Overview of emerging nonvolatile memory technologies , 2014, Nanoscale Research Letters.

[4]  Marian Verhelst,et al.  An always-on 3.8μJ/86% CIFAR-10 mixed-signal binary CNN processor with all memory on chip in 28nm CMOS , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[5]  Shuchang Zhou,et al.  DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients , 2016, ArXiv.

[6]  Siddharth Garg,et al.  Addressing Process Variations at the Microarchitecture and System Level , 2013, Found. Trends Electron. Des. Autom..

[7]  Xiaochen Peng,et al.  Fully parallel RRAM synaptic array for implementing binary neural network with (+1, −1) weights and (+1, 0) neurons , 2018, 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC).

[8]  Dharmendra S. Modha,et al.  Deep neural networks are robust to weight binarization and other non-linear distortions , 2016, ArXiv.

[9]  Borivoje Nikolic,et al.  Measurement and Analysis of Variability in 45 nm Strained-Si CMOS Technology , 2009, IEEE Journal of Solid-State Circuits.

[10]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[11]  Yu Wang,et al.  Binary convolutional neural network on RRAM , 2017, 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC).

[12]  Hao Yu,et al.  An energy-efficient and high-throughput bitwise CNN on sneak-path-free digital ReRAM crossbar , 2017, 2017 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Xiaochen Peng,et al.  XNOR-RRAM: A scalable and parallel resistive synaptic architecture for binary neural networks , 2018, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[15]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[16]  Eriko Nurvitadhi,et al.  Accelerating Binarized Neural Networks: Comparison of FPGA, CPU, GPU, and ASIC , 2016, 2016 International Conference on Field-Programmable Technology (FPT).

[17]  Yingchieh Ho,et al.  High-R Poly Resistance Deviation Improvement From Suppressions of Back-End Mechanical Stresses , 2017, IEEE Transactions on Electron Devices.

[18]  Jun Deguchi,et al.  A Neuromorphic Chip Optimized for Deep Learning and CMOS Technology With Time-Domain Analog and Digital Mixed-Signal Processing , 2017, IEEE Journal of Solid-State Circuits.

[19]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[20]  James D. Meindl,et al.  Impact of die-to-die and within-die parameter fluctuations on the maximum clock frequency distribution for gigascale integration , 2002, IEEE J. Solid State Circuits.

[21]  Jia Wang,et al.  DaDianNao: A Machine-Learning Supercomputer , 2014, 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture.

[22]  H-S Philip Wong,et al.  Multi-level control of conductive nano-filament evolution in HfO2 ReRAM by pulse-train operations. , 2014, Nanoscale.