Mixed-precision training of deep neural networks using computational memory

Deep neural networks have revolutionized the field of machine learning by providing unprecedented human-like performance in solving many real-world problems such as image and speech recognition. Training of large DNNs, however, is a computationally intensive task, and this necessitates the development of novel computing architectures targeting this application. A computational memory unit where resistive memory devices are organized in crossbar arrays can be used to locally store the synaptic weights in their conductance states. The expensive multiply accumulate operations can be performed in place using Kirchhoff's circuit laws in a non-von Neumann manner. However, a key challenge remains the inability to alter the conductance states of the devices in a reliable manner during the weight update process. We propose a mixed-precision architecture that combines a computational memory unit storing the synaptic weights with a digital processing unit and an additional memory unit accumulating weight updates in high precision. The new architecture delivers classification accuracies comparable to those of floating-point implementations without being constrained by challenges associated with the non-ideal weight update characteristics of emerging resistive memories. A two layer neural network in which the computational memory unit is realized using non-linear stochastic models of phase-change memory devices achieves a test accuracy of 97.40% on the MNIST handwritten digit classification problem.

[1]  Olivier Bichler,et al.  Phase change memory as synapse for ultra-dense neuromorphic systems: Application to complex visual pattern extraction , 2011, 2011 International Electron Devices Meeting.

[2]  M. Breitwisch Phase Change Memory , 2008, 2008 International Interconnect Technology Conference.

[3]  Hyung-Min Lee,et al.  Analog CMOS-based resistive processing unit for deep neural network training , 2017, 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS).

[4]  Andrew D Kent,et al.  A new spin on magnetic memories. , 2015, Nature nanotechnology.

[5]  Catherine Graves,et al.  Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[6]  E. Eleftheriou,et al.  All-memristive neuromorphic computing with level-tuned neurons , 2016, Nanotechnology.

[7]  Pritish Narayanan,et al.  Experimental Demonstration and Tolerancing of a Large-Scale Neural Network (165 000 Synapses) Using Phase-Change Memory as the Synaptic Weight Element , 2014, IEEE Transactions on Electron Devices.

[8]  Gökmen Tayfun,et al.  Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations , 2016, Front. Neurosci..

[9]  Haralampos Pozidis,et al.  Programming algorithms for multilevel phase-change memory , 2011, 2011 IEEE International Symposium of Circuits and Systems (ISCAS).

[10]  D. Ielmini,et al.  Distributed-Poole-Frenkel modeling of anomalous resistance scaling and fluctuations in phase-change memory (PCM) devices , 2009, 2009 IEEE International Electron Devices Meeting (IEDM).

[11]  Wei D. Lu,et al.  Sparse coding with memristor networks. , 2017, Nature nanotechnology.

[12]  B. Rajendran,et al.  Supervised learning in spiking neural networks with MLC PCM synapses , 2017, 2017 75th Annual Device Research Conference (DRC).

[13]  Dan Alistarh,et al.  ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning , 2017, ICML.

[14]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[15]  Thomas P. Parnell,et al.  Temporal correlation detection using computational phase-change memory , 2017, Nature Communications.

[16]  Yusuf Leblebici,et al.  Stochastic weight updates in phase-change memory-based synapses and their influence on artificial neural networks , 2017, 2017 13th Conference on Ph.D. Research in Microelectronics and Electronics (PRIME).

[17]  Giacomo Indiveri,et al.  Rounding Methods for Neural Networks with Low Resolution Synaptic Weights , 2015, ArXiv.

[18]  Onur Mutlu,et al.  Buddy-RAM: Improving the Performance and Efficiency of Bulk Bitwise Operations Using DRAM , 2016, ArXiv.

[19]  Evangelos Eleftheriou,et al.  Detecting Correlations Using Phase-Change Neurons and Synapses , 2016, IEEE Electron Device Letters.

[20]  Pritish Narayanan,et al.  Deep Learning with Limited Numerical Precision , 2015, ICML.

[21]  M. Kozicki,et al.  Electrochemical metallization memories—fundamentals, applications, prospects , 2011, Nanotechnology.

[22]  H-S Philip Wong,et al.  Memory leads the way to better computing. , 2015, Nature nanotechnology.

[23]  Pritish Narayanan,et al.  Neuromorphic computing using non-volatile memory , 2017 .

[24]  Damien Querlioz,et al.  Simulation of a memristor-based spiking neural network immune to device variations , 2011, The 2011 International Joint Conference on Neural Networks.

[25]  Shimeng Yu,et al.  Metal–Oxide RRAM , 2012, Proceedings of the IEEE.

[26]  Daniel Krebs,et al.  Crystal growth within a phase change memory cell , 2014, Nature Communications.

[27]  Dharmendra S. Modha,et al.  Deep neural networks are robust to weight binarization and other non-linear distortions , 2016, ArXiv.

[28]  Manuel Le Gallo,et al.  Stochastic phase-change neurons. , 2016, Nature nanotechnology.

[29]  A. Sebastian,et al.  Compressed sensing recovery using computational memory , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[30]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[31]  Leon O. Chua Resistance switching memories are memristors , 2011 .

[32]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[33]  G. W. Burr,et al.  Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element , 2015, 2014 IEEE International Electron Devices Meeting.

[34]  Heiner Giefers,et al.  Mixed-precision in-memory computing , 2017, Nature Electronics.

[35]  Catherine Dubourdieu,et al.  A 250 mV Cu/SiO2/W Memristor with Half-Integer Quantum Conductance States. , 2016, Nano letters.