Energy-efficient Design of MTJ-based Neural Networks with Stochastic Computing

Hardware implementations of Artificial Neural Networks (ANNs) using conventional binary arithmetic units are computationally expensive, energy-intensive, and have large area overheads. Stochastic Computing (SC) is an emerging paradigm that replaces these conventional units with simple logic circuits and is particularly suitable for fault-tolerant applications. We propose an energy-efficient use of Magnetic Tunnel Junctions (MTJs), a spintronic device that exhibits probabilistic switching behavior, as Stochastic Number Generators (SNGs), which forms the basis of our NN implementation in the SC domain. Further, the error resilience of target applications of NNs allows approximating the synaptic weights in our MTJ-based NN implementation, in ways brought about by properties of the MTJ-SNG, to achieve energy-efficiency. An algorithm is designed that, given an error tolerance, can perform such approximations in a single-layer NN in an optimal way owing to the convexity of the problem formulation. We then use this algorithm and develop a heuristic approach for approximating multi-layer NNs. Classification problems were evaluated on the optimized NNs and results showed substantial savings in energy for little loss in accuracy.

[1]  Jian-Gang Zhu,et al.  Magnetic tunnel junctions , 2006 .

[2]  Kaushik Roy,et al.  Spintastic: Spin-based stochastic logic for energy-efficient computing , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[3]  Shouyi Yin,et al.  Optimization of Softmax Layer in Deep Neural Network Using Integral Stochastic Computation , 2018, J. Low Power Electron..

[4]  Z. Diao,et al.  Spin-transfer torque switching in magnetic tunnel junctions and spin-transfer torque random access memory , 2007 .

[5]  Naoya Onizawa,et al.  VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing , 2017, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[6]  Fabrizio Lombardi,et al.  An energy-efficient stochastic computational deep belief network , 2018, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[7]  Kiyoung Choi,et al.  Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[8]  Lirida A. B. Naviner,et al.  A novel circuit design of true random number generator using magnetic tunnel junction , 2016, 2016 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).

[9]  Kaushik Roy,et al.  Design of power-efficient approximate multipliers for approximate artificial neural networks , 2016, 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[10]  Apostolos Dollas,et al.  Unary Processing , 1987, Adv. Comput..

[11]  A. Naeemi,et al.  A Probability-Density Function Approach to Capture the Stochastic Dynamics of the Nanomagnet and Impact on Circuit Performance , 2016, IEEE Transactions on Electron Devices.

[12]  W. J. Poppelbaum.  Statistical Processors , 1976, Adv. Comput..

[13]  Qiang Xu,et al.  ApproxANN: An approximate computing framework for artificial neural network , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[14]  Yiran Chen,et al.  Spintronics based stochastic computing for efficient Bayesian inference system , 2017, 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC).

[15]  Ji Li,et al.  Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[16]  2013 , 2018, Eu minha tía e o golpe do atraso.

[17]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[18]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[19]  Howard C. Card,et al.  Stochastic Neural Computation I: Computational Elements , 2001, IEEE Trans. Computers.

[20]  Weisheng Zhao,et al.  Multi-level cell Spin Transfer Torque MRAM based on stochastic switching , 2013, 2013 13th IEEE International Conference on Nanotechnology (IEEE-NANO 2013).

[21]  S. Parkin,et al.  Magnetic Tunnel Junctions , 2007 .

[22]  Kang L. Wang,et al.  Low-power non-volatile spintronic memory: STT-RAM and beyond , 2013 .

[23]  Brian R. Gaines,et al.  Stochastic Computing Systems , 1969 .

[24]  Lirida A. B. Naviner,et al.  Stochastic computation with Spin Torque Transfer Magnetic Tunnel Junction , 2015, 2015 IEEE 13th International New Circuits and Systems Conference (NEWCAS).

[25]  Y Suzuki,et al.  High-Speed Spin-Transfer Switching in GMR Nano-Pillars With Perpendicular Anisotropy , 2011, IEEE Transactions on Magnetics.

[26]  Guangyu Sun,et al.  PDS: Pseudo-differential sensing scheme for STT-MRAM , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[27]  Youguang Zhang,et al.  Stochastic spintronic device based synapses and spiking neurons for neuromorphic computation , 2016, 2016 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).

[28]  Kiyoung Choi,et al.  Approximate de-randomizer for stochastic circuits , 2015, 2015 International SoC Design Conference (ISOCC).

[29]  Jie Han,et al.  Approximate computing: An emerging paradigm for energy-efficient design , 2013, 2013 18th IEEE European Test Symposium (ETS).

[30]  Marian Verhelst,et al.  Energy-Efficiency and Accuracy of Stochastic Computing Circuits in Emerging Technologies , 2014, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[31]  A. James.  2010 , 2011, Philo of Alexandria: an Annotated Bibliography 2007-2016.

[32]  Yiran Chen,et al.  Asymmetry of MTJ switching and its implication to STT-RAM designs , 2012, 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[33]  David J. Lilja,et al.  Neural Network Classifiers Using a Hardware-Based Approximate Activation Function with a Hybrid Stochastic Multiplier , 2019, ACM J. Emerg. Technol. Comput. Syst..

[34]  Yiran Chen,et al.  SPINBIS: Spintronics-Based Bayesian Inference System With Stochastic Computing , 2019, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[35]  Qing Wu,et al.  A novel true random number generator based on a stochastic diffusive memristor , 2017, Nature Communications.

[36]  M. S. Tarkov.  Mapping weight matrix of a neural network’s layer onto memristor crossbar , 2015, Optical Memory and Neural Networks.

[37]  Jiantao Zhou,et al.  Stochastic Memristive Devices for Computing and Neuromorphic Applications , 2013, Nanoscale.

[38]  C. Martin.  2015 , 2015, Les 25 ans de l’OMC: Une rétrospective en photos.

[39]  G. Cauwenberghs,et al.  Memristor-based neural networks: Synaptic versus neuronal stochasticity , 2016 .

[40]  Hai Li,et al.  Classification accuracy improvement for neuromorphic computing systems with one-level precision synapses , 2017, 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC).

[41]  Terrence J. Sejnowski,et al.  Analysis of hidden units in a layered network trained to classify sonar targets , 1988, Neural Networks.

[42]  Z. Li,et al.  Magnetization dynamics with a spin-transfer torque , 2003 .

[43]  Dejan Markovic,et al.  Design of a Fast and Low-Power Sense Amplifier and Writing Circuit for High-Speed MRAM , 2015, IEEE Transactions on Magnetics.

[44]  John P. Hayes,et al.  Survey of Stochastic Computing , 2013, TECS.

[45]  Kaushik Roy,et al.  Magnetic tunnel junction enabled all-spin stochastic spiking neural network , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[46]  David J. Lilja,et al.  Low-Cost Stochastic Hybrid Multiplier for Quantized Neural Networks , 2019, ACM J. Emerg. Technol. Comput. Syst..

[47]  Catherine Graves,et al.  Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[48]  Xin Li,et al.  An Architecture for Fault-Tolerant Computation with Stochastic Logic , 2011, IEEE Transactions on Computers.

[49]  Sandhya Samarasinghe,et al.  Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition , 2006 .

[50]  Zhengya Zhang,et al.  A Native Stochastic Computing Architecture Enabled by Memristors , 2014, IEEE Transactions on Nanotechnology.

[51]  Josep L. Rosselló,et al.  A New Stochastic Computing Methodology for Efficient Neural Network Implementation , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[52]  Chris H. Kim,et al.  A technology-agnostic MTJ SPICE model with user-defined dimensions for STT-MRAM scalability studies , 2015, 2015 IEEE Custom Integrated Circuits Conference (CICC).

[53]  Yu Wang,et al.  The stochastic modeling of TiO2 memristor and its usage in neuromorphic system design , 2014, 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC).

[54]  H. Ohno,et al.  A perpendicular-anisotropy CoFeB-MgO magnetic tunnel junction. , 2010, Nature materials.

[55]  Ankur Srivastava,et al.  Power optimizations in MTJ-based Neural Networks through Stochastic Computing , 2017, 2017 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[56]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[57]  Yiming Huai,et al.  Spin-Transfer Torque MRAM (STT-MRAM): Challenges and Prospects , 2008 .

[58]  Denis Gifford,et al.  1976 , 2018, The British Film Catalogue.

[59]  Pai-Shun Ting,et al.  Stochastic Logic Realization of Matrix Operations , 2014, 2014 17th Euromicro Conference on Digital System Design.

[60]  Nikhil Rangarajan,et al.  Energy-Efficient Computing With Probabilistic Magnetic Bits—Performance Modeling and Comparison Against Probabilistic CMOS Logic , 2017, IEEE Transactions on Magnetics.

[61]  Tianshi Chen,et al.  DaDianNao: A Neural Network Supercomputer , 2017, IEEE Transactions on Computers.

[62]  Sachin S. Sapatnekar,et al.  Spin-Based Computing: Device Concepts, Current Status, and a Case Study on a High-Performance Microprocessor , 2015, Proceedings of the IEEE.

[63]  Kaushik Roy,et al.  AxNN: Energy-efficient neuromorphic systems using approximate computing , 2014, 2014 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[64]  Paulo Cortez,et al.  Modeling wine preferences by data mining from physicochemical properties , 2009, Decis. Support Syst..

[65]  Hiroshi Imamura,et al.  Spin dice: A scalable truly random number generator based on spintronics , 2014 .

[66]  Ji Li,et al.  DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks , 2016, 2016 IEEE 34th International Conference on Computer Design (ICCD).