Efficient training algorithms for neural networks based on memristive crossbar circuits

We have adapted backpropagation algorithm for training multilayer perceptron classifier implemented with memristive crossbar circuits. The proposed training approach takes into account switching dynamics of a particular, though very typical, type of memristive devices and weight update restrictions imposed by crossbar topology. The simulation results show that for crossbar-based multilayer perceptron with one hidden layer of 300 neurons misclassification rate on MNIST benchmark could be as low as 1.47% and 4.06% for batch and stochastic algorithms, respectively, which is comparable to the best reported results for similar neural networks.

[1]  Konstantin K. Likharev,et al.  Defect‐tolerant nanoelectronic pattern classifiers , 2007, Int. J. Circuit Theory Appl..

[2]  V. Cros,et al.  Spin-torque building blocks. , 2014, Nature Materials.

[3]  Yong Zhang,et al.  A digital neuromorphic VLSI architecture with memristor crossbar synaptic array for machine learning , 2012, 2012 IEEE International SOC Conference.

[4]  Byoungil Lee,et al.  Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing. , 2012, Nano letters.

[5]  Steven R. Young,et al.  A 1 TOPS/W Analog Deep Machine-Learning Engine With Floating-Gate Storage in 0.13 µm CMOS , 2014, IEEE Journal of Solid-State Circuits.

[6]  D. Strukov,et al.  Resistive switching phenomena in thin films: Materials, devices, and applications , 2012 .

[7]  Dharmendra S. Modha,et al.  A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[8]  R. Williams,et al.  How We Found The Missing Memristor , 2008, IEEE Spectrum.

[9]  Wei D. Lu,et al.  Pattern recognition with memristor networks , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[10]  Wolfram Schiffmann,et al.  Optimization of the Backpropagation Algorithm for Training Multilayer Perceptrons , 1994 .

[11]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[12]  Fabien Alibart,et al.  Pattern classification by memristive crossbar circuits using ex situ and in situ training , 2013, Nature Communications.

[13]  Jennifer Hasler,et al.  Vector-Matrix Multiply and Winner-Take-All as an Analog Classifier , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[14]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[15]  Konstantin K. Likharev,et al.  CrossNets: Neuromorphic Hybrid CMOS/Nanoelectronic Networks , 2011 .

[16]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[17]  T. Hasegawa,et al.  Short-term plasticity and long-term potentiation mimicked in single inorganic synapses. , 2011, Nature materials.

[18]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[19]  Ligang Gao,et al.  High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm , 2011, Nanotechnology.

[20]  Jacques-Olivier Klein,et al.  Robust learning approach for neuro-inspired nanoscale crossbar architecture , 2014, ACM J. Emerg. Technol. Comput. Syst..

[21]  Konstantin K. Likharev,et al.  Hybrid CMOS/Nanoelectronic Circuits: Opportunities and Challenges , 2008 .

[22]  Srihari Cadambi,et al.  A dynamically configurable coprocessor for convolutional neural networks , 2010, ISCA.

[23]  Berin Martini,et al.  NeuFlow: A runtime reconfigurable dataflow processor for vision , 2011, CVPR 2011 WORKSHOPS.

[24]  Dmitri B. Strukov,et al.  Nanotechnology: Smart connections , 2011, Nature.

[25]  Chris Yakopcic,et al.  Energy efficient perceptron pattern recognition using segmented memristor crossbar arrays , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[26]  J Joshua Yang,et al.  Memristive devices for computing. , 2013, Nature nanotechnology.

[27]  Yiran Chen,et al.  Memristor Crossbar-Based Neuromorphic Computing System: A Case Study , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[28]  D. Strukov,et al.  Phenomenological modeling of memristive devices , 2014, 1406.4219.

[29]  R. Schaller,et al.  Technological innovation in the semiconductor industry: A case study of the International Technology Roadmap for Semiconductors (ITRS) , 2001, PICMET '01. Portland International Conference on Management of Engineering and Technology. Proceedings Vol.1: Book of Summaries (IEEE Cat. No.01CH37199).

[30]  Damien Querlioz,et al.  Simulation of a memristor-based spiking neural network immune to device variations , 2011, The 2011 International Joint Conference on Neural Networks.

[31]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[32]  Yukihiro Kaneko,et al.  Ferroelectric Artificial Synapses for Recognition of a Multishaded Image , 2014, IEEE Transactions on Electron Devices.

[33]  Kunle Olukotun,et al.  A highly scalable Restricted Boltzmann Machine FPGA implementation , 2009, 2009 International Conference on Field Programmable Logic and Applications.

[34]  Janusz A. Starzyk,et al.  Memristor Crossbar Architecture for Synchronous Neural Networks , 2014, IEEE Transactions on Circuits and Systems I: Regular Papers.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[37]  Hoi-Jun Yoo,et al.  A 201.4 GOPS 496 mW Real-Time Multi-Object Recognition Processor With Bio-Inspired Neural Perception Engine , 2009, IEEE Journal of Solid-State Circuits.