Flexible memristor based neuromorphic system for implementing multi-layer neural network algorithms

Abstract This paper describes a memristor-based neuromorphic system that can be used for ex situ training of various multi-layer neural network algorithms. This system is based on an analogue neuron circuit that is capable of performing an accurate dot product calculation. The presented ex situ programming technique can be used to map many key neural algorithms directly onto the grid of resistances in a memristor crossbar. Using this weight-to-crossbar mapping approach along with the memristor based circuit architecture, complex neural algorithms can be easily implemented using this system. Some existing memristor based circuits provide an approximated dot product based on conductance summation, but neuron outputs are not directly correlated to the numerical values obtained in a traditional software approach. To show the effectiveness and versatility of this circuit, two different powerful neural networks were simulated. These include a Restricted Boltzmann Machine for character recognition and a Multilayer Perceptron trained to perform Sobel edge detection. Following these simulations, an analysis was presented that shows how both memristor accuracy and neuron circuit gain relates to output error. This work presents a novel memristor based architecture that that is capable of implementing multiple different learning algorithms using the same hardware, which is based on crossbar structures such as the one displayed. The example presented shows the result of the memristor architecture when implementing Sobel edge detection using a multilayer perceptron.

[1]  Fabien Alibart,et al.  Pattern classification by memristive crossbar circuits using ex situ and in situ training , 2013, Nature Communications.

[2]  Chris Yakopcic,et al.  Generalized Memristive Device SPICE Model and its Application in Circuit Design , 2013, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[3]  Kaushik Roy,et al.  Low-power approximate convolution computing unit with domain-wall motion based “Spin-Memristor” for image processing applications , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[4]  Majid Ahmadi,et al.  Brain-inspired pattern classification with memristive neural network using the Hodgkin-Huxley neuron , 2016, 2016 IEEE International Conference on Electronics, Circuits and Systems (ICECS).

[5]  W. Lu,et al.  High-density Crossbar Arrays Based on a Si Memristive System , 2008 .

[6]  Yiran Chen,et al.  Memristor Crossbar-Based Neuromorphic Computing System: A Case Study , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Ben de Lacy Costello,et al.  Evolving Spiking Networks with Variable Resistive Memories , 2014, Evolutionary Computation.

[8]  Johannes Schemmel,et al.  Wafer-scale integration of analog neural networks , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[9]  Ioannis Vourkas,et al.  Massively Parallel Analog Computing: Ariadne’s Thread Was Made of Memristors , 2018, IEEE Transactions on Emerging Topics in Computing.

[10]  Andrew Adamatzky,et al.  UAV Horizon Tracking Using Memristors and Cellular Automata Visual Processing , 2013, TAROS.

[11]  Tao Zhang,et al.  PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[12]  Georgios Ch. Sirakoulis,et al.  A Digital Memristor Emulator for FPGA-Based Artificial Neural Networks , 2016, 2016 1st IEEE International Verification and Security Workshop (IVSW).

[13]  Shukai Duan,et al.  A Memristive Multilayer Cellular Neural Network With Applications to Image Processing , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Janusz A. Starzyk,et al.  Memristor Crossbar Architecture for Synchronous Neural Networks , 2014, IEEE Transactions on Circuits and Systems I: Regular Papers.

[15]  Yong Zhang,et al.  A digital neuromorphic VLSI architecture with memristor crossbar synaptic array for machine learning , 2012, 2012 IEEE International SOC Conference.

[16]  Chris Yakopcic,et al.  Memristor crossbar deep network implementation based on a Convolutional neural network , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[17]  Chris Yakopcic,et al.  Hybrid crossbar architecture for a memristor based cache , 2013, Microelectron. J..

[18]  Shimeng Yu,et al.  Investigating the switching dynamics and multilevel capability of bipolar metal oxide resistive switching memory , 2011 .

[19]  Andrew Adamatzky,et al.  Filamentary extension of the mem-con theory of memristance and its application to titanium dioxide sol-gel memristors , 2012, 2012 IEEE International Conference on Electronics Design, Systems and Applications (ICEDSA).

[20]  Chris Yakopcic,et al.  Extremely parallel memristor crossbar architecture for convolutional neural network implementation , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[21]  Guoqiang Xie,et al.  Expression of emotion using a system combined artificial neural network and memristor-based crossbar array , 2016, 2016 35th Chinese Control Conference (CCC).

[22]  Hao Jiang,et al.  A memristor-based neuromorphic engine with a current sensing scheme for artificial neural network applications , 2017, 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC).

[23]  Zheng Li,et al.  Continuous real-world inputs can open up alternative accelerator designs , 2013, ISCA.

[24]  Jacques-Olivier Klein,et al.  Robust neural logic block (NLB) based on memristor crossbar array , 2011, 2011 IEEE/ACM International Symposium on Nanoscale Architectures.

[25]  Wei D. Lu,et al.  Pattern recognition with memristor networks , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[26]  Amir F. Atiya,et al.  A new algorithm for learning in piecewise-linear neural networks , 2000, Neural Networks.

[27]  S.N. Danilin,et al.  Determining Operation Tolerances of Memristor-Based Artificial Neural Networks , 2016, 2016 International Conference on Engineering and Telecommunication (EnT).

[28]  J. Yang,et al.  Feedback write scheme for memristive switching devices , 2011 .

[29]  Siddharth Gaba,et al.  Synaptic behaviors and modeling of a metal oxide memristive device , 2011 .

[30]  P. K. Dubey,et al.  Recognition, Mining and Synthesis Moves Comp uters to the Era of Tera , 2005 .

[31]  Damien Querlioz,et al.  Simulation of a memristor-based spiking neural network immune to device variations , 2011, The 2011 International Joint Conference on Neural Networks.

[32]  Daniel Soudry,et al.  A fully analog memristor-based neural network with online gradient training , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[33]  Yu Wang,et al.  PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[34]  Andrew Adamatzky,et al.  Towards evolving spiking networks with memristive synapses , 2011, 2011 IEEE Symposium on Artificial Life (ALIFE).

[35]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[36]  L. Chua.  Memristor-The missing circuit element , 1971 .

[37]  G. Subramanyam,et al.  A Memristor Device Model , 2011, IEEE Electron Device Letters.

[38]  Georgios Ch. Sirakoulis,et al.  On the analog computational characteristics of memristive networks , 2013, 2013 IEEE 20th International Conference on Electronics, Circuits, and Systems (ICECS).

[39]  Andrew Adamatzky,et al.  Evolution of Plastic Learning in Spiking Networks via Memristive Connections , 2012, IEEE Transactions on Evolutionary Computation.

[40]  Chris Yakopcic,et al.  Memristor crossbar based multicore neuromorphic processors , 2014, 2014 27th IEEE International System-on-Chip Conference (SOCC).

[41]  Ligang Gao,et al.  High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm , 2011, Nanotechnology.

[42]  Miao Hu,et al.  ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[43]  Wei Lu,et al.  Two-terminal resistive switches (memristors) for memory and logic applications , 2011, 16th Asia and South Pacific Design Automation Conference (ASP-DAC 2011).

[44]  Yu Wang,et al.  Training itself: Mixed-signal training acceleration for memristor-based neural network , 2014, 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC).

[45]  D. Stewart,et al.  The missing memristor found , 2008, Nature.

[46]  Ricardo Cavicchioli,et al.  Advection shapes Southern Ocean microbial assemblages independent of distance and environment effects , 2013, Nature Communications.

[47]  Chris Yakopcic,et al.  Memristor SPICE model and crossbar simulation based on devices with nanosecond switching time , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[48]  Chris Yakopcic,et al.  Determining optimal switching speed for memristors in neuromorphic system , 2015 .

[49]  Jacques-Olivier Klein,et al.  Robust learning approach for neuro-inspired nanoscale crossbar architecture , 2014, ACM J. Emerg. Technol. Comput. Syst..

[50]  Mikko H. Lipasti,et al.  BenchNN: On the broad potential application scope of hardware neural network accelerators , 2012, 2012 IEEE International Symposium on Workload Characterization (IISWC).

[51]  Konstantin K. Likharev,et al.  Defect‐tolerant nanoelectronic pattern classifiers , 2007, Int. J. Circuit Theory Appl..

[52]  Muhammad Khalid,et al.  Memristor Crossbar-Based Pattern Recognition Circuit Using Perceptron Learning Rule , 2016, 2016 IEEE International Symposium on Nanoelectronic and Information Systems (iNIS).

[53]  Moongu Jeon,et al.  Hybrid neuromorphic system for automatic speech recognition , 2016 .

[54]  Mrinmoy Sarkar,et al.  Associative memory algorithm for visual pattern recognition with memristor array and CMOS neuron , 2016, 2016 9th International Conference on Electrical and Computer Engineering (ICECE).

[55]  Bernabé Linares-Barranco,et al.  On Spike-Timing-Dependent-Plasticity, Memristive Devices, and Building a Self-Learning Visual Cortex , 2011, Front. Neurosci..

[56]  Yu Wang,et al.  Leveraging Stochastic Memristor Devices in Neuromorphic Hardware Systems , 2016, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[57]  Chris Yakopcic,et al.  Energy efficient perceptron pattern recognition using segmented memristor crossbar arrays , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[58]  Chris Yakopcic,et al.  Exploring the design space of specialized multicore neural processors , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[59]  Chris Yakopcic,et al.  SPICE analysis of dense memristor crossbars for low power neuromorphic processor designs , 2015, 2015 National Aerospace and Electronics Conference (NAECON).

[60]  Witold Pedrycz,et al.  Contrastive divergence for memristor-based restricted Boltzmann machine , 2015, Engineering applications of artificial intelligence.

[61]  Chris Yakopcic,et al.  Memristor based neuromorphic circuit for ex-situ training of multi-layer neural network algorithms , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[62]  Yiran Chen,et al.  Memristor crossbar based hardware realization of BSB recall function , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[63]  Chris Yakopcic,et al.  Impact of memristor switching noise in a neuromorphic crossbar , 2015, 2015 National Aerospace and Electronics Conference (NAECON).

[64]  Andrew S. Cassidy,et al.  Building block of a programmable neuromorphic substrate: A digital neurosynaptic core , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[65]  Chris Yakopcic,et al.  Memristor-based neuron circuit and method for applying learning algorithm in SPICE? , 2014 .

[66]  Avinoam Kolodny,et al.  Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[67]  Tarek M. Taha,et al.  Enabling back propagation training of memristor crossbar neuromorphic processors , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[68]  David Moore,et al.  Silver chalcogenide based memristor devices , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[69]  Massimiliano Di Ventra,et al.  Memcomputing Implementation of Ant Colony Optimization , 2016, Neural Processing Letters.

[70]  Chao Du,et al.  Feature Extraction Using Memristor Networks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[71]  Farnood Merrikh-Bayat,et al.  Efficient training algorithms for neural networks based on memristive crossbar circuits , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[72]  Dharmendra S. Modha,et al.  A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).