Memristor crossbar deep network implementation based on a Convolutional neural network

This paper presents a simulated memristor crossbar implementation of a deep Convolutional Neural Network (CNN). In the past few years deep neural networks implemented on GPU clusters have become the state of the art in image classification. They provide excellent classification ability at the cost of a more complex data manipulation process. However once these systems are trained, we show that the analog crossbar circuits in this paper can highly parallelize the recognition phase of a CNN algorithm. One of the drawbacks of using memristors to carry out computations is that the data stored will likely have less precision when compared to typical 32-bit floating point memory. However, we show the proposed system is capable of operating with zero loss in classification accuracy if the memristors utilized are able to store at least 16 unique values (essentially acting as 4-bit devices). To the best of our knowledge, this is the first paper that presents a memristor based circuit for implementing CNN recognition. This is also the first paper that provides a circuit for precise memristor based analog convolution.

[1]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Ligang Gao,et al.  High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm , 2011, Nanotechnology.

[3]  Chris Yakopcic,et al.  Memristor based neuromorphic circuit for ex-situ training of multi-layer neural network algorithms , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[4]  Yiran Chen,et al.  Memristor crossbar based hardware realization of BSB recall function , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[5]  Dan Ciresan,et al.  Multi-Column Deep Neural Networks for offline handwritten Chinese character classification , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[6]  Chris Yakopcic,et al.  Determining optimal switching speed for memristors in neuromorphic system , 2015 .

[7]  Jacques-Olivier Klein,et al.  Robust learning approach for neuro-inspired nanoscale crossbar architecture , 2014, ACM J. Emerg. Technol. Comput. Syst..

[8]  Janusz A. Starzyk,et al.  Memristor Crossbar Architecture for Synchronous Neural Networks , 2014, IEEE Transactions on Circuits and Systems I: Regular Papers.

[9]  Wei Lu,et al.  Two-terminal resistive switches (memristors) for memory and logic applications , 2011, 16th Asia and South Pacific Design Automation Conference (ASP-DAC 2011).

[10]  Yu Wang,et al.  Training itself: Mixed-signal training acceleration for memristor-based neural network , 2014, 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC).

[11]  Zheng Li,et al.  Continuous real-world inputs can open up alternative accelerator designs , 2013, ISCA.

[12]  Marco Wiering,et al.  2011 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN) , 2011, IJCNN 2011.

[13]  Andrew S. Cassidy,et al.  Building block of a programmable neuromorphic substrate: A digital neurosynaptic core , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[14]  Chris Yakopcic,et al.  Memristor-based neuron circuit and method for applying learning algorithm in SPICE? , 2014 .

[15]  Avinoam Kolodny,et al.  Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[17]  L. Chua.  Memristor-The missing circuit element , 1971 .

[18]  Johannes Schemmel,et al.  Wafer-scale integration of analog neural networks , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[19]  J. Yang,et al.  Feedback write scheme for memristive switching devices , 2011 .

[20]  P. K. Dubey,et al.  Recognition, Mining and Synthesis Moves Comp uters to the Era of Tera , 2005 .

[21]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[22]  Jacques-Olivier Klein,et al.  Robust neural logic block (NLB) based on memristor crossbar array , 2011, 2011 IEEE/ACM International Symposium on Nanoscale Architectures.

[23]  Chris Yakopcic,et al.  Exploring the design space of specialized multicore neural processors , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[24]  Rasmus Berg Palm,et al.  Prediction as a candidate for learning deep hierarchical models of data , 2012 .

[25]  Witold Pedrycz,et al.  Contrastive divergence for memristor-based restricted Boltzmann machine , 2015, Engineering applications of artificial intelligence.

[26]  Farnood Merrikh-Bayat,et al.  Efficient training algorithms for neural networks based on memristive crossbar circuits , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[27]  Dharmendra S. Modha,et al.  A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[28]  W. Lu,et al.  High-density Crossbar Arrays Based on a Si Memristive System , 2008 .

[29]  Yiran Chen,et al.  Memristor Crossbar-Based Neuromorphic Computing System: A Case Study , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Fabien Alibart,et al.  Pattern classification by memristive crossbar circuits using ex situ and in situ training , 2013, Nature Communications.

[31]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[32]  Mikko H. Lipasti,et al.  BenchNN: On the broad potential application scope of hardware neural network accelerators , 2012, 2012 IEEE International Symposium on Workload Characterization (IISWC).

[33]  Konstantin K. Likharev,et al.  Defect‐tolerant nanoelectronic pattern classifiers , 2007, Int. J. Circuit Theory Appl..

[34]  Chris Yakopcic,et al.  Energy efficient perceptron pattern recognition using segmented memristor crossbar arrays , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[35]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[36]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[37]  Geoffrey E. Hinton,et al.  Acoustic Modeling Using Deep Belief Networks , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[38]  Tarek M. Taha,et al.  Enabling back propagation training of memristor crossbar neuromorphic processors , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[39]  David Moore,et al.  Silver chalcogenide based memristor devices , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).