Stable and compact design of Memristive GoogLeNet Neural Network

Abstract According to the requirements of edge intelligence for circuit volume, power consumption and computing performance, a Memristive GoogLeNet Neural Network (MGNN) circuit is designed using memristor which is a new device integrating storage and computing as the basic circuit element. This circuit adopts 1 × 1 convolution and multi-scale convolution feature fusion to reduce the number of layers required by the network while ensuring the recognition accuracy of circuit. In order to reduce the size of the memristor crossbars in the circuit, we design word-line pruning and bit-line pruning methods of Memristive Convolution (MC) layers. We also use the parameter distribution of the memristive neural network to further reduce the size of memristor crossbars. The Memristive Batch Normalization (MBN) layer and Memristive Dropout (MD) are merged into front MC layers according mathematical analysis for cutting the number of network layers and decreasing the power consumption of the circuit. We also design the channel optimization and layer optimization methods of MC layers which greatly reduce the negative effect of multi-state conductance of memristors on the accuracy, improve the stability of the circuit, and reduce the circuit volume and power consumption. Experiments show that this circuit can get 89.83% accuracy on the CIFAR-10 data set, and the power consumption of a single neuron is only 1.3 μ W . When the number of memristor multi-state conductance is 2 4 = 16 , the accuracy of the MGNN circuit close to float MGNN can still be obtained.

[1]  Weiwei Xia,et al.  Memristor Crossbars with 4.5 Terabits-per-Inch-Square Density and Two Nanometer Dimension , 2018, ArXiv.

[2]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[3]  Shiping Wen,et al.  Event-based sliding-mode synchronization of delayed memristive neural networks via continuous/periodic sampling algorithm , 2020, Appl. Math. Comput..

[4]  Csaba Andras Moritz,et al.  SkyNet: Memristor-based 3D IC for artificial neural networks , 2017, 2017 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).

[5]  R. Williams,et al.  How We Found The Missing Memristor , 2008, IEEE Spectrum.

[6]  Parami Wijesinghe,et al.  An All-Memristor Deep Spiking Neural Computing System: A Step Toward Realizing the Low-Power Stochastic Brain , 2017, IEEE Transactions on Emerging Topics in Computational Intelligence.

[7]  J. Yang,et al.  Memristor crossbar arrays with 6-nm half-pitch and 2-nm critical dimension , 2018, Nature Nanotechnology.

[8]  Mauro Forti,et al.  Memristor standard cellular neural networks computing in the flux-charge domain , 2017, Neural Networks.

[9]  Felipe Gomez Castaneda,et al.  Memristive recurrent neural network , 2018 .

[10]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[11]  Shiping Wen,et al.  CKFO: Convolution Kernel First Operated Algorithm With Applications in Memristor-Based Convolutional Neural Network , 2021, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[12]  Sheng-Yang Sun,et al.  Cascaded Neural Network for Memristor based Neuromorphic Computing , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[13]  Ru Huang,et al.  A comprehensive review on emerging artificial neuromorphic devices , 2020, Applied Physics Reviews.

[14]  Manuel Le Gallo,et al.  Stochastic phase-change neurons. , 2016, Nature nanotechnology.

[15]  Xukan Ran,et al.  Deep Learning With Edge Computing: A Review , 2019, Proceedings of the IEEE.

[16]  W. Lu,et al.  High-density Crossbar Arrays Based on a Si Memristive System , 2008 .

[17]  Bin Gao,et al.  Fully hardware-implemented memristor convolutional neural network , 2020, Nature.

[18]  Wei Liu,et al.  Bottom-up precise synthesis of stable platinum dimers on graphene , 2017, Nature Communications.

[19]  Yin Yang,et al.  Memristor-Based Echo State Network With Online Least Mean Square , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[20]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[21]  Farnood Merrikh-Bayat,et al.  Efficient training algorithms for neural networks based on memristive crossbar circuits , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[22]  Shiping Wen,et al.  Global exponential synchronization of delayed memristive neural networks with reaction-diffusion terms , 2019, Neural Networks.

[23]  Alex Pappachen James,et al.  Memristive LSTM network hardware architecture for time-series predictive modeling problems , 2018, 2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS).

[24]  Zhigang Zeng,et al.  Memristive LSTM Network for Sentiment Analysis , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[25]  Shiping Wen,et al.  Event-triggered distributed control for synchronization of multiple memristive neural networks under cyber-physical attacks , 2020, Inf. Sci..

[26]  H.-S. Philip Wong,et al.  Face classification using electronic synapses , 2017, Nature Communications.

[27]  Gert Cauwenberghs,et al.  33.1 A 74 TMACS/W CMOS-RRAM Neurosynaptic Core with Dynamically Reconfigurable Dataflow and In-situ Transposable Weights for Probabilistic Graphical Models , 2020, 2020 IEEE International Solid- State Circuits Conference - (ISSCC).

[28]  Leon O. Chua,et al.  Neuromemristive Circuits for Edge Computing: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Rajeev Balasubramonian,et al.  Improving memristor memory with sneak current sharing , 2015, 2015 33rd IEEE International Conference on Computer Design (ICCD).

[30]  Kaushik Roy,et al.  Towards spike-based machine intelligence with neuromorphic computing , 2019, Nature.

[31]  Meng-Fan Chang,et al.  25.2 A Reconfigurable RRAM Physically Unclonable Function Utilizing Post-Process Randomness Source With <6×10−6 Native Bit Error Rate , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[32]  Eby G. Friedman,et al.  Memristor-Based Circuit Design for Multilayer Neural Networks , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[33]  Sheng-Yang Sun,et al.  Cascaded Architecture for Memristor Crossbar Array Based Larger-Scale Neuromorphic Computing , 2019, IEEE Access.

[34]  Mehdi Bennis,et al.  Wireless Network Intelligence at the Edge , 2018, Proceedings of the IEEE.

[35]  Xu Chen,et al.  Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing , 2019, Proceedings of the IEEE.

[36]  Tingwen Huang,et al.  An Efficient Memristor-Based Circuit Implementation of Squeeze-and-Excitation Fully Convolutional Neural Networks , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Muhammad Ali Imran,et al.  An Overview of Neuromorphic Computing for Artificial Intelligence Enabled Hardware-Based Hopfield Neural Network , 2020, IEEE Access.

[38]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[39]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Thomas K. Tiemann,et al.  Introductory Business Statistics with Interactive Spreadsheets - 1st Canadian Edition , 2015 .

[41]  Catherine Graves,et al.  Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[42]  Yu Wang,et al.  PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[43]  Jason P. Jue,et al.  All One Needs to Know about Fog Computing and Related Edge Computing Paradigms , 2019 .

[44]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.

[45]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Catherine D. Schuman,et al.  A Survey of Neuromorphic Computing and Neural Networks in Hardware , 2017, ArXiv.

[47]  L.O. Chua,et al.  Memristive devices and systems , 1976, Proceedings of the IEEE.

[48]  Zhigang Zeng,et al.  Memristive Fully Convolutional Network: An Accurate Hardware Image-Segmentor in Deep Learning , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[49]  Mohsen Guizani,et al.  Bringing Deep Learning at the Edge of Information-Centric Internet of Things , 2019, IEEE Communications Letters.

[50]  Mianxiong Dong,et al.  Secure and Efficient Vehicle-to-Grid Energy Trading in Cyber Physical Systems: Integration of Blockchain and Edge Computing , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[51]  D. Stewart,et al.  The missing memristor found , 2008, Nature.

[52]  Yiran Chen,et al.  Memristor-Based Design of Sparse Compact Convolutional Neural Network , 2020, IEEE Transactions on Network Science and Engineering.

[53]  Sungho Kim,et al.  Pattern Recognition Using Carbon Nanotube Synaptic Transistors with an Adjustable Weight Update Protocol. , 2017, ACS nano.