Interstice: Inverter-Based Memristive Neural Networks Discretization for Function Approximation Applications

In this article, the accuracy of inverter-based memristive neural networks (NNs) for function approximation applications is improved under the presence of process variations. The improvement is achieved by using a design approach, called INTERSTICE (Inverter-based Memristive Neural Networks Dis cretization for Function Approximation Applications), which discretizes the output values by employing a classifier. More precisely, in the INTERSTICE approach, the output range is divided into <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> subranges where each subrange is considered as a class. To train the classifier, the training samples are labeled where each label shows belonging to a specific class. To evaluate the efficacy of the design technique, some function approximation applications such as BlackScholes, FFT, <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula>-means, and Sobel are considered. Compared to PHAX, a recently published inverter-based memristive NN, INTERSTICE provides lower mean squared error (MSE) values in the presence of memristor and transistor variations. More specifically, the improvements in the mean of MSE (<inline-formula> <tex-math notation="LaTeX">$\mu _{\mathrm {MSE}}$ </tex-math></inline-formula>) are in the range of 40%–80% when considering 10% variations in the memristor resistance and transistor parameters. In addition, for most of the benchmarks, INTERSTICE improves the <inline-formula> <tex-math notation="LaTeX">$\mu _{\mathrm {MSE}}$ </tex-math></inline-formula> values of the nominal case (the case where all circuit elements are ideal) compared to PHAX. As another advantage compared to the PHAX, in INTERSTICE, digital outputs can be generated based on the selected classes which eliminates the need for an analog-to-digital converter at the output port connected to the digital part of the system. Finally, achieving lower <inline-formula> <tex-math notation="LaTeX">$\mu _{\mathrm {MSE}}$ </tex-math></inline-formula> values using fewer memristors and consuming lower energy is also attainable with this design approach.

[1]  Najim Dehak,et al.  Age Estimation in Short Speech Utterances Based on LSTM Recurrent Neural Networks , 2018, IEEE Access.

[2]  David J. Mountain,et al.  Memristor Crossbar Tiles in a Flexible, General Purpose Neural Processor , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[3]  Chris Yakopcic,et al.  Efficacy of memristive crossbars for neuromorphic processors , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[4]  Chenchen Liu,et al.  Rescuing memristor-based neuromorphic design with high defects , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[5]  Qingjiang Li,et al.  Low-Consumption Neuromorphic Memristor Architecture Based on Convolutional Neural Networks , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[6]  Arash Fayyazi,et al.  OCTAN: An On-Chip Training Algorithm for Memristive Neuromorphic Circuits , 2019, IEEE Transactions on Circuits and Systems I: Regular Papers.

[7]  Wenjun Zeng,et al.  Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[8]  Vahid Rashtchi,et al.  Code Acceleration Using Memristor-Based Approximate Matrix Multiplier: Application to Convolutional Neural Networks , 2018, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[9]  Tao Zhang,et al.  PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[10]  Chi-Ying Tsui,et al.  A high-throughput and energy-efficient RRAM-based convolutional neural network using data encoding and dynamic quantization , 2018, 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC).

[11]  Jong-Ho Lee,et al.  Efficient precise weight tuning protocol considering variation of the synaptic devices and target accuracy , 2020, Neurocomputing.

[12]  Mehdi Kamal,et al.  TOSAM: An Energy-Efficient Truncation- and Rounding-Based Scalable Approximate Multiplier , 2019, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[13]  Kyeong-Sik Min,et al.  Partial-Gated Memristor Crossbar for Fast and Power-Efficient Defect-Tolerant Training , 2019, Micromachines.

[14]  Cong Xu,et al.  Impact of process variations on emerging memristor , 2010, Design Automation Conference.

[15]  Chris Yakopcic,et al.  A fast training method for memristor crossbar based multi-layer neural networks , 2017 .

[16]  Chris Yakopcic,et al.  Ex-situ training of dense memristor crossbar for neuromorphic applications , 2015, Proceedings of the 2015 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH´15).

[17]  Ninghui Sun,et al.  DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning , 2014, ASPLOS.

[18]  Shahin Nazarian,et al.  SpRRAM: A Predefined Sparsity Based Memristive Neuromorphic Circuit for Low Power Application , 2018, ArXiv.

[19]  Cory Merkel,et al.  A Low-Power Domino Logic Architecture for Memristor-Based Neuromorphic Computing , 2019, ICONS.

[20]  Kyeong-Sik Min,et al.  Non-Ideal Effects of Memristor-CMOS Hybrid Circuits for Realizing Multiple-Layer Neural Networks , 2019, 2019 IEEE International Symposium on Circuits and Systems (ISCAS).

[21]  Miao Hu,et al.  ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[22]  Mehdi Kamal,et al.  PHAX: Physical Characteristics Aware Ex-Situ Training Framework for Inverter-Based Memristive Neuromorphic Circuits , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[23]  Xuefei Ning,et al.  Fault-tolerant training with on-line fault detection for RRAM-based neural computing systems , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[24]  Engin Ipek,et al.  Making Memristive Neural Network Accelerators Reliable , 2018, 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[25]  Yiran Chen,et al.  Vortex: Variation-aware training for memristor X-bar , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[26]  Mehdi Kamal,et al.  An Ultra Low-Power Memristive Neuromorphic Circuit for Internet of Things Smart Sensors , 2018, IEEE Internet of Things Journal.

[27]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[28]  Mehdi Kamal,et al.  Robust neuromorphic computing in the presence of process variation , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[29]  Baogang Zhang,et al.  Handling stuck-at-faults in memristor crossbar arrays using matrix transformations , 2019, ASP-DAC.

[30]  Mehdi Kamal,et al.  Res-DNN: A Residue Number System-Based DNN Accelerator Unit , 2020, IEEE Transactions on Circuits and Systems I: Regular Papers.

[31]  Luis Ceze,et al.  Neural Acceleration for General-Purpose Approximate Programs , 2014, IEEE Micro.

[32]  Hamid Sarbazi-Azad,et al.  Chapter Six - Topology Specialization for Networks-on-Chip in the Dark Silicon Era , 2018, Adv. Comput..

[33]  Chris Yakopcic,et al.  Generalized Memristive Device SPICE Model and its Application in Circuit Design , 2013, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[34]  Qiangfei Xia,et al.  Review of memristor devices in neuromorphic computing: materials sciences and device challenges , 2018, Journal of Physics D: Applied Physics.

[35]  Qi Zhang,et al.  Dynamic Feature Matching for Partial Face Recognition , 2019, IEEE Transactions on Image Processing.

[36]  L. Chua Memristor-The missing circuit element , 1971 .

[37]  Jose A. Moreno-Cadenas,et al.  Memristive recurrent neural network , 2018, Neurocomputing.

[38]  Wei Lu,et al.  Two-terminal resistive switches (memristors) for memory and logic applications , 2011, 16th Asia and South Pacific Design Automation Conference (ASP-DAC 2011).

[39]  D. Stewart,et al.  The missing memristor found , 2008, Nature.