Study of Quantized Hardware Deep Neural Networks Based on Resistive Switching Devices, Conventional versus Convolutional Approaches

A comprehensive analysis of two types of artificial neural networks (ANN) is performed to assess the influence of quantization on the synaptic weights. Conventional multilayer-perceptron (MLP) and convolutional neural networks (CNN) have been considered by changing their features in the training and inference contexts, such as number of levels in the quantization process, the number of hidden layers on the network topology, the number of neurons per hidden layer, the image databases, the number of convolutional layers, etc. A reference technology based on 1T1R structures with bipolar memristors including HfO2 dielectrics was employed, accounting for different multilevel schemes and the corresponding conductance quantization algorithms. The accuracy of the image recognition processes was studied in depth. This type of studies are essential prior to hardware implementation of neural networks. The obtained results support the use of CNNs for image domains. This is linked to the role played by convolutional layers at extracting image features and reducing the data complexity. In this case, the number of synaptic weights can be reduced in comparison to MLPs.

[1]  Cristian Zambelli,et al.  Toward Reliable Multi-Level Operation in RRAM Arrays: Improving Post-Algorithm Stability and Assessing Endurance/Data Retention , 2019, IEEE Journal of the Electron Devices Society.

[2]  Juan Pablo Carbajal,et al.  Memristors for the Curious Outsiders , 2018, Technologies.

[3]  F. J. Alonso,et al.  Time series statistical analysis: A powerful tool to evaluate the variability of resistive switching memories , 2019, Journal of Applied Physics.

[4]  C. Aggarwal Neural Networks and Deep Learning: A Textbook , 2018 .

[5]  Piero Olivo,et al.  Impact of the precursor chemistry and process conditions on the cell-to-cell variability in 1T-1R based HfO2 RRAM devices , 2018, Scientific Reports.

[6]  Jordi Suñé,et al.  $${ SIM}^2{ RRAM}$$SIM2RRAM: a physical model for RRAM devices simulation , 2017 .

[7]  Sarah Webb Deep learning for biology , 2018, Nature.

[8]  P. Mazumder,et al.  Learning in Energy‐Efficient Neuromorphic Computing , 2019 .

[9]  Nagarajan Raghavan,et al.  Recommended Methods to Study Resistive Switching Devices , 2018, Advanced Electronic Materials.

[10]  Huayi Wu,et al.  City-Wide Traffic Flow Forecasting Using a Deep Convolutional Neural Network , 2020, Sensors.

[11]  Yaoqin Xie,et al.  A Technical Review of Convolutional Neural Network-Based Mammographic Breast Cancer Diagnosis , 2019, Comput. Math. Methods Medicine.

[12]  Ming Liu,et al.  Two-dimensional materials for next-generation computing technologies , 2020, Nature Nanotechnology.

[13]  Kaushik Roy,et al.  Roadmap on emerging hardware and technology for machine learning , 2020, Nanotechnology.

[14]  Benjamin Schrauwen,et al.  Memristor Models for Machine Learning , 2014, Neural Computation.

[15]  G. Gonzalez-Cordero,et al.  In-depth study of the physics behind resistive switching in TiN/Ti/HfO2/W structures , 2017 .

[16]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[17]  Francisco Jimenez-Molinos,et al.  Phase-type distributions for studying variability in resistive memories , 2019, J. Comput. Appl. Math..

[18]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[19]  Giancarlo Fortino,et al.  A survey on deep learning in medicine: Why, how and when? , 2021, Inf. Fusion.

[20]  Francisco Jimenez-Molinos,et al.  Analysis of the statistics of device-to-device and cycle-to-cycle variability in TiN/Ti/Al:HfO2/TiN RRAMs , 2019, Microelectronic Engineering.

[21]  Gyanendra K. Verma,et al.  Convolutional neural network: a review of models, methodologies and applications to object detection , 2019, Progress in Artificial Intelligence.

[22]  E. Pérez,et al.  Kinetic Monte Carlo analysis of data retention in Al:HfO2-based resistive random access memories , 2020, Semiconductor Science and Technology.

[23]  Leon O. Chua,et al.  Neuromemristive Circuits for Edge Computing: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Zhigang Zeng,et al.  Memristive Quantized Neural Networks: A Novel Approach to Accelerate Deep Learning On-Chip. , 2019, IEEE transactions on cybernetics.

[25]  Piero Olivo,et al.  Optimized programming algorithms for multilevel RRAM in hardware neural networks , 2021, 2021 IEEE International Reliability Physics Symposium (IRPS).

[26]  C. Zambelli,et al.  Multilevel HfO2-based RRAM devices for low-power neuromorphic networks , 2019, APL Materials.

[27]  Songbai Ji,et al.  Convolutional neural network for efficient estimation of regional brain strains , 2019, Scientific Reports.

[28]  Luping Shi,et al.  Memristor devices for neural networks , 2018, Journal of Physics D: Applied Physics.

[29]  Sen Song,et al.  Bridging Biological and Artificial Neural Networks with Emerging Neuromorphic Devices: Fundamentals, Progress, and Challenges , 2019, Advanced materials.

[30]  Xiaobo Sharon Hu,et al.  A Parallel Multibit Programing Scheme With High Precision for RRAM-Based Neuromorphic Systems , 2020, IEEE Transactions on Electron Devices.

[31]  Andre Esteva,et al.  A guide to deep learning in healthcare , 2019, Nature Medicine.

[32]  H. Hwang,et al.  Optimized Programming Scheme Enabling Linear Potentiation in Filamentary HfO2 RRAM Synapse for Neuromorphic Systems , 2016, IEEE Transactions on Electron Devices.

[33]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[34]  Praveen Kumar Reddy Maddikunta,et al.  A review on deep learning for future smart cities , 2020, Internet Technol. Lett..

[35]  C. Wenger,et al.  Inherent Stochastic Learning in CMOS-Integrated HfO2 Arrays for Neuromorphic Computing , 2019, IEEE Electron Device Letters.

[36]  Meng-Fan Chang,et al.  Resistive Memory‐Based In‐Memory Computing: From Device and Large‐Scale Integration System Perspectives , 2019, Adv. Intell. Syst..

[37]  Fabian Ewald Fassnacht,et al.  Convolutional Neural Networks enable efficient, accurate and fine-grained segmentation of plant species and communities from high-resolution UAV imagery , 2019, Scientific Reports.

[38]  Shukai Duan,et al.  Modeling affections with memristor-based associative memory neural networks , 2017, Neurocomputing.

[39]  Fabio L. Traversa,et al.  Accelerating Deep Learning with Memcomputing , 2018, Neural Networks.

[40]  Yanfei Wang,et al.  Automated design of a convolutional neural network with multi-scale filters for cost-efficient seismic data classification , 2020, Nature Communications.

[41]  Mireia Bargallo Gonzalez,et al.  Resistive switching in HfO2 based valence change memories, a comprehensive 3D kinetic Monte Carlo approach , 2020, Journal of Physics D: Applied Physics.

[42]  Piero Olivo,et al.  Statistical analysis of resistive switching characteristics in ReRAM test arrays , 2014, 2014 International Conference on Microelectronic Test Structures (ICMTS).

[43]  Natasha M. Astudillo,et al.  Classification with Stochastic Learning Methods and Convolutional Neural Networks , 2020, SN Computer Science.

[44]  Binhua Tang,et al.  Recent Advances of Deep Learning in Bioinformatics and Computational Biology , 2019, Front. Genet..

[45]  Abul Bashar,et al.  SURVEY ON EVOLVING DEEP LEARNING NEURAL NETWORK ARCHITECTURES , 2019, December 2019.

[46]  E. Pérez,et al.  Programming Pulse Width Assessment for Reliable and Low-Energy Endurance Performance in Al:HfO2-Based RRAM Arrays , 2020, Electronics.