Examining the Role and Limits of Batchnorm Optimization to Mitigate Diverse Hardware-noise in In-memory Computing

In-Memory Computing (IMC) platforms such as analog crossbars are gaining focus as they facilitate the acceleration of low-precision Deep Neural Networks (DNNs) with high area- & compute-efficiencies. However, the intrinsic non-idealities in crossbars, which are often non-deterministic and non-linear, degrade the performance of the deployed DNNs. In addition to quantization errors, most frequently encountered non-idealities during inference include crossbar circuit-level parasitic resistances and device-level non-idealities such as stochastic read noise and temporal drift. In this work, our goal is to closely examine the distortions caused by these non-idealities on the dot-product operations in analog crossbars and explore the feasibility of a nearly training-less solution via crossbar-aware fine-tuning of batchnorm parameters in real-time to mitigate the impact of the non-idealities. This enables reduction in hardware costs in terms of memory and training energy for IMC noise-aware retraining of the DNN weights on crossbars.

[1]  R. Rovatti,et al.  Combined HW/SW Drift and Variability Mitigation for PCM-Based Analog In-Memory Computing for Neural Network Applications , 2023, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[2]  M. Rasch,et al.  Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators , 2023, Nature communications.

[3]  Kanghyeon Byun,et al.  Recent Advances in Synaptic Nonvolatile Memory Devices and Compensating Architectural and Algorithmic Methods Toward Fully Integrated Neuromorphic Chips , 2022, Advanced Materials Technologies.

[4]  Priyadarshini Panda,et al.  NEAT: Nonlinearity Aware Training for Accurate, Energy-Efficient, and Robust Implementation of Neural Networks on 1T-1R Crossbars , 2022, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[5]  P. Panda,et al.  Examining the Robustness of Spiking Neural Networks on Non-ideal Memristive Crossbars , 2022, ISLPED.

[6]  A. Mallik,et al.  DIANA: An End-to-End Energy-Efficient Digital and ANAlog Hybrid Neural Network SoC , 2022, 2022 IEEE International Solid- State Circuits Conference (ISSCC).

[7]  Kaushik Roy,et al.  NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems , 2021, ArXiv.

[8]  Kaoutar El Maghraoui,et al.  A Flexible and Fast PyTorch Toolkit for Simulating Training and Inference on Analog Crossbar Arrays , 2021, 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS).

[9]  Amjad J. Humaidi,et al.  Review of deep learning: concepts, CNN architectures, challenges, applications, future directions , 2021, Journal of Big Data.

[10]  Wei Wei,et al.  Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization. , 2020 .

[11]  Mostafa Rahimi Azghadi,et al.  MemTorch: A Simulation Framework for Deep Memristive Cross-Bar Architectures , 2020, 2020 IEEE International Symposium on Circuits and Systems (ISCAS).

[12]  K. Roy,et al.  Pathways to efficient neuromorphic computing with non-volatile memory technologies , 2020 .

[13]  E. Eleftheriou,et al.  Memory devices and applications for in-memory computing , 2020, Nature Nanotechnology.

[14]  Kaushik Roy,et al.  GENIEx: A Generalized Approach to Emulating Non-Ideality in Memristive Xbars using Neural Networks , 2020, 2020 57th ACM/IEEE Design Automation Conference (DAC).

[15]  Xiaochen Peng,et al.  DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-Chip Training , 2020, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[16]  Shubham Jain,et al.  TxSim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems , 2020, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[17]  Fan Zhang,et al.  Mitigate Parasitic Resistance in Resistive Crossbar-based Convolutional Neural Networks , 2019, ACM J. Emerg. Technol. Comput. Syst..

[18]  Ali Chehab,et al.  RRAM Device Models: A Comparative Analysis With Experimental Validation , 2019, IEEE Access.

[19]  Xiaoyu Sun,et al.  Impact of Non-Ideal Characteristics of Resistive Synaptic Devices on Implementing Convolutional Neural Networks , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[20]  Evangelos Eleftheriou,et al.  Accurate deep neural network inference using computational phase-change memory , 2019, Nature Communications.

[21]  E. Eleftheriou,et al.  A phase-change memory model for neuromorphic computing , 2018, Journal of Applied Physics.

[22]  Kaushik Roy,et al.  RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[23]  Steven J. Plimpton,et al.  Multiscale Co-Design Analysis of Energy, Latency, Area, and Accuracy of a ReRAM Analog Neural Training Accelerator , 2017, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[24]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[26]  X. Liao,et al.  ReHy: A ReRAM-based Digital/Analog Hybrid PIM Architecture for Accelerating CNN Training , 2021, IEEE Transactions on Parallel and Distributed Systems.