Memristor-Based Neuromorphic Hardware Improvement for Privacy-Preserving ANN

Because of collecting a large amount of personal data, when the artificial neural network (ANN) is used in human-related topics, it has raised great concerns on privacy preservation. A robust solution is to introduce a noise injection mechanism as differential privacy that promises strong theoretical privacy guarantees. However, privacy-preserving ANN with noisy input data has a substantial risk of reducing the recognition accuracy. Therefore, it is urgently needed to have technologies that can make users’ data applied to neural networks while strictly protecting sensitive information. In this paper, a linear optimization (LO) method is proposed to address this accuracy degradation by optimizing the performance of memristor in weight updating processes. Instead of complying with the traditional hardware and algorithm, the LO method calculates update parameters along a piecewise line by using different input pulses. The proposed method can mitigate the nonlinear problem of memristor without prereading the precise current conductance each time, thereby avoiding complex peripheral circuits. The effectiveness of the proposed LO method with two-segment, three-segment, and four-segment models is investigated, respectively. The results show that under different nonlinearity and different perturbation noise required by differential privacy theory, the LO method can increase the recognition accuracy of Modified National Institute of Standards and Technology (MNIST) handwriting digits by 39.67% on average, which provides more space and margin for privacy-preserving technology.

[1]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[2]  Boris Murmann,et al.  Approximate SRAM for Energy-Efficient, Privacy-Preserving Convolutional Neural Networks , 2017, 2017 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[3]  Siddharth Gaba,et al.  Nanoscale resistive memory with intrinsic diode characteristics and long endurance , 2010 .

[4]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[5]  D. Ielmini,et al.  Modeling the Universal Set/Reset Characteristics of Bipolar RRAM by Field- and Temperature-Driven Filament Growth , 2011, IEEE Transactions on Electron Devices.

[6]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.

[7]  J. Kim,et al.  Neuromorphic speech systems using advanced ReRAM-based synapse , 2013, 2013 IEEE International Electron Devices Meeting.

[8]  Sungho Kim,et al.  Impact of Synaptic Device Variations on Pattern Recognition Accuracy in a Hardware Neural Network , 2018, Scientific Reports.

[9]  Amos Beimel,et al.  Bounds on the sample complexity for private learning and private data release , 2010, Machine Learning.

[10]  Shimeng Yu,et al.  Mitigating effects of non-ideal synaptic device characteristics for on-chip learning , 2015, 2015 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[11]  O. Richard,et al.  10×10nm2 Hf/HfOx crossbar resistive RAM with excellent performance, reliability and low-energy operation , 2011, 2011 International Electron Devices Meeting.

[12]  Cynthia Dwork,et al.  Differential Privacy , 2006, ICALP.

[13]  Bing Chen,et al.  Very Low-Programming-Current RRAM With Self-Rectifying Characteristics , 2016, IEEE Electron Device Letters.

[14]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[15]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[16]  Úlfar Erlingsson,et al.  Scalable Private Learning with PATE , 2018, ICLR.

[17]  Mohammad Al-Rubaie,et al.  Privacy-Preserving Machine Learning: Threats and Solutions , 2018, IEEE Security & Privacy.

[18]  Jiaming Zhang,et al.  Analogue signal and image processing with large memristor crossbars , 2017, Nature Electronics.

[19]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[20]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[21]  Byung Joon Choi,et al.  High‐Speed and Low‐Energy Nitride Memristors , 2016 .

[22]  Shimeng Yu,et al.  Fully parallel write/read in resistive synaptic array for accelerating on-chip learning , 2015, Nanotechnology.

[23]  Philip S. Yu,et al.  Not Just Privacy: Improving Performance of Private Deep Learning in Mobile Cloud , 2018, KDD.

[24]  Xiaochen Peng,et al.  NeuroSim+: An integrated device-to-algorithm framework for benchmarking synaptic devices and array architectures , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[25]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[26]  L. Chua.  Memristor-The missing circuit element , 1971 .

[27]  R. Williams,et al.  Sub-nanosecond switching of a tantalum oxide memristor , 2011, Nanotechnology.

[28]  Wei Lu,et al.  The future of electronics based on memristive systems , 2018, Nature Electronics.

[29]  I-Ting Wang,et al.  3D Ta/TaOx/TiO2/Ti synaptic array and linearity tuning of weight update for hardware neural network applications , 2016, Nanotechnology.

[30]  H. Hwang,et al.  Improved Synaptic Behavior Under Identical Pulses Using AlOx/HfO2 Bilayer RRAM Array for Neuromorphic Systems , 2016, IEEE Electron Device Letters.