Proposal, analysis and demonstration of Analog/Digital-mixed Neural Networks based on memristive device arrays

Artificial neural network (NN) circuits are proposed and analyzed from the viewpoints of flexibility and robustness based on memristive devices. The typical 3-layered fully-connected NN is a primitive unit of Deep Learning (DL) and plays a key role in determining DL performance. We propose three types of NN structure at a circuit level — fully-digital, analog/digital mixed, fully-analog — focusing on digital or analog calculations such as a multiplier accumulator, and examine their figures of merit. One key property of the analog/digital-mixed structure is demonstrated on reconfigurable analog ICs. Important aspects regarding neuromorphic computing are highlighted in our experiments on spike-timing-dependent plasticity on memristive devices and non-linear characteristics.

[1]  Giacomo Indiveri,et al.  A VLSI array of low-power spiking neurons and bistable synapses with spike-timing dependent plasticity , 2006, IEEE Transactions on Neural Networks.

[2]  Masafumi Yamamoto,et al.  Epitaxial Growth of Fe/MgO/Fe Heterostructures on SrTiO3(001) Substrates by Magnetron Sputtering , 2005 .

[3]  Bernabe Linares-Barranco.  Memristors fire away , 2018 .

[4]  Tetsuya Asai,et al.  Live demonstration: Feature extraction system using restricted Boltzmann machines on FPGA , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[5]  Jun Deguchi,et al.  A Neuromorphic Chip Optimized for Deep Learning and CMOS Technology With Time-Domain Analog and Digital Mixed-Signal Processing , 2017, IEEE Journal of Solid-State Circuits.

[6]  Tetsuya Asai,et al.  Robustness of hardware-oriented restricted Boltzmann machines in deep belief networks for reliable processing , 2016 .

[7]  Takayuki Ishikawa,et al.  Highly spin-polarized tunneling in fully epitaxial Co2Cr0.6Fe0.4Al∕MgO∕Co50Fe50 magnetic tunnel junctions with exchange biasing , 2007 .

[8]  R. Waser,et al.  Resistive Switching: From Fundamentals of Nanoionic Redox Processes to Memristive Device Applications , 2016 .

[9]  Tobi Delbrück,et al.  CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory–Processing– Learning–Actuating System for High-Speed Visual Object Recognition and Tracking , 2009, IEEE Transactions on Neural Networks.

[10]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[11]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[12]  Yoshifumi Nishi,et al.  Bipolar Resistive Switch Effects in Calcium Fluoride Thin Films Deposited on Silicon Bottom Electrodes , 2013 .

[13]  S. Menzel,et al.  Origin of the SET Kinetics of the Resistive Switching in Tantalum Oxide Thin Films , 2014, IEEE Electron Device Letters.

[14]  Tetsuya Asai,et al.  Error Tolerance Analysis of Deep Learning Hardware Using a Restricted Boltzmann Machine Toward Low-Power Memory Implementation , 2017, IEEE Transactions on Circuits and Systems II: Express Briefs.

[15]  Tetsuya Asai,et al.  Memory-error tolerance of scalable and highly parallel architecture for restricted Boltzmann machines in Deep Belief Network , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[16]  Masafumi Yamamoto,et al.  Fabrication of Fully Epitaxial Magnetic Tunnel Junctions Using Full-Heusler Alloy Co2Cr0.6Fe0.4Al Thin Film and MgO Tunnel Barrier , 2005 .

[17]  Tetsuya Asai,et al.  FPGA Implementation of a Scalable and Highly Parallel Architecture for Restricted Boltzmann Machines , 2016 .

[18]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[19]  Rainer Waser,et al.  Effect of RESET Voltage on Distribution of SET Switching Time of Bipolar Resistive Switching in a Tantalum Oxide Thin Film , 2015, IEEE Transactions on Electron Devices.

[20]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[21]  Damien Querlioz,et al.  Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity , 2012, Neural Networks.

[22]  Bernabé Linares-Barranco,et al.  Hardware implementation of convolutional STDP for on-line visual feature learning , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[23]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[24]  Osamu Fujita,et al.  A sparse memory-access neural network engine with 96 parallel data-driven processing units , 1995, Proceedings ISSCC '95 - International Solid-State Circuits Conference.