Periodic activation functions in memristor-based analog neural networks

This work explores the use of periodic activation functions in memristor-based analog neural networks. We propose a hardware neuron based on a folding amplifier that produces a periodic output voltage. Furthermore, the amplifier's fold factor be adjusted to change the number of low-to-high or high-to-low output voltage transitions. We also propose a memristor-based synapse circuit and training circuitry for realizing the Perceptron learning rule. Behavioral models of our circuits were developed for simulating a single-layer, single-output feedforward neural network. The network was trained to detect the edges of a grayscale image. Our results show that neurons with a single fold-with an activation function similar to a sigmoidal activation function-perform the worst for this application, since they are unable to learn functions with multiple decision boundaries. Conversely, the 4-fold neuron performs the best (up to ≈65% better than the 1-fold neuron), as its activation function is periodic, and it is able to learn functions with four decision boundaries.

[1]  Kyeong-Sik Min,et al.  Self-Adaptive Write Circuit for Low-Power and Variation-Tolerant Memristors , 2010, IEEE Transactions on Nanotechnology.

[2]  R. Williams,et al.  Exponential ionic drift: fast switching and low volatility of thin-film memristors , 2009 .

[3]  Dhireesha Kudithipudi,et al.  Towards Thermal Profiling in CMOS/Memristor Hybrid RRAM Architectures , 2012, 2012 25th International Conference on VLSI Design.

[4]  Alan B. Grebene,et al.  Analog Integrated Circuit Design , 1978 .

[5]  Leon O. Chua.  Resistance switching memories are memristors , 2011 .

[6]  R. Dittmann,et al.  Redox‐Based Resistive Switching Memories – Nanoionic Mechanisms, Prospects, and Challenges , 2009, Advanced materials.

[7]  Howard C. Card,et al.  Analog CMOS neural networks based on Gilbert multipliers with in-circuit learning , 1993, Proceedings of 36th Midwest Symposium on Circuits and Systems.

[8]  Bin Li,et al.  Neural network based edge detection for automated medical diagnosis , 2011, 2011 IEEE International Conference on Information and Automation.

[9]  J. Yang,et al.  Memristive switching mechanism for metal/oxide/metal nanodevices. , 2008, Nature nanotechnology.

[10]  Kenneth W. Martin,et al.  Analog integrated circuit design. 2nd ed. , 2012 .

[11]  J.N. Babanezhad,et al.  A 20-V four-quadrant CMOS analog multiplier , 1985, IEEE Journal of Solid-State Circuits.

[12]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[13]  L. Chua.  Memristor-The missing circuit element , 1971 .

[14]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[15]  Nathan R. McDonald,et al.  Al/CuxO/Cu Memristive Devices: Fabrication, Characterization, and Modeling , 2012 .

[16]  Teuvo Kohonen,et al.  An introduction to neural computing , 1988, Neural Networks.

[17]  R. Williams,et al.  Coupled ionic and electronic transport model of thin-film semiconductor memristive behavior. , 2009, Small.

[18]  Bernard Widrow,et al.  Perceptrons, adalines, and backpropagation , 1998 .

[19]  Edgar Sanchez-Sinencio,et al.  CMOS transconductance multipliers: a tutorial , 1998 .