Biologically-Inspired On-Chip Learning in Pulsed Neural Networks

Self-learning chips to implement many popular ANN (artificial neural network) algorithms are very difficult to design. We explain why this is so and say what lessons previous work teaches us in the design of self-learning systems. We offer a contribution to the “biologically-inspired” approach, explaining what we mean by this term and providing an example of a robust, self-learning design that can solve simple classical-conditioning tasks. We give details of the design of individual circuits to perform component functions, which can then be combined into a network to solve the task. We argue that useful conclusions as to the future of on-chip learning can be drawn from this work.

[1]  Takashi Morie,et al.  An all-analog expandable neural network LSI with on-chip backpropagation learning , 1994, IEEE J. Solid State Circuits.

[2]  Kurt W. Fleischer,et al.  Analog VLSI Implementation of Multi-dimensional Gradient Descent , 1992, NIPS 1992.

[3]  A. Andreou,et al.  MOS circuit for nonlinear Hebbian learning , 1992 .

[4]  Kurt W. Fleischer,et al.  Analog VLSI Implementation of Gradient Descent , 1992, NIPS.

[5]  D. Levine.  Introduction to Neural and Cognitive Modeling , 2018 .

[6]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[7]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[8]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[9]  Richard E. Howard,et al.  Adaptive Neural Networks Using MOS Charge Storage , 1988, NIPS.

[10]  Mark A. Gluck,et al.  Learning with Temporal Derivatives in Pulse-Coded Neuronal Systems , 1988, NIPS.

[11]  Alan F. Murray,et al.  Pulse-stream VLSI neural networks mixing analog and digital techniques , 1991, IEEE Trans. Neural Networks.

[12]  Leonard K. Kaczmarek,et al.  The Neuron: Cell and Molecular Biology , 1991 .

[13]  Ronald S. Gyurcsik,et al.  Building blocks for a temperature-compensated analog VLSI neural network with on-chip learning , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[14]  M. Bouton.  Conditioning, remembering, and forgetting. , 1994 .

[15]  Joshua Alspector,et al.  Experimental Evaluation of Learning in a Neural Microsystem , 1991, NIPS.

[16]  James S. Morgan,et al.  A drive-reinforcement neural network model of simple instrumental conditioning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[17]  H. Shinohara,et al.  A refreshable analog VLSI neural network chip with 400 neurons and 40 K synapses , 1992 .

[18]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[19]  Kwabena Boahen,et al.  A retinomorphic vision system , 1996, IEEE Micro.

[20]  John Wawrzynek,et al.  Systems technologies for silicon auditory models , 1994, IEEE Micro.

[21]  T. Lehmann.  Teaching pulsed integrated neural systems: a psychobiological approach , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[22]  Torsten Lehmann,et al.  On-chip learning in pulsed silicon neural networks , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[23]  Howard C. Card,et al.  Tolerance to analog hardware of on-chip learning in backpropagation networks , 1995, IEEE Trans. Neural Networks.

[24]  Mona E. Zaghloul,et al.  Silicon Implementation of Pulse Coded Neural Networks , 1994 .

[25]  Chih-Ming Ho,et al.  Analog VLSI system for active drag reduction , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[26]  J. Byrne.  Cellular analysis of associative learning. , 1987, Physiological reviews.

[27]  Torsten Lehmann,et al.  Nonlinear backpropagation: doing backpropagation without derivatives of the activation function , 1997, IEEE Trans. Neural Networks.

[28]  P. Fromherz,et al.  Silicon-Neuron Junction: Capacitive Stimulation of an Individual Neuron on a Silicon Chip. , 1995, Physical review letters.

[29]  Torsten Lehmann,et al.  Mixed analog/digital matrix-vector multiplier for neural network synapses , 1996 .

[30]  David P. M. Northmore,et al.  Switched-capacitor neuromorphs with wide-range variable dynamics , 1995, IEEE Trans. Neural Networks.

[31]  Tetsuro Itakura,et al.  Neuro chips with on-chip back-propagation and/or Hebbian learning , 1992 .

[32]  Torsten Lehmann.  ECCOPUNCH: the Edinburgh classical conditioning pulsed neural chip , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[33]  Gert Cauwenberghs,et al.  An analog VLSI recurrent neural network learning a continuous-time trajectory , 1996, IEEE Trans. Neural Networks.

[34]  Igor M. Filanovsky,et al.  CMOS Schmitt trigger design , 1994 .

[35]  Lex A. Akers,et al.  A neural processing node with on-chip learning , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[36]  Davide Badoni,et al.  Electronic implementation of an analogue attractor neural network with stochastic learning , 1995 .

[37]  Patrick A. Shoemaker,et al.  A hierarchical clustering network based on a model of olfactory processing , 1992 .

[38]  Torsten Lehmann.  Classical conditioning with pulsed integrated neural networks: circuits and system , 1998 .

[39]  J. J. Paulos,et al.  On-chip learning in the analog domain with limited precision circuits , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[40]  Lloyd W. Massengill,et al.  Weight decay and resolution effects in feedforward artificial neural networks , 1991, IEEE Trans. Neural Networks.

[41]  Carver A. Mead,et al.  VLSI implementation of neural networks , 1990 .

[42]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[43]  Robert B. Allen,et al.  Performance of a Stochastic Learning Microchip , 1990, NIPS.

[44]  Ulrich Ramacher,et al.  Recent Developments In Neurodynamics And Their Impact On The Design Of Neuro-Chips , 1993, Int. J. Neural Syst..

[45]  José Luis Huertas,et al.  A CMOS analog adaptive BAM with on-chip learning and weight refreshing , 1993, IEEE Trans. Neural Networks.

[46]  A. Klopf.  A neuronal model of classical conditioning , 1988 .

[47]  H. C. Card,et al.  Neural learning in analogue hardware: effects of component variation from fabrication and from noise , 1993 .

[48]  LehmannTorsten,et al.  Biologically-Inspired On-Chip Learning in Pulsed Neural Networks , 1999 .

[49]  Lionel Tarassenko,et al.  Perturbation Techniques for on-Chip Learning with Analogue VLSI MLPs , 1996, J. Circuits Syst. Comput..

[50]  Alan F. Murray,et al.  Pulse-stream circuits for on-chip learning in analogue VLSI neural networks , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[51]  Marwan A. Jabri,et al.  Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks , 1992, IEEE Trans. Neural Networks.

[52]  Tor Sverre Lande,et al.  An analog feed-forward neural network with on-chip learning , 1996 .