Is VLSI neural learning robust against circuit limitations?

An investigation is made of the tolerance of various in-circuit learning algorithms to component imprecision and other circuit limitations in artificial neural networks. Supervised learning mechanisms including backpropagation and contrastive Hebbian leaning, and unsupervised soft competitive learning are all shown to be tolerant of those levels of arithmetic inaccuracy, noise, nonlinearity, weight decay, and statistical variation from fabrication that the authors have experienced in 1.2 /spl mu/m analog CMOS circuits employing Gilbert multipliers as the primary computational element. These learning circuits also function properly in the presence of offset errors in analog multipliers and adders, provided that the computed weight updates are constrained by the circuitry to be made only when they exceed certain minimum or threshold values. These results are also relevant for compact (low bit rate) digital implementations.<<ETX>>

[1]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[2]  Geoffrey E. Hinton,et al.  The Bootstrap Widrow-Hoff Rule as a Cluster-Formation Algorithm , 1990, Neural Computation.

[3]  Geoffrey E. Hinton,et al.  Self-organizing neural network that discovers surfaces in random-dot stereograms , 1992, Nature.

[4]  Edward A. Rietman,et al.  Back-propagation learning and nonidealities in analog neural network hardware , 1991, IEEE Trans. Neural Networks.

[5]  S. Kayano,et al.  A self-learning neural network chip with 125 neurons and 10 K self-organization synapses , 1990 .

[6]  J. L. Holt,et al.  Back propagation simulations using limited precision calculations , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[7]  Mohammed Ismail,et al.  Analog VLSI Implementation of Neural Systems , 2011, The Kluwer International Series in Engineering and Computer Science.

[8]  H. C. Card,et al.  The effects of analog hardware properties on backpropagation networks with on-chip learning , 1993, IEEE International Conference on Neural Networks.

[9]  Jerzy B. Lont.  Analog CMOS implementation of a multi-layer perceptron with nonlinear synapses , 1992, IEEE Trans. Neural Networks.

[10]  Jenq-Neng Hwang,et al.  Finite precision error analysis of neural network electronic hardware implementations , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[11]  Howard C. Card,et al.  Analog CMOS neural networks based on Gilbert multipliers with in-circuit learning , 1993, Proceedings of 36th Midwest Symposium on Circuits and Systems.

[12]  John Lazzaro,et al.  Winner-Take-All Networks of O(N) Complexity , 1988, NIPS.

[13]  Lloyd W. Massengill,et al.  Weight decay and resolution effects in feedforward artificial neural networks , 1991, IEEE Trans. Neural Networks.

[14]  Bing J. Sheu,et al.  A high-precision VLSI winner-take-all circuit for self-organizing neural networks , 1993 .

[15]  Alan F. Murray.  Multilayer Perceptron Learning Optimized for On-Chip Implementation: A Noise-Robust System , 1992, Neural Computation.

[16]  Brian R. Gaines,et al.  Stochastic Computing Systems , 1969 .

[17]  Howard C. Card,et al.  Analog hardware tolerance of soft competitive learning , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[18]  Michel Declercq,et al.  Implementation of a learning Kohonen neuron based on a new multilevel storage technique , 1991 .

[19]  K. Wojtek Przytula.  Parallel digital implementations of neural networks , 1991, Proceedings of the International Conference on Application Specific Array Processors.

[20]  Robert B. Allen,et al.  Relaxation Networks for Large Supervised Learning Problems , 1990, NIPS.

[21]  John J. Paulos,et al.  The Effects of Precision Constraints in a Backpropagation Learning Network , 1990, Neural Computation.

[22]  W. Hubbard,et al.  A programmable analog neural network chip , 1989 .

[23]  Alan F. Murray.  Analogue noise-enhanced learning in neural network circuits , 1991 .

[24]  Thomas K. Miller,et al.  A digital architecture employing stochasticism for the simulation of Hopfield neural nets , 1989 .

[25]  H. C. Card,et al.  Analog CMOS deterministic Boltzmann circuits , 1993 .