The Prospects for Analogue Neural VLSI

In recent years, the efforts of analogue, neural-hardware designers have shifted from generic analogue neurocomputers to "niche" markets in sensor fusion and robotics, and we explain why this is so. We describe the main differences between digital and analogue computation, and consider the advantages of pure analogue and pulsed methods of design. We then investigate some important issues in analogue design of neural machines, namely weight storage (volatile and non-volatile), on-chip learning, and arithmetic accuracy and its relationship to noise. Finally, we outline those areas in which analogue techniques are likely to prove most useful, and speculate as to their likely long-term utility.

[1]  Lex A. Akers,et al.  A neural processing node with on-chip learning , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[2]  Chung-Yu Wu,et al.  CMOS current-mode neural associative memory design with on-chip learning , 1996, IEEE Trans. Neural Networks.

[3]  Mohamed I. Elmasry,et al.  Mixed analog/digital hardware synthesis of artificial neural networks , 1994, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[4]  Bing J. Sheu,et al.  Programmable-weight building blocks for analog VLSI neural network processors , 1996 .

[5]  E. Hirschman,et al.  The Moving Target , 1982 .

[6]  Jack L. Meador,et al.  Programmable impulse neural circuits , 1991, IEEE Trans. Neural Networks.

[7]  William Eccleston,et al.  Neural network implementation using a single MOST per synapse , 1995, IEEE Trans. Neural Networks.

[8]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[9]  Kimmo Kaski,et al.  Feasibility of synchronous pulse-density modulation arithmetic in integrated circuit implementations of artificial neural networks , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[10]  Ugur Cilingiroglu.  A charge-based neural Hamming classifier , 1993 .

[11]  Kazuo Kyuma,et al.  A 1.2 GFLOPS neural network chip for high-speed neural network servers , 1996, IEEE J. Solid State Circuits.

[12]  Takashi Morie,et al.  Deterministic Boltzmann Machine Learning Improved for Analog LSI Implementation , 1993 .

[13]  Lionel Tarassenko,et al.  Perturbation Techniques for on-Chip Learning with Analogue VLSI MLPs , 1996, J. Circuits Syst. Comput..

[14]  Ulrich Nehmzow,et al.  Experiments in Competence Acquisition for Autonomous Mobile Robots , 1992 .

[15]  N. Lipley,et al.  Moving targets , 1996, Nature.

[16]  Alan F. Murray,et al.  A concise application demonstrator for pulsed neural VLSI , 1997 .

[17]  Eytan Domany,et al.  Learning by Choice of Internal Representations , 1988, Complex Syst..

[18]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[19]  Mark R. DeYong,et al.  Applications of hybrid analog-digital neural networks in signal processing: simple circuits for frequency and phase detection and shifting , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[20]  Alan F. Murray.  Analogue noise-enhanced learning in neural network circuits , 1991 .

[21]  Ronald S. Gyurcsik,et al.  Toward a general-purpose analog VLSI neural network with on-chip learning , 1997, IEEE Trans. Neural Networks.

[22]  Tetsuro Itakura,et al.  Neuro chips with on-chip back-propagation and/or Hebbian learning , 1992 .

[23]  Carver A. Mead,et al.  A novel associative memory implemented using collective computation , 1990 .

[24]  Tor Sverre Lande,et al.  An analog feed-forward neural network with on-chip learning , 1996 .

[25]  Richard Rohwer,et al.  The "Moving Targets" Training Algorithm , 1989, NIPS.

[26]  John Taylor,et al.  A scalable high-speed current-mode winner-take-all network for VLSI neural applications , 1995 .

[27]  Marcello Chiaberge,et al.  A pulse stream system for low-power neuro-fuzzy computation , 1995 .

[28]  A. F. Murray,et al.  Pulse-stream techniques and circuits , 1996 .

[29]  Rahul Sarpeshkar,et al.  Pulse-Based Analog VLSI Velocity Sensors , 1997 .

[30]  Evangelia Micheli-Tzanakou,et al.  A neuromime in VLSI , 1996, IEEE Trans. Neural Networks.

[31]  Bing J. Sheu,et al.  A nonvolatile analog neural memory using floating-gate MOS transistors , 1992 .

[32]  Simon M. Tam,et al.  Implementation and performance of an analog nonvolatile neural network , 1993 .

[33]  John J. Paulos,et al.  A neural network learning algorithm tailored for VLSI implementation , 1994, IEEE Trans. Neural Networks.

[34]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[35]  Ronald S. Gyurcsik,et al.  Building blocks for a temperature-compensated analog VLSI neural network with on-chip learning , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[36]  Joshua Alspector,et al.  Experimental Evaluation of Learning in a Neural Microsystem , 1991, NIPS.

[37]  A. Andreou,et al.  MOS circuit for nonlinear Hebbian learning , 1992 .

[38]  Ezz I. El-Masry,et al.  Implementations of artificial neural networks using current-mode pulse width modulation technique , 1997, IEEE Trans. Neural Networks.

[39]  L. M. Reyneri.  A performance analysis of pulse stream neural and fuzzy computing systems , 1995 .

[40]  A. E. Owen,et al.  Amorphous silicon analogue memory devices , 1989 .

[41]  Anders Krogh,et al.  A Cost Function for Internal Representations , 1989, NIPS.

[42]  Tadashi Shibata,et al.  A neuron-MOS neural network using self-learning-compatible synapse circuits , 1995, IEEE J. Solid State Circuits.

[43]  J. J. Paulos,et al.  On-chip learning in the analog domain with limited precision circuits , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[44]  Alan F. Murray,et al.  Fully-Programmable Analogue VLSI Devices for the Implementation of Neural Networks , 1989 .

[45]  Marwan A. Jabri,et al.  A hybrid analog and digital VLSI neural network for intracardiac morphology classification , 1995 .

[46]  Torsten Lehmann.  ECCOPUNCH: the Edinburgh classical conditioning pulsed neural chip , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[47]  Robert B. Allen,et al.  Performance of a Stochastic Learning Microchip , 1990, NIPS.

[48]  Marwan A. Jabri,et al.  Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks , 1991, Neural Computation.

[49]  Michael A. Shanblatt,et al.  Random noise effects in pulse-mode digital multilayer neural networks , 1995, IEEE Trans. Neural Networks.

[50]  Carver A. Mead,et al.  VLSI implementation of neural networks , 1990 .

[51]  José E. Franca,et al.  Digitally programmable analog building blocks for the implementation of artificial neural networks , 1996, IEEE Trans. Neural Networks.

[52]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[53]  Torsten Lehmann,et al.  On-chip learning in pulsed silicon neural networks , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[54]  L. Carley,et al.  Trimming analog circuits using floating-gate analog MOS memory , 1989, IEEE International Solid-State Circuits Conference, 1989 ISSCC. Digest of Technical Papers.

[55]  Alan F. Murray,et al.  A Novel Computational and Signalling Method for VLSI Neural Networks , 1987, ESSCIRC '87: 13th European Solid-State Circuits Conference.

[56]  Torsten Lehmann,et al.  Nonlinear backpropagation: doing backpropagation without derivatives of the activation function , 1997, IEEE Trans. Neural Networks.

[57]  David P. M. Northmore,et al.  Switched-capacitor neuromorphs with wide-range variable dynamics , 1995, IEEE Trans. Neural Networks.

[58]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[59]  Y. Hirai,et al.  A digital neuro-chip with unlimited connectability for large scale neural networks , 1989, International 1989 Joint Conference on Neural Networks.

[60]  D. L. Bisset,et al.  An implementation of fully analogue sum-of-product neural models in VLSI , 1989 .

[61]  José Luis Huertas,et al.  A CMOS analog adaptive BAM with on-chip learning and weight refreshing , 1993, IEEE Trans. Neural Networks.

[62]  Takashi Morie,et al.  An all-analog expandable neural network LSI with on-chip backpropagation learning , 1994, IEEE J. Solid State Circuits.

[63]  Alberto Prieto,et al.  VLSI Implementation of a Neural Model Using Spikes , 1997 .

[64]  Mona E. Zaghloul,et al.  CMOS design of pulse coded adaptive neural processing element using neural-type cells , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[65]  S.K. Lai,et al.  Comparison and trends in today's dominant E2technologies , 1986, 1986 International Electron Devices Meeting.

[66]  R. S. Gyurcsik,et al.  An analog VLSI neural network with on-chip perturbation learning , 1997, IEEE J. Solid State Circuits.

[67]  Daniele D. Caviglia,et al.  An experimental analog VLSI neural network with on-chip back-propagation learning , 1992, ESSCIRC '92: Eighteenth European Solid-State Circuits conference.

[68]  Wayne Carl Westerman,et al.  Neuromorphic Synapses for Artificial Dendrites , 1997 .

[69]  H. C. Card,et al.  CMOS mean field learning , 1991 .

[70]  Alan F. Murray.  Analog VLSI and multi-layer perceptrons - Accuracy, noise and on-chip learning , 1992, Neurocomputing.

[71]  S. Tam,et al.  An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses , 1990, International 1989 Joint Conference on Neural Networks.

[72]  Alan F. Murray.  Silicon implementations of neural networks , 1989 .

[73]  Marwan A. Jabri,et al.  Algorithmic and implementation issues in analog low power learning neural network chips , 1993, J. VLSI Signal Process..

[74]  Philip Heng Wai Leong,et al.  A low-power VLSI arrhythmia classifier , 1995, IEEE Trans. Neural Networks.

[75]  J. G. Elias,et al.  A neuromorphic impulsive circuit for processing dynamic signals , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[76]  Alan F. Murray,et al.  Generic Analog Neural Computation - The Epsilon Chip , 1992, NIPS.

[77]  Hannu Tenhunen,et al.  Fully digital neural network implementation based on pulse density modulation , 1989, 1989 Proceedings of the IEEE Custom Integrated Circuits Conference.

[78]  Gert Cauwenberghs,et al.  Analog VLSI Stochastic Perturbative Learning Architectures , 1997 .

[79]  Alan F. Murray.  Multilayer Perceptron Learning Optimized for On-Chip Implementation: A Noise-Robust System , 1992, Neural Computation.

[80]  Alan Murray,et al.  Standard CMOS Floating Gate Memories for Non-Volatile Weight Storage in Analogue VLSI Neural Networks , 1997 .

[81]  Alan F. Murray,et al.  Pulse-stream circuits for on-chip learning in analogue VLSI neural networks , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[82]  D A Durfee,et al.  Comparison of floating gate neural network memory cells in standard VLSI CMOS technology , 1992, IEEE Trans. Neural Networks.

[83]  Marwan A. Jabri,et al.  Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks , 1992, IEEE Trans. Neural Networks.

[84]  H. Shinohara,et al.  A refreshable analog VLSI neural network chip with 400 neurons and 40 k synapses , 1992, 1992 IEEE International Solid-State Circuits Conference Digest of Technical Papers.

[85]  Lawrence D. Jackel,et al.  VLSI implementation of a neural network memory with several hundreds of neurons , 1987 .

[86]  John Lazzaro.  Low-power silicon spiking neurons and axons , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[87]  Gert Cauwenberghs,et al.  An analog VLSI recurrent neural network learning a continuous-time trajectory , 1996, IEEE Trans. Neural Networks.

[88]  H. C. Card,et al.  Analog CMOS deterministic Boltzmann circuits , 1993 .

[89]  P. Denyer,et al.  Most transconductance multipliers for array applications , 1981 .

[90]  Philip H. W. Leong,et al.  ANN Based Classification for Heart Defibrillators , 1991, NIPS 1991.

[91]  J. L. Lamb,et al.  Binary synaptic connections based on memory switching in a-Si:H , 1987 .

[92]  N. El-Leithy,et al.  Implementation of pulse-coded neural networks , 1988, Proceedings of the 27th IEEE Conference on Decision and Control.

[93]  Torsten Lehmann,et al.  Biologically-Inspired On-Chip Learning in Pulsed Neural Networks , 1999 .

[94]  Jan Van der Spiegel,et al.  Design and performance of a prototype analog neural computer , 1992, Neurocomputing.