An investigation of practical issues in translating algorithms based on back-propagation into analogue VLSI circuits

One of the most widely-used artificial neural networks is the multi-layer perceptron, trained by error-back-propagation (the 'back-propagation algorithm'). Commonly, the network is implemented as a serial-computer simulation, but there has been considerable interest in translating it into hardware. The most difficult translation into analogue VLSI is the 'learning' part of the algorithm, that is the part which involves calculating the output errors and making appropriate modifications to the analogue weights representing the connections between nodes. For this reason, most analogue hardware implementations train weights held off-chip in a digital representation; the weights are converted to an analogue representation for storage on the chip which comprises the network. This thesis examines the Virtual Targets algorithm, based on back-propagation, but with some modifications which render it more amenable to translation into analogue VLSI circuits which can 'learn on-chip'. I describe several circuits, designed to exploit our research group's pulse-stream approach to analogue VLSI, which provide four-quadrant multiplication, and calculate differences, signs and error-derivatives. Results, from simulation and from a chip fabricated with the circuits, are given. A consideration of other approaches to the problem of learning on-chip makes it clear that key issues are weight-storage, and a means of modifying the weights. I explain why calculating exact weight-changes is difficult, and give the results of simulation experiments leading to a further simplification of the Virtual Targets algorithm which makes it possible to train the network using fixed increments and decrements of the weights. I show the results of tests of circuits on a second chip, designed with implementation of the entire algorithm in mind, and assess the likelihood of such an implementation being successful. I place this analysis in the context of the search for 'intelligent' machines, and ask how far designs such as my own might contribute to such a machine. I also make some suggestions on the most fruitful directions for analogue designs of artificial neural networks.

[1]  Movement of the Heart and Blood in Animals: An Anatomical Essay , 1957 .

[2]  J. Brugler,et al.  Charge pumping in MOS devices , 1969 .

[3]  Yuichiro Anzai,et al.  Pattern Recognition and Machine Learning , 1992, Springer US.

[4]  Jerome A. Feldman,et al.  Connectionist Models and Their Applications: Introduction , 1985 .

[5]  Terry Winograd,et al.  Understanding computers and cognition - a new foundation for design , 1987 .

[6]  P. Groves,et al.  Introduction to biological psychology , 1988 .

[7]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[8]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[9]  Robert B. Allen,et al.  Performance of a Stochastic Learning Microchip , 1990, NIPS.

[10]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[11]  William J. B. Oldham,et al.  Hardware Implementation Of An Artificial Neural Network , 1988, Photonics West - Lasers and Applications in Science and Engineering.

[12]  A. Klopf.  A neuronal model of classical conditioning , 1988 .

[13]  Richard E. Howard,et al.  Adaptive Neural Networks Using MOS Charge Storage , 1988, NIPS.

[14]  Graeme Mitchison,et al.  Learning algorithms and networks of neurons , 1989 .

[15]  Anders Krogh,et al.  A Cost Function for Internal Representations , 1989, NIPS.

[16]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[17]  M. Duranton,et al.  Learning on VLSI: a general purpose digital neurochip , 1989, International 1989 Joint Conference on Neural Networks.

[18]  S. Tam,et al.  An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses , 1990, International 1989 Joint Conference on Neural Networks.

[19]  J. Alspector,et al.  Neural-style microsystems that learn , 1989, IEEE Communications Magazine.

[20]  P. Churchland.  A Neurocomputational Perspective: The Nature of Mind and the Structure of Science , 1989 .

[21]  W. Hubbard,et al.  A programmable analog neural network chip , 1989 .

[22]  John Lazzaro,et al.  A Silicon Model Of Auditory Localization , 1989, Neural Computation.

[23]  Marc Duranton,et al.  The LNeuro-Chip: A Digital VLSI with on-Chip Learning Mechanism , 1990 .

[24]  F. J. Kub,et al.  Programmable analog vector-matrix multipliers , 1990 .

[25]  D. B. Schwartz,et al.  Learning, function approximation and analog VLSI , 1990, IEEE International Symposium on Circuits and Systems.

[26]  S. Kayano,et al.  A self-learning neural network chip with 125 neurons and 10 K self-organization synapses , 1990 .

[27]  R. Beer,et al.  Intelligence as Adaptive Behavior: An Experiment in Computational Neuroethology , 1990 .

[28]  Vijay K. Samalam,et al.  An Analog VLSI Splining Network , 1990, NIPS.

[29]  Carver A. Mead,et al.  VLSI implementation of neural networks , 1990 .

[30]  Mona E. Zaghloul,et al.  Design of modifiable-weight synapse CMOS analog cell , 1990, IEEE International Symposium on Circuits and Systems.

[31]  Thomas Jackson,et al.  Neural Computing - An Introduction , 1990 .

[32]  David E. Rumelhart,et al.  Brain style computation: learning and generalization , 1990 .

[33]  John R. Searle,et al.  Minds, brains, and programs , 1980, Behavioral and Brain Sciences.

[34]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[35]  Fathi M. A. Salam,et al.  An all-MOS analog feedforward neural circuit with learning , 1990, IEEE International Symposium on Circuits and Systems.

[36]  Harry M. Collins,et al.  Artificial experts - social knowledge and intelligent machines , 1990, Inside technology.

[37]  Rodney A. Brooks,et al.  A robot that walks: emergent behaviors , 1991 .

[38]  Toshiyuki Furuta,et al.  Neural network LSI chip with on-chip learning , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[39]  C. Schneider,et al.  Erratum: Analogue CMOS Hebbian synapses , 1991 .

[40]  Fathi M. A. Salam,et al.  A real-time experiment using a 50-neuron CMOS analog silicon chip with on-chip digital learning , 1991, IEEE Trans. Neural Networks.

[41]  C. Schneider,et al.  Analog CMOS synaptic learning circuits adapted from invertebrate biology , 1991 .

[42]  R. A. Brooks,et al.  Intelligence without Representation , 1991, Artif. Intell..

[43]  D. J. Myers,et al.  A high performance digital processor for implementing large artificial neural networks , 1991, Proceedings of the IEEE 1991 Custom Integrated Circuits Conference.

[44]  Kimmo Kaski,et al.  DIGITAL VLSI ARCHITECTURE OF BACKPROPAGATION ALGORITHM WITH ON-CHIP LEARNING , 1991 .

[45]  H. C. Card,et al.  CMOS mean field learning , 1991 .

[46]  Daniele D. Caviglia,et al.  Selfrefreshing analogue memory cell for variable synaptic weights , 1991 .

[47]  Howard C. Card,et al.  Analog VLSI Models of Mean Field Networks , 1991 .

[48]  Fathi M. A. Salam,et al.  Implementation of feedforward artificial neural nets with learning using standard CMOS VLSI technology , 1991, 1991., IEEE International Sympoisum on Circuits and Systems.

[49]  James C. Bezdek,et al.  On the relationship between neural networks, pattern recognition and intelligence , 1992, Int. J. Approx. Reason..

[50]  Seokjin Kim,et al.  A programmable analog CMOS synapse for neural networks , 1992 .

[51]  Alan F. Murray,et al.  Generic Analog Neural Computation - The Epsilon Chip , 1992, NIPS.

[52]  John Wawrzynek,et al.  Silicon Auditory Processors as Computer Peripherals , 1992, NIPS.

[53]  Tadahiro Ohmi,et al.  A self-learning neural-network LSI using neuron MOSFETs , 1992, 1992 Symposium on VLSI Technology Digest of Technical Papers.

[54]  Alan F. Murray.  Multilayer Perceptron Learning Optimized for On-Chip Implementation: A Noise-Robust System , 1992, Neural Computation.

[55]  Alan F. Murray,et al.  Integrated pulse stream neural networks: results, issues, and pointers , 1992, IEEE Trans. Neural Networks.

[56]  Jack L. Meador,et al.  Competitive learning in asynchronous-pulse-density integrated circuits , 1992 .

[57]  H. Shinohara,et al.  A refreshable analog VLSI neural network chip with 400 neurons and 40 k synapses , 1992, 1992 IEEE International Solid-State Circuits Conference Digest of Technical Papers.

[58]  Jean-Didier Legat,et al.  Analog storage of adjustable synaptic weights , 1992, Defense, Security, and Sensing.

[59]  J. J. Paulos,et al.  On-chip learning in the analog domain with limited precision circuits , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[60]  Howard C. Card,et al.  Analog CMOS contrastive Hebbian networks , 1992, Defense, Security, and Sensing.

[61]  R. M. Siegel,et al.  Bright Air, Brilliant Fire: On the Matter of the Mind , 1992 .

[62]  H. C. Card,et al.  Analog CMOS deterministic Boltzmann circuits , 1993 .

[63]  Eric I. Knudsen,et al.  A Connectionist Model of the Owl's Sound Localization System , 1993, NIPS.

[64]  H. C. Card,et al.  Neural learning in analogue hardware: effects of component variation from fabrication and from noise , 1993 .

[65]  Yiwen Wang.  A modular analog CMOS LSI for feedforward neural networks with on-chip BEP learning , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[66]  José Luis Huertas,et al.  A CMOS analog adaptive BAM with on-chip learning and weight refreshing , 1993, IEEE Trans. Neural Networks.

[67]  John G. Elias,et al.  Artificial Dendritic Trees , 1993, Neural Computation.

[68]  Lionel Tarassenko,et al.  On-Chip Learning With Analogue Vlsi Neural Networks , 1993, Int. J. Neural Syst..

[69]  David Blair Kirk.  Accurate and Precise Computation using Analog VLSI, with Applications to Computer Graphics and Neural Networks , 1993 .

[70]  Mohammed Ismail,et al.  High Frequency CMOS Transconductors , 1993 .

[71]  Torsten Lehmann.  A Hardware Efficient Cascadable Chip Set For Ann'S With On-Chip Backpropagation , 1993, Int. J. Neural Syst..

[72]  S. Zeki.  A vision of the brain , 1993 .

[73]  Lex A. Akers,et al.  A neural processing node with on-chip learning , 1993, 1993 IEEE International Symposium on Circuits and Systems.

[74]  Y. Wang.  Analog CMOS implementation of backward error propagation , 1993, IEEE International Conference on Neural Networks.

[75]  Donald J. Baxter.  Process-tolerant VLSI neural networks for applications in optimisation , 1993 .

[76]  Alan F. Murray,et al.  Pulsestream Synapses with Non-Volatile Analogue Amorphous-Silicon Memories , 1994, NIPS.

[77]  Alan F. Murray,et al.  Pulse-stream circuits for on-chip learning in analogue VLSI neural networks , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[78]  Tor Sverre Lande,et al.  An analog continuous-time micropower Hopfield net , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[79]  M. Bouton.  Conditioning, remembering, and forgetting. , 1994 .

[80]  R. S. Gyurcsik,et al.  An analog VLSI neural network architecture with on-chip learning , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[81]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[82]  Tor Sverre Lande,et al.  An analog continuous-time neural network , 1994 .

[83]  Alan F. Murray,et al.  Pulse stream VLSI neural networks , 1994, IEEE Micro.

[84]  Joydeep Ghosh,et al.  OTA-based neural network architectures with on-chip tuning of synapses , 1994 .

[85]  Dilip Sarkar,et al.  Methods to speed up error back-propagation learning algorithm , 1995, CSUR.

[86]  Torsten Lehmann,et al.  Hardware Learning in Analogue VLSI Neural Networks , 1995 .

[87]  Alan F. Murray,et al.  Competence Acquisition in an Autonomous Mobile Robot using Hardware Neural Techniques , 1995, NIPS.

[88]  Clark S. Lindsey,et al.  Implementing the new Zero Instruction Set Computer (ZISC036) from IBM for a Higgs search , 1995 .

[89]  Alister Hamilton,et al.  Pulse based signal processing: VLSI implementation of a Palmo filter , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[90]  Terrence J. Sejnowski,et al.  The Computational Brain , 1996, Artif. Intell..

[91]  A. P..  A Hierarchical Clustering Network Based on a Model of Olfactory Processing , .