Hardware neural systems for applications : a pulsed analog approach

In recent years the considerable interest in the biologically inspired computational paradigm of artificial neural networks has led to a drive to realise these structures as VLSI hardware. At Edinburgh University research has focused on pulse stream neural networks in which neural states are encoded in the time domain as a stream of digital pulses. To date research centred on developing analog CMOS circuits to implement neural functions. In this thesis these circuits are developed and higher, system level issues addressed in order to produce a neural network system suited to use in real-world applications. To discover the key requirements for use in real-world applications, examples of application based hardware systems are reviewed, as is the field of pulse stream neural networks. These requirements led to the design of a VLSI chip, EPSILON H; a pulse stream neural chip optimised for use on the boundary of the analog domain of the real-world and the digital domain of conventional computing. The EPSILON processor card (EPC) places this chip in a system level framework that oversees chip operation and provides interfaces to analog signals, a standard digital bus and other EPCs. The system level approach taken provides a versatile platform for prototyping applications while operating with minimal host supervision. To demonstrate the versatility of this approach several applications were developed that utilised this hardware. Foremost amongst these was an autonomous mobile robot that utilises the analog nature of the hardware to provide a direct interface to real-world sensors. Also presented are a series of experiments investigating back-propagation learning on a variety of MLP problems. This study reveals the limits and practicalities of training hardware neural networks, in particular the effects of limited weight dynamic range were found to be of primary importance. From this work conclusions are drawn as to the effectiveness and future development of hardware neural computation; specifically the ability to interface to the analog domain and the issues involved in interfacing to conventional computing devices are highlighted.

[1]  Ángel Rodríguez-Vázquez,et al.  Switched-capacitor neural networks for linear programming , 1988 .

[2]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[3]  Dean Pomerleau,et al.  Rapidly Adapting Artificial Neural Networks for Autonomous Navigation , 1990, NIPS.

[4]  D. Hammerstrom,et al.  A neural network systems component , 1993, IEEE International Conference on Neural Networks.

[5]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[6]  Alan F. Murray,et al.  Pulse stream VLSI neural systems: into robotics , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[7]  Robert M. Pap,et al.  Handbook of neural computing applications , 1990 .

[8]  C.W. Anderson,et al.  Learning to control an inverted pendulum using neural networks , 1989, IEEE Control Systems Magazine.

[9]  Marwan A. Jabri,et al.  Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks , 1992, IEEE Trans. Neural Networks.

[10]  Lawrence D. Jackel,et al.  VLSI implementation of a neural network memory with several hundreds of neurons , 1987 .

[11]  Marcello Chiaberge,et al.  A comparison between analog and pulse stream VLSI hardware for neural networks and fuzzy systems , 1994, Proceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems.

[12]  M. A. Holler,et al.  Neural network recognition of objects based on impact dynamics , 1992, IEEE Conference on Nuclear Science Symposium and Medical Imaging.

[13]  Rodney A. Brooks,et al.  A robot that walks; emergent behaviors from a carefully evolved network , 1989, Proceedings, 1989 International Conference on Robotics and Automation.

[14]  Shih-Chii Liu,et al.  Adaptive Retina with Center-Surround Receptive Field , 1995, NIPS.

[15]  S. Tam,et al.  An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses , 1990, International 1989 Joint Conference on Neural Networks.

[16]  B. Widrow,et al.  An adaptive 'broom balancer' with visual inputs , 1988, IEEE 1988 International Conference on Neural Networks.

[17]  Rodney A. Brooks,et al.  HERBERT: A Second Generation Mobile Robot , 1988 .

[18]  Yun Xie.  Training Algorithms for Limited Precision Feedforward Neural Networks , 1991 .

[19]  Jakob Carlström.  Minimization of Quantization Errors in Digital Implementations of Multi Layer Perceptrons , 1995 .

[20]  Edward A. Rietman,et al.  Back-propagation learning and nonidealities in analog neural network hardware , 1991, IEEE Trans. Neural Networks.

[21]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[22]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[23]  T. Skinner,et al.  Automating operator services using automatic speech recognition , 1992, [1992] Conference Record of the Twenty-Sixth Asilomar Conference on Signals, Systems & Computers.

[24]  Alan F. Murray,et al.  Asynchronous arithmetic for VLSI neural systems , 1987 .

[25]  Steven Rehfuss,et al.  Neurocomputing hardware: present and future , 1993, Artificial Intelligence Review.

[26]  O. K. Ersoy,et al.  Hardware implementation of a 'wired-once' neural net in thin-film technology on a glass substrate , 1990 .

[27]  Phillip E Allen,et al.  CMOS Analog Circuit Design , 1987 .

[28]  Alan F. Murray,et al.  Results from Pulse-Stream VLSI Neural Network Devices , 1991 .

[29]  Alan F. Murray,et al.  Competence Acquisition in an Autonomous Mobile Robot using Hardware Neural Techniques , 1995, NIPS.

[30]  Armando Manduca,et al.  Neural Network Diagnosis Of Avascular Necrosis From Magnetic Resonance Images , 1991, Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society Volume 13: 1991.

[31]  Ulrich Nehmzow,et al.  Experiments in Competence Acquisition for Autonomous Mobile Robots , 1992 .

[32]  Markus Höhfeld,et al.  Learning with limited numerical precision using the cascade-correlation algorithm , 1992, IEEE Trans. Neural Networks.

[33]  Alan F. Murray,et al.  Pulse Stream Neural Networks for Applications , 1995 .

[34]  Philip H. W. Leong,et al.  A VLSI neural network for morphology classification , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[35]  Ernst. Nordstrom.  A Hybrid Admission Control Scheme for Broadband ATM Traffic , 1993 .

[36]  Dimitris Anastassiou,et al.  Switched-capacitor neural networks , 1987 .

[37]  Lawrence D. Jackel,et al.  Electronic neural networks , 1987, AT&T Technical Journal.

[38]  Alan F. Murray,et al.  Analogue computation using VLSI neural network devices , 1990 .

[39]  Alan F. Murray,et al.  Asynchronous VLSI neural networks using pulse-stream arithmetic , 1988 .

[40]  Alan F. Murray,et al.  Generic Analog Neural Computation - The Epsilon Chip , 1992, NIPS.

[41]  Teuvo Kohonen,et al.  An introduction to neural computing , 1988, Neural Networks.

[42]  Marwan A. Jabri,et al.  A Novel Channel Selection System in Cochlear Implants Using Artificial Neural Network , 1995, NIPS.

[43]  Lars Asplund,et al.  Neural networks for admission control in an ATM network , 1994 .

[44]  Ulrich Nehmzow,et al.  Flexible Control of Mobile Robots through Autonomous Competence Acquisition , 1995 .

[45]  Stephen J. Roberts,et al.  Supervised and unsupervised learning in radial basis function classifiers , 1994 .

[46]  Eric A. Vittoz,et al.  Micropower Techniques , 1994 .

[47]  L. R. Kern,et al.  Design and development of a real-time neural processor using the Intel 80170NX ETANN , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[48]  Andrew G. Barto,et al.  Improving Elevator Performance Using Reinforcement Learning , 1995, NIPS.

[49]  Charles W. Butler,et al.  Naturally intelligent systems , 1990 .

[50]  Lawrence D. Jackel,et al.  Recent developments of electronic neural nets in North America , 1993, J. VLSI Signal Process..

[51]  Alexander Zelinsky,et al.  Learning to Coordinate Behaviours for Robot Navigation , 1997 .

[52]  S. Chakrabarti,et al.  Robust radar target classifier using artificial neural networks , 1995, IEEE Trans. Neural Networks.

[53]  Ron Meir,et al.  A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks , 1992, NIPS.

[54]  Bernard Angéniol,et al.  Self-organizing feature maps and the travelling salesman problem , 1988, Neural Networks.

[55]  Y. Matsuura,et al.  Word recognition using a neural network and a phonetically based DTW , 1994, Proceedings of IEEE Workshop on Neural Networks for Signal Processing.

[56]  Alice M. Chiang,et al.  CCD Neural Network Processors for Pattern Recognition , 1991, NIPS.

[57]  Alan F. Murray,et al.  Integrated pulse stream neural networks: results, issues, and pointers , 1992, IEEE Trans. Neural Networks.

[58]  Marwan A. Jabri,et al.  A hybrid analog and digital VLSI neural network for intracardiac morphology classification , 1995 .

[59]  John Hallam,et al.  Really Useful Robots , 1989, IAS.

[60]  Alan H. Kramer,et al.  Compact EEPROM-based Weight Functions , 1990, NIPS.

[61]  Lars Asplund,et al.  Statistical prerocessing for service quality estimation in a broadband network , 1993 .

[62]  I. Guyon,et al.  Handwritten digit recognition: applications of neural network chips and automatic learning , 1989, IEEE Communications Magazine.

[63]  Ugur Cilingiroglu,et al.  A purely capacitive synaptic matrix for fixed-weight neural networks , 1991 .

[64]  Randall L. Geiger,et al.  VLSI Design Techniques for Analog and Digital Circuits , 1989 .

[65]  L. W. Massengill,et al.  Threshold non-linearity effects on weight-decay tolerance in analog neural networks , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[66]  Jenq-Neng Hwang,et al.  Finite Precision Error Analysis of Neural Network Hardware Implementations , 1993, IEEE Trans. Computers.

[67]  Alan F. Murray,et al.  Pulse Stream Vlsi Circuits And Systems: The Epsilon Neural Network Chipset , 1993, Int. J. Neural Syst..

[68]  Dan Hammerstrom,et al.  Piriform model execution on a neurocomputer , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[69]  Donald J. Baxter.  Process-tolerant VLSI neural networks for applications in optimisation , 1993 .

[70]  Andreas G. Andreou,et al.  Current-mode subthreshold MOS circuits for analog VLSI neural systems , 1991, IEEE Trans. Neural Networks.

[71]  John J. Paulos,et al.  The Effects of Precision Constraints in a Backpropagation Learning Network , 1990, Neural Computation.

[72]  P. Denyer,et al.  Most transconductance multipliers for array applications , 1981 .

[73]  Philip H. W. Leong,et al.  ANN Based Classification for Heart Defibrillators , 1991, NIPS 1991.

[74]  Hal McCartor,et al.  Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip , 1990, NIPS 1990.