Hardware-friendly learning algorithms for neural networks: an overview

The hardware implementation of artificial neural networks and their learning algorithms is a fascinating area of research with far-reaching applications. However, the mapping from an ideal mathematical model to compact and reliable hardware is far from evident. This paper presents an overview of various methods that simplify the hardware implementation of neural network models. Adaptations that are proper to specific learning rules or network architectures are discussed. These range from the use of perturbation in multilayer feedforward networks and local learning algorithms to quantization effects in self-organizing feature maps. Moreover, in more general terms, the problems of inaccuracy, limited precision, and robustness are treated.

[1]  R. Battiti,et al.  TOTEM: a digital processor for neural networks and Reactive Tabu Search , 1994, Proceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems.

[2]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[3]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[4]  F. Smieja.  Neural network constructive algorithms: Trading generalization for learning efficiency? , 1993 .

[5]  Cris Koutsougeras,et al.  Complex domain backpropagation , 1992 .

[6]  Emile Fiesler,et al.  Results on the Steepness in Backpropagation Neural Networks , 1994 .

[7]  Steve W. Piche,et al.  The selection of weight accuracies for Madalines , 1995, IEEE Trans. Neural Networks.

[8]  Indu Saxena,et al.  The Effects of Optical Thresholding in Backpropagation Neural Networks , 1995 .

[9]  Jenq-Neng Hwang,et al.  Finite Precision Error Analysis of Neural Network Hardware Implementations , 1993, IEEE Trans. Computers.

[10]  Gert Cauwenberghs,et al.  A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization , 1992, NIPS.

[11]  Robert I. Damper,et al.  Determining and improving the fault tolerance of multilayer perceptrons in a pattern-recognition application , 1993, IEEE Trans. Neural Networks.

[12]  P. Garda,et al.  RA: an analog neurocomputer for the synchronous Boltzmann machine , 1994, Proceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems.

[13]  Roberto Battiti,et al.  Training neural nets with the reactive tabu search , 1995, IEEE Trans. Neural Networks.

[14]  Jim Austin.  A review of RAM based neural networks , 1994, Proceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems.

[15]  Emile Fiesler,et al.  The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks , 1996, Neural Computation.

[16]  James D. Keeler,et al.  An Optoelectronic Architecture for Multilayer Learning in a Single Photorefractive Crystal , 1990, Neural Computation.

[17]  Leon O. Chua,et al.  The CNN paradigm , 1993 .

[18]  Marwan A. Jabri,et al.  Summed Weight Neuron Perturbation: An O(N) Improvement Over Weight Perturbation , 1992, NIPS.

[19]  Philippe Réfrégier,et al.  Theoretical investigation of the robustness of multilayer perceptrons: analysis of the linear case and extension to nonlinear networks , 1995, IEEE Trans. Neural Networks.

[20]  Jerzy B. Lont.  Analog CMOS implementation of a multi-layer perceptron with nonlinear synapses , 1992, IEEE Trans. Neural Networks.

[21]  Demetri Psaltis,et al.  IV Adaptive Multilayer Optical Networks , 1993 .

[22]  Ron Meir,et al.  A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks , 1992, NIPS.

[23]  Marwan A. Jabri,et al.  Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks , 1992, IEEE Trans. Neural Networks.

[24]  John J. Paulos,et al.  A neural network learning algorithm tailored for VLSI implementation , 1994, IEEE Trans. Neural Networks.

[25]  K. Goser,et al.  A Chip for Selforganizing Feature Maps , 1994 .

[26]  Alan F. Murray,et al.  Integrated pulse stream neural networks: results, issues, and pointers , 1992, IEEE Trans. Neural Networks.

[27]  H.C. Card,et al.  Analog Cmos Neural Circuits - In Situ Learning , 1992, Int. J. Neural Syst..

[28]  Patrick Thiran,et al.  Quantization effects in digitally behaving circuit implementations of Kohonen networks , 1994, IEEE Trans. Neural Networks.

[29]  Anthony N. Michel,et al.  Robustness and perturbation analysis of a class of artificial neural networks , 1994, Neural Networks.

[30]  Paolo Ienne,et al.  How to modify Kohonen's self-organising feature maps for an efficient digital parallel implementation , 1995 .

[31]  Edward A. Rietman,et al.  Back-propagation learning and nonidealities in analog neural network hardware , 1991, IEEE Trans. Neural Networks.

[32]  C. Lee Giles,et al.  Synaptic noise in dynamically-driven recurrent neural networks: convergence and generalization , 1994 .

[33]  S. Rueping,et al.  A chip for self-organizing feature maps , 1994, Proceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems.

[34]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[35]  H. John Caulfield,et al.  Weight discretization paradigm for optical neural networks , 1990, Other Conferences.

[36]  Hon Keung Kwan,et al.  Multilayer feedforward neural networks with single powers-of-two weights , 1993, IEEE Trans. Signal Process..

[37]  Indu Saxena,et al.  Adaptive Multilayer Optical Neural Network with Optical Thresholding , 1995 .

[38]  Daniel L. Palumbo,et al.  Performance and fault-tolerance of neural networks for optimization , 1993, IEEE Trans. Neural Networks.

[39]  D Casasent,et al.  High-capacity neural networks on nonideal hardware. , 1994, Applied optics.

[40]  Antonette M. Logar,et al.  An iterative method for training multilayer networks with threshold functions , 1994, IEEE Trans. Neural Networks.

[41]  Michel Verleysen,et al.  Analog implementation of a Kohonen map with on-chip learning , 1993, IEEE Trans. Neural Networks.

[42]  Torsten Lehmann,et al.  Nonlinear backpropagation: doing backpropagation without derivatives of the activation function , 1997, IEEE Trans. Neural Networks.

[43]  Sung-Kwon Park,et al.  The geometrical learning of binary neural networks , 1995, IEEE Trans. Neural Networks.

[44]  Joshua Alspector,et al.  Experimental Evaluation of Learning in a Neural Microsystem , 1991, NIPS.