Digital VLSI backpropagation networks

An overview is presented of digital VLSI implementations of artificial neural networks (ANNs) configured as multilayer perceptrons employing the backpropagation learning algorithm. Several other network architectures and learning algorithms are also mentioned for comparison. We focus on those implementations which employ parallel hardware in the learning computations, not simply in the retrieval or classification process. The treatment extends from serial and parallel general-purpose simulators, which are simply programmed to implement these learning algorithms, to full custom CMOS chips or neurocomputers dedicated to one version of the learning model. Among the themes of this paper are topologies, bit-serial communications, arithmetic systems, and trade-offs between flexibility and performance.

[1]  T G Clarkson,et al.  The pRAM: an adaptive VLSI chip , 1993, IEEE Trans. Neural Networks.

[2]  Alan F. Murray.  Silicon implementations of neural networks , 1989 .

[3]  Graham A. Jullien,et al.  Number Theoretic Techniques in Digital Signal Processing , 1991 .

[4]  Katsunari Shibata,et al.  A self-learning digital neural network using wafer-scale LSI , 1993 .

[5]  Peter B. Denyer,et al.  VLSI Signal Processing: A Bit-Serial Approach , 1985 .

[6]  W. E. Blanz,et al.  GANGLION-a fast field-programmable gate array implementation of a connectionist classifier , 1992 .

[7]  Hal McCartor,et al.  Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip , 1990, NIPS 1990.

[8]  Stuart Haber,et al.  A VLSI-efficient technique for generating multiple uncorrelated noise sources and its application to stochastic neural networks , 1991 .

[9]  M. Vellasco,et al.  VLSI architectures for neural networks , 1989, IEEE Micro.

[10]  Y. Amemiya,et al.  A high-speed digital neural network chip with low-power chain-reaction architecture , 1992 .

[11]  Alan F. Murray,et al.  Pulse-stream VLSI neural networks mixing analog and digital techniques , 1991, IEEE Trans. Neural Networks.

[12]  Takeshi Sakata,et al.  A single 1.5-V digital chip for a 106 synapse neural network , 1993, IEEE Trans. Neural Networks.

[13]  Widrow,et al.  DARPA Neural Network Stdy , 1988 .

[14]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[15]  George Papadourakis,et al.  A Neural Net Associative Memory for Real-Time Applications , 1990, Neural Computation.

[16]  Howard C. Card,et al.  Parallel Random Number Generation for VLSI Systems Using Cellular Automata , 1989, IEEE Trans. Computers.

[17]  Charles L. Seitz,et al.  Concurrent architectures , 1990 .

[18]  H. T. Kung.  Why systolic architectures? , 1982, Computer.

[19]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[20]  Michel Weinfeld,et al.  A Fully Digital Integrated CMOS Hopfield Network Including the Learning Algorithm , 1989 .

[21]  K. Wojtek Przytula.  Parallel digital implementations of neural networks , 1991, Proceedings of the International Conference on Application Specific Array Processors.

[22]  Brian R. Gaines,et al.  Stochastic Computing Systems , 1969 .

[23]  Christian Lehmann,et al.  A generic systolic array building block for neural networks with on-chip learning , 1993, IEEE Trans. Neural Networks.

[24]  Y. Suzuki,et al.  Digital systems for artificial neural networks , 1989, IEEE Circuits and Devices Magazine.

[25]  Javier R. Movellan,et al.  Contrastive Hebbian Learning in the Continuous Hopfield Model , 1991 .

[26]  C. Thomborson,et al.  A Complexity Theory for VLSI , 1980 .

[27]  Kimmo Kaski,et al.  Pulse-density modulation technique in VLSI implementations of neural network algorithms , 1990 .

[28]  Stamatis Vassiliadis,et al.  A VLSI pipelined neuroemulator , 1995 .

[29]  A. Weigend.  Introduction to the theory of neural computation: John A. Hertz, Anders S. Krogh and Richard G. Palmer☆ , 1993 .

[30]  Brian R. Gaines,et al.  Stochastic computing , 1967, AFIPS '67 (Spring).

[31]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[32]  Max Stanford Tomlinson,et al.  A digital neural network architecture for VLSI , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[33]  John Wawrzynek,et al.  The design of a neuro-microprocessor , 1993, IEEE Trans. Neural Networks.

[34]  A. Masaki,et al.  Neural networks in CMOS: a case study , 1990, IEEE Circuits and Devices Magazine.

[35]  Richard Lyon.  VLSI and machines that hear , 1990 .

[36]  Mariagiovanna Sami,et al.  Multistage Interleaved Architectures for Implementation of Neural Networks , 1989 .

[37]  Eugene J. H. Kerckhoffs,et al.  Speeding up backpropagation training on a hypercube computer , 1992, Neurocomputing.

[38]  Alan F. Murray,et al.  Bit-Serial Neural Networks , 1987, NIPS.

[39]  Francesco Piazza,et al.  Fast neural networks without multipliers , 1993, IEEE Trans. Neural Networks.

[40]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[41]  H.C. Card,et al.  Analog Cmos Neural Circuits - In Situ Learning , 1992, Int. J. Neural Syst..

[42]  Mohamed I. Elmasry,et al.  The digi-neocognitron: a digital neocognitron neural network model for VLSI , 1992, IEEE Trans. Neural Networks.

[43]  Renato Stefanelli,et al.  Mapping neural nets onto a massively parallel architecture: a defect-tolerance solution , 1991, Proc. IEEE.

[44]  G. G. Pechanek,et al.  A dataflow approach for neural networks , 1995 .

[45]  R. Pinkham,et al.  An 11-million Transistor Neural Network Execution Engine , 1991, 1991 IEEE International Solid-State Circuits Conference. Digest of Technical Papers.

[46]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[47]  Trevor Clarkson,et al.  VLSI Design of Neural Networks , 1992 .

[48]  Shaun M. Lawson.  A preliminary view of Japan's high performance neurocomputers , 1992, Neurocomputing.

[49]  Tony R. Martinez,et al.  Digital Neural Networks , 1988, Proceedings of the 1988 IEEE International Conference on Systems, Man, and Cybernetics.

[50]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[51]  Leonardo Maria Reyneri,et al.  An Analysis on the Performance of Silicon Implementations of Backpropagation Algorithms for Artificial Neural Networks , 1991, IEEE Trans. Computers.

[52]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[53]  Soheil Shams,et al.  Implementation of Multilayer Neural Networks on Parallel Programmable Digital Computers , 1991 .

[54]  Dharma P. Agrawal,et al.  Generalized Hypercube and Hyperbus Structures for a Computer Network , 1984, IEEE Transactions on Computers.

[55]  J. Ouali,et al.  Fast Generation of Neuro-ASICs , 1990 .

[56]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[57]  W. Robertson,et al.  An hierarchical VLSI neural network architecture , 1992 .

[58]  Ulrich Ramacher,et al.  Synapse-X: a general-purpose neurocomputer architecture , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[59]  I. Aleksander,et al.  WISARD·a radical step forward in image recognition , 1984 .

[60]  Thomas K. Miller,et al.  A digital architecture employing stochasticism for the simulation of Hopfield neural nets , 1989 .

[61]  S. Y. Kung,et al.  Parallel architectures for artificial neural nets , 1988, IEEE 1988 International Conference on Neural Networks.