Using and Designing Massively Parallel Computers for Artificial Neural Neural Networks

Abstract During the past 10 years the fields of artificial neural networks (ANNs) and massively parallel computing have been evolving rapidly. In this paper we study the attempts to make ANN algorithms run on massively parallel computers as well as designs of new parallel systems tuned for ANN computing. Following a brief survey of the most commonly used models, the different dimensions of parallelism in ANN computing are identified, and the possibilities for mapping onto the structures of different parallel architectures are analyzed. Different classes of parallel architectures used or designed for ANN are identified. Reported implementations are reviewed and discussed. It is concluded that the regularity of ANN computations suits SIMD architectures perfectly and that broadcast or ring communication can be very efficiently utilized. Bit-serial processing is very interesting for ANN, but hardware support for multiplication should be included. Future artificial neural systems for real-time applications will require flexible processing modules that can be put together to form MIMSIMD systems.

[1]  Francesco Piazza,et al.  Multi-layer perceptrons with discrete weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[2]  Alexander Singer.  Exploiting the Inherent Parallelism of Artificial Neural Networks to Achieve 1300 Million Interconnects per Second , 1990 .

[3]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[4]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[5]  Michael A. Arbib,et al.  Schemas and Neural Networks for Sixth Generation Computing. Invited Survey , 1989, J. Parallel Distributed Comput..

[6]  Anargyros Krikelis,et al.  Implementing Neural Networks with the Associative String Processor , 1991 .

[7]  R. M. Lea,et al.  ASP: a cost-effective parallel microcomputer , 1988, IEEE Micro.

[8]  D. Roweth,et al.  Implementing Neural Network Models on Parallel Computers , 1987, Comput. J..

[9]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[10]  Carsten Peterson,et al.  Explorations of the mean field theory learning algorithm , 1989, Neural Networks.

[11]  R Lewin,et al.  RNA plasmid discovered in maize mitochondria. , 1986, Science.

[12]  Carl Diegert.  Out-of-core backpropagation , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[13]  J. R. Brown,et al.  Artificial neural network on a SIMD architecture , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[14]  David Rogers,et al.  Kanerva's Sparse Distributed Memory: an Associative Memory Algorithm Well-Suited to the Connection Machine , 1989, Int. J. High Speed Comput..

[15]  J. A. Sirat,et al.  Unlimited accuracy in layered networks , 1989 .

[16]  Jerry L. Potter.  The Massively Parallel Processor , 1985 .

[17]  Bertil Svensson,et al.  LUCAS Associative Array Processor: Design, Programming and Application Studies , 1986 .

[18]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[19]  Jeanny Herault,et al.  Smart: How to Simulate Huge Networks , 1990 .

[20]  Peter Christy,et al.  Software to support massively parallel computing on the MasPar MP-1 , 1990, Digest of Papers Compcon Spring '90. Thirty-Fifth IEEE Computer Society International Conference on Intellectual Leverage.

[21]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[22]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[23]  Manoj Kumar,et al.  The GF11 Parallel Computer , 1993, Parallel Comput..

[24]  Tom Blank,et al.  The MasPar MP-1 architecture , 1990, Digest of Papers Compcon Spring '90. Thirty-Fifth IEEE Computer Society International Conference on Intellectual Leverage.

[25]  Michael J. Witbrock,et al.  An implementation of backpropagation learning on GF11, a large SIMD parallel computer , 1990, Parallel Comput..

[26]  Karl Goser,et al.  Systolic Synthesis of Neural Networks , 1990 .

[27]  D. Hubel.  Eye, brain, and vision , 1988 .

[28]  David Rogers,et al.  Statistical Prediction with Kanerva's Sparse Distributed Memory , 1988, NIPS.

[29]  J. Beichter,et al.  Design of a 1st Generation Neurocomputer , 1991 .

[30]  S. Makram-Ebeid,et al.  A rationalized error back-propagation learning algorithm , 1989, International 1989 Joint Conference on Neural Networks.

[31]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[32]  Bertil Svensson,et al.  Execution of neural network algorithms on an array of bit-serial processors , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[33]  Bernard Widrow,et al.  Sensitivity of feedforward neural networks to weight errors , 1990, IEEE Trans. Neural Networks.

[34]  S. Y. Kung,et al.  Parallel architectures for artificial neural nets , 1988, IEEE 1988 International Conference on Neural Networks.

[35]  Manoel Fernando Tenorio,et al.  The Cocktail Party Problem: Speech/Data Signal Separation Comparison between Backpropagation and SONN , 1989, NIPS.

[36]  Etienne Deprit.  Implementing recurrent back-propagation on the connection machine , 1989, Neural Networks.

[37]  H. T. Kung,et al.  The Warp Computer: Architecture, Implementation, and Performance , 1987, IEEE Transactions on Computers.

[38]  S. Y. King.  Parallel architectures for artificial neural nets , 1988, [1988] Proceedings. International Conference on Systolic Arrays.

[39]  Colin Whitby-Strevens.  Transputers-past, present and future , 1990, IEEE Micro.

[40]  D. Roweth,et al.  Neural network models , 1988, Parallel Comput..

[41]  W. Daniel Hillis,et al.  Data parallel algorithms , 1986, CACM.

[42]  L. B. Lmeida.  Backpropagation in perceptrons with feedback , 1988 .

[43]  John R. Nickolls,et al.  The design of the MasPar MP-1: a cost effective massively parallel computer , 1990, Digest of Papers Compcon Spring '90. Thirty-Fifth IEEE Computer Society International Conference on Intellectual Leverage.

[44]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[45]  Guy E. Blelloch,et al.  An implementation of network learning on the Connection Machine , 1988 .

[46]  James Sutton,et al.  iWarp: a 100-MOPS, LIW microprocessor for multicomputers , 1991, IEEE Micro.

[47]  Stephen S. Wilson.  Neural Computing on a One Dimensional SIMD Array , 1989, IJCAI.

[48]  A. F. Murray.  Silicon implementation of neural networks , 1991 .

[49]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[50]  Hecht-Nielsen.  Theory of the backpropagation neural network , 1989 .

[51]  S. S. Wilson,et al.  The AIS-5000 Parallel Processor , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  P. Treleaven,et al.  THE PYGMALION NEURAL NETWORK PROGRAMMING ENVIRONMENT , 1990 .

[53]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[54]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Sherryl Tomboulian.  Introduction to a System for Implementing Neural Net Connections on SIMD Architectures , 1987, NIPS.

[56]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[57]  A. Masaki,et al.  Neural networks in CMOS: a case study , 1990, IEEE Circuits and Devices Magazine.

[58]  Tomas Nordström,et al.  Designing parallel computers for self organizing maps , 1991 .

[59]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[60]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[61]  Kenji Nakayama,et al.  A digital multilayer neural network with limited binary expressions , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[62]  P. Koikkalainen,et al.  Specification and implementation environment for neural networks using communicating sequential processes , 1988, IEEE 1988 International Conference on Neural Networks.

[63]  Bernard Faure,et al.  Implementation of Back-Propagation on a VLSI Asynchronous Cellular Architecture , 1990 .

[64]  John von Neumann,et al.  The Computer and the Brain , 1960 .

[65]  C. Chen,et al.  Systolic array implementations of neural nets on the MasPar MP-1 massively parallel processor , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[66]  H. T. Kung.  Why systolic architectures? , 1982, Computer.

[67]  Alfred Ultsch,et al.  Kohonen Networks on Transputers: Implementation and Animation , 1990 .

[68]  Jeff A. Bilmes,et al.  The RAP: a ring array processor for layered network calculations , 1990, [1990] Proceedings of the International Conference on Application Specific Array Processors.

[69]  Manoel Fernando Tenorio,et al.  Self-organizing network for optimum supervised learning , 1990, IEEE Trans. Neural Networks.

[70]  Daniele D. Caviglia,et al.  Effects of weight discretization on the back propagation learning method: algorithm design and hardware realization , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[71]  K. W. Przytula,et al.  Mapping of neural networks onto programmable parallel machines , 1990, IEEE International Symposium on Circuits and Systems.

[72]  M. Duranton,et al.  Learning on VLSI: a general purpose digital neurochip , 1989, International 1989 Joint Conference on Neural Networks.

[73]  S. Tomboulian,et al.  Neural Network Simulation on the MasPar MP-1 Massively Parallel Processor , 1990 .

[74]  M. Vellasco,et al.  VLSI architectures for neural networks , 1989, IEEE Micro.

[75]  Michael J. Flynn,et al.  Some Computer Organizations and Their Effectiveness , 1972, IEEE Transactions on Computers.

[76]  T. Watanabe,et al.  Neural network simulation on a massively parallel cellular array processor: AAP-2 , 1989, International 1989 Joint Conference on Neural Networks.

[77]  Manoel Fernando Tenorio,et al.  Topology synthesis networks: self-organization of structure and weight adjustment as a learning paradigm , 1990, Parallel Comput..

[78]  Edward W. Davis,et al.  BLITZEN: a highly integrated massively parallel machine , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[79]  José A. B. Fortes,et al.  Performance of Connectionist Learning Algorithms on 2-D SIMD Processor Arrays , 1989, NIPS.

[80]  Y. Fujimoto.  An enhanced parallel planar lattice architecture for large scale neural network simulations , 1990 .

[81]  D. J. Hunt.  AMT DAP—a processor array in a workstation environment , 1989 .

[82]  S. C. Barash,et al.  The systolic array neurocomputer: fine-grained parallelism at the synaptic level , 1989, International 1989 Joint Conference on Neural Networks.

[83]  K. C. Bowler,et al.  An Introduction to OCCAM 2 Programming , 1989 .

[84]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[85]  Alexander Singer,et al.  Implementations of artificial neural networks on the Connection Machine , 1990, Parallel Comput..

[86]  Jenq-Neng Hwang,et al.  A systolic neural network architecture for hidden Markov models , 1989, IEEE Trans. Acoust. Speech Signal Process..

[87]  Louis A. Jaeckel.  Some methods of encoding simple visual images for use with a sparse distributed memory, with applications to character recognition , 1989 .

[88]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[89]  James L. McClelland.  Explorations In Parallel Distributed Processing , 1988 .

[90]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[91]  N Petkov.  Systolic simulation of multilayer, feedforward neural networks , 1990 .

[92]  Helge J. Ritter,et al.  Large-scale simulations of self-organizing neural networks on parallel computers: application to biological modelling , 1990, Parallel Comput..

[93]  Nathan H. Brown.  Neural Network Implementation Approaches for the Connection Machine , 1987, NIPS.

[94]  A. Masaki,et al.  A wafer scale integration neural network utilizing completely digital circuits , 1989, International 1989 Joint Conference on Neural Networks.

[95]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.