Competitive neural networks on message-passing parallel computers

The paper reports two techniques for parallelizing on a MIMD multicomputer a class of learning algorithms (competitive learning) for artificial neural networks widely used in pattern recognition and understanding. The first technique presented, following the divide et impera strategy, achieves O(n/p + logP) time for n neurons and P processors interconnected as a tree. A modification of the algorithm allows the application of a systolic technique with the processors interconnected as a ring; this technique has the advantage that the communication time does not depend on the number of processors. The two techniques are also compared on the basis of predicted and measured performance on a transputer-based MIMD machine. As the number of processors grows the advantage of the systolic approach increases. On the contrary, the divide et impera approach is more advantageous in the retrieving phase.

[1]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[2]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[3]  Roman Bek,et al.  Discourse on one way in which a quantum-mechanics language on the classical logical base can be built up , 1978, Kybernetika.

[4]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[5]  D. Roweth,et al.  Implementing Neural Network Models on Parallel Computers , 1987, Comput. J..

[6]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[7]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[8]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[9]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[10]  P. van Renterghem.  Transputers for Industrial Applications , 1989, Concurr. Pract. Exp..

[11]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[12]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[13]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[14]  Wolfgang Banzhaf,et al.  Learning in a competitive network , 1990, Neural Networks.

[15]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[16]  Jill P. Mesirov,et al.  The backpropagation algorithm on grid and hypercube architectures , 1990, Parallel Comput..

[17]  David M. Clark,et al.  A convergence theorem for Grossberg learning , 1990, Neural Networks.

[18]  Helge J. Ritter,et al.  Large-scale simulations of self-organizing neural networks on parallel computers: application to biological modelling , 1990, Parallel Comput..

[19]  Shun-ichi Amari,et al.  Mathematical foundations of neurocomputing , 1990, Proc. IEEE.

[20]  Geoffrey E. Hinton,et al.  A time-delay neural network architecture for isolated word recognition , 1990, Neural Networks.

[21]  Bertil Svensson,et al.  Using and Designing Massively Parallel Computers for Artificial Neural Neural Networks , 1992, J. Parallel Distributed Comput..

[22]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.