Scalable Massively Parallel Artificial Neural Networks

Artificial Neural Networks (ANN) can be very effective for pattern recognition, function approximation, scientific classification, control, and the analysis of time series data; however they can require very large training times for large networks. Once the network is trained for a particular problem, however, it can produce results in a very short time. Traditional ANNs using back-propagation algorithm do not scale well as each neuron in one level is fully connected to each neuron in the previous level. In the present work only the neurons at the edges of the domains were involved in communication, in order to reduce the communication costs and maintain scalability. Ghost neurons were created at these processor boundaries for information communication. An object-oriented, massively-parallel ANN software package SPANN (Scalable Parallel Artificial Neural Network) has been developed and is described here. MPI was used to parallelize the C++ code. The back-propagation algorithm was used to train the network. In preliminary tests, the software was used to identify character sets consisting of 48 characters and with increasing resolutions. The code correctly identified all the characters when adequate training was used in the network. The training of a problem sizewith2billionneuronweightsonanIBMBlueGene/Lcomputerusing1000dualPowerPC 440 processors required less than 30 minutes.Various comparisons in training time, forward propagation time, and error reduction were also made.

[1]  Stephen Grossberg,et al.  Adaptive Resonance Theory , 2010, Encyclopedia of Machine Learning.

[2]  Hans P. Moravec.  Robot: Mere Machine to Transcendent Mind , 1998 .

[3]  Paul J. Werbos.  Backpropagation: basics and new developments , 1998 .

[4]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[5]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[6]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[7]  Mohamad Adnan Al-Alaoui,et al.  A cloning approach to classifier training , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[8]  O. Egecioglu,et al.  Communication Parameter Tests and Parallel Back Propagation Algorithms on iPSC/2 Hypercube Multiprocessor , 1990, Proceedings of the Fifth Distributed Memory Computing Conference, 1990..

[9]  Guy E. Blelloch,et al.  Network Learning on the Connection Machine , 1987, IJCAI.

[10]  Minesh B. Amin,et al.  A Scalable Parallel Formulation of the Backpropagation Algorithm for Hypercubes and Related Architectures , 1994, IEEE Trans. Parallel Distributed Syst..

[11]  Wulfram Gerstner,et al.  Spiking Neuron Models , 2002 .

[12]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation (3rd Edition) , 2007 .

[13]  Jill P. Mesirov,et al.  An Efficient Implementation of the Back-propagation Algorithm on the Connection Machine CM-2 , 1989, NIPS.

[14]  Jill P. Mesirov,et al.  The backpropagation algorithm on grid and hypercube architectures , 1990, Parallel Comput..

[15]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[16]  Yusuf Özyörük,et al.  A new efficient algorithm for computational aeroacoustics on massively parallel computers , 1995 .

[17]  Thomas Dean,et al.  A Computational Model of the Cerebral Cortex , 2005, AAAI.

[18]  Lyle N. Long,et al.  Massively parallel three-dimensional Euler/Navier-Stokes method , 1991 .

[19]  Donald A. Sofge,et al.  Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches , 1992 .

[20]  Jean-Luc Gaudiot,et al.  Implementing regularly structured neural networks on the DREAM machine , 1995, IEEE Trans. Neural Networks.

[21]  Sherryl Tomboulian.  Introduction to a System for Implementing Neural Net Connections on SIMD Architectures , 1987, NIPS.

[22]  S. Grossberg.  Adaptive Resonance Theory , 2006 .

[23]  Magdy A. Bayoumi,et al.  Efficient Mapping Algorithm of Multilayer Neural Network on Torus Architecture , 2003, IEEE Trans. Parallel Distributed Syst..

[24]  Yoshiyasu Takefuji,et al.  Neural network parallel computing , 1992, The Kluwer international series in engineering and computer science.

[25]  Victor Shtern.  Core C++: A Software Engineering Approach , 2000 .

[26]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[27]  Vernon B Mountcastle,et al.  Introduction. Computation in cortical columns. , 2003, Cerebral cortex.

[28]  Viktor K. Prasanna,et al.  Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines , 1991, IEEE Trans. Computers.

[29]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[30]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[31]  M.A. Bayoumi,et al.  An efficient mapping of multilayer perceptron with backpropagation ANNs on hypercubes , 1993, Proceedings of 1993 5th IEEE Symposium on Parallel and Distributed Processing.

[32]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[33]  Udo Seiffert,et al.  Artificial Neural Networks on Massively Parallel Computer Hardware , 2004, ESANN.

[34]  Peter A. Darnell,et al.  C: A Software Engineering Approach , 1991, Springer Books on Professional Computing.

[35]  R. Kurzweil.  The Age of Spiritual Machines , 1999 .

[36]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[37]  J. Nazuno.  Haykin, Simon. Neural networks: A comprehensive foundation, Prentice Hall, Inc. Segunda Edición, 1999 , 2000 .

[38]  Soumitra Dutta,et al.  Bond rating: A non-conservative application of neural networks , 1988 .

[39]  Magdy Bayoumi,et al.  An efficient implementation of multi-layer perceptron on mesh architecture , 2002, 2002 IEEE International Symposium on Circuits and Systems. Proceedings (Cat. No.02CH37353).

[40]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[41]  David Mumford,et al.  On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[42]  D Mumford,et al.  On the computational architecture of the neocortex. II. The role of cortico-cortical loops. , 1992, Biological cybernetics.

[43]  Shashi Shekhar,et al.  Generalization Performance of Feed-Forward Neural Networks , 1992 .

[44]  W. Daniel Hillis,et al.  The connection machine , 1985 .

[45]  H. White,et al.  Economic prediction using neural networks: the case of IBM daily stock returns , 1988, IEEE 1988 International Conference on Neural Networks.

[46]  Michael J. Witbrock,et al.  An implementation of backpropagation learning on GF11, a large SIMD parallel computer , 1990, Parallel Comput..

[47]  Tack-Don Han,et al.  Mapping of neural networks onto the memory-processor integrated architecture , 1998, Neural Networks.

[48]  Jenq-Neng Hwang,et al.  Parallel algorithms/architectures for neural networks , 1989, J. VLSI Signal Process..

[49]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.

[50]  J. Taylor,et al.  Neural networks of the brain: their analysis and relation to brain images , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[51]  Hyunsoo Yoon,et al.  Multilayer Neural Networks on Distributed-Memory Multiprocessors , 1990 .

[52]  Daoqi Yang,et al.  C++ and Object-Oriented Numeric Computing for Scientists and Engineers , 2000, Springer New York.

[53]  D. Mumford.  On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[54]  A. PETROWSKI,et al.  PARALLEL IMPLEMENTATIONS OF NEURAL NETWORK SIMULATIONS , 2007 .

[55]  D. Mumford,et al.  On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[56]  David I. Lewin,et al.  Supercomputing '93 Showcases Parallel Computing Finalists , 1993 .