A perspective on the future of massively parallel computing: fine-grain vs. coarse-grain parallel models comparison & contrast

Models, architectures and languages for parallel computation have been of utmost research interest in computer science and engineering for several decades. A great variety of parallel computation models has been proposed and studied, and different parallel and distributed architectures designed as some possible ways of harnessing parallelism and improving performance of the general purpose computers.Massively parallel connectionist models such as artificial neural networks (ANNs) and cellular automata (CA) have been primarily studied in domain-specific contexts, namely, learning and complex dynamics, respectively. However, they can also be viewed as generic abstract models of massively parallel computers that are in many respects fundamentally different from the "main stream" parallel and distributed computation models.We compare and contrast herewith the parallel computers as they have been built by the engineers with those built by Nature. We subsequently venture onto a high-level discussion of the properties and potential advantages of the proposed massively parallel computers of the future that would be based on the fine-grained connectionist parallel models, rather than on either various multiprocessor architectures, or networked distributed systems, which are the two main architecture paradigms in building parallel computers of the late 20th and early 21st centuries. The comparisons and contrasts herein are focusing on the fundamental conceptual characteristics of various models rather than any particular engineering idiosyncrasies, and are carried out at both structural and functional levels. The fundamental distinctions between the fine-grain connectionist parallel models and their "classical" coarse-grain counterparts are discussed, and some important expected advantages of the hypothetical massively parallel computers based on the connectionist paradigms conjectured.We conclude with some brief remarks on the role that the paradigms, concepts, and design ideas originating from the connectionist models have already had in the existing parallel design, and what further role the connectionist models may have in the foreseeable future of parallel and distributed computing.