Language and Run-Time Support for Network Parallel Computing

Network parallel computing is the use of diverse computing resources interconnected by general purpose networks to run parallel applications. This paper describes NetFx, an extension of the Fx compiler system which uses the Fx model of task parallelism to distribute and manage computations across the sequential and parallel machines of a network. A central problem in network parallel computing is that the compiler is presented with a heterogeneous and dynamic target. Our approach is based on a novel run-time system that presents a simple communication interface to the compiler, yet uses compiler knowledge to customize communication between tasks executing over the network. The run-time system is designed to support complete applications developed with different compilers and parallel program generators. It presents a standard communication interface for point-to-point transfer of distributed data sets between tasks. This allows the compiler to be portable, and enables communication generation without knowledge of exactly how the tasks will be mapped at run-time and what low level communication primitives will be used. The compiler also generates a set of custom routines, called address computation functions, to translate between different data distributions. The run-time system performs the actual communication using a mix of generic and custom address computation functions depending on run-time parameters like the type and number of nodes assigned to the communicating tasks and the data distributions of the variables being communicated. This mechanism enables the run-time system to exploit compile-time optimizations, and enables the compiler to manage foreign tasks that use non-standard data distributions. We outline several important applications of network parallel computing and describe the NetFx programming model and run-time system.