Asynchronous finite-difference schemes for partial differential equations

Current trends in massively parallel computing systems suggest that the number of processing elements (PEs) used in simulations will continue to grow over time. A known problem in this context is the overhead associated with communication and/or synchronization between PEs as well as idling due to load imbalances. Simulation at extreme levels of parallelism will then require an elimination, or at least a tight control of these overheads. In this work, we present an analysis of common finite difference schemes for partial differential equations (PDEs) when no synchronization between PEs is enforced. PEs are allowed to continue computations regardless of messages status and are thus asynchronous. We show that while stability is conserved when these schemes are used asynchronously, accuracy is greatly degraded. Since message arrivals at PEs are essentially random processes, so is the behavior of the error. Within a statistical framework we show that average errors drop always to first-order regardless of the original scheme. The value of the error is found to depend on both grid spacing as well as characteristics of the computing system including number of processors and statistics of the delays. We propose new schemes that are robust to asynchrony. The analytical results are compared against numerical simulations.