Accessing Sections of Out-of-Core Arrays Using an Extended Two-Phase Method

In out-of-core computations, data needs to be moved back and forth between main memory and disks during program execution. In this paper, we propose a technique called the Extended Two-Phase Method, for accessing sections of out-of-core arrays e ciently. This is an extension and generalization of the Two-Phase Method for reading in-core arrays from les, which was previously proposed in [7, 3]. The Extended Two-Phase Method uses collective I/O in which all processors cooperate to perform I/O in an e cient manner by combining several I/O requests into fewer larger requests, eliminating multiple disk accesses for the same data and reducing contention for disks. We describe the algorithms for reading as well as writing array sections. Performance results on the Intel Touchstone Delta for many di erent access patterns are presented and analyzed. It is observed that the Extended Two-Phase Method gives consistently good performance over a wide range of access patterns. This work was supported in part by NSF Young Investigator Award CCR-9357840, grants from Intel SSD and IBM Corp., in part by USRA CESDIS Contract # 5555-26 and also in part by ARPA under contract # DABT63-91C-0028. The content of the information does not necessarily re ect the position or the policy of the US Government and no o cial endorsement should be inferred. Rajeev Thakur is supported by a Syracuse University Graduate Fellowship. This work was performed in part using the Intel Touchstone Delta System operated by Caltech on behalf of the Concurrent Supercomputing Consortium. Access to this facility was provided by CRPC.