AnthillSched: A Scheduling Strategy for Irregular and Iterative I/O-Intensive Parallel Jobs

Irregular and iterative I/O-intensive jobs need a different approach from parallel job schedulers. The focus in this case is not only the processing requirements anymore: memory, network and storage capacity must all be considered in making a scheduling decision. Job executions are irregular and data dependent, alternating between CPU-bound and I/O-bound phases. In this paper, we propose and implement a parallel job scheduling strategy for such jobs, called AnthillSched, based on a simple heuristic: we map the behavior of a parallel application with minimal resources as we vary its input parameters. From that mapping we infer the best scheduling for a certain set of input parameters given the available resources. To test and verify AnthillSched we used logs obtained from a real system executing data mining jobs. Our main contributions are the implementation of a parallel job scheduling strategy in a real system and the performance analysis of AnthillSched, which allowed us to discard some other scheduling alternatives considered previously.

[1]  Dino Pedreschi,et al.  Knowledge Discovery in Databases: PKDD 2004 , 2004, Lecture Notes in Computer Science.

[2]  Anand Sivasubramaniam,et al.  Improving parallel job scheduling by combining gang scheduling and backfilling techniques , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[3]  Dror G. Feitelson,et al.  Gang scheduling with memory considerations , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[4]  Srinivasan Parthasarathy,et al.  Asynchronous and Anticipatory Filter-Stream Based Parallel Algorithm for Frequent Itemset Mining , 2004, PKDD.

[5]  Dror G. Feitelson,et al.  Job Characteristics of a Production Parallel Scientivic Workload on the NASA Ames iPSC/860 , 1995, JSSPP.

[6]  Achim Streit.  A Self-Tuning Job Scheduler Family with Dynamic Policy Switching , 2002, JSSPP.

[7]  Yves Robert,et al.  A realistic model and an efficient heuristic for scheduling with heterogeneous processors , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[8]  Larry Rudolph,et al.  Evaluation of Design Choices for Gang Scheduling Using Distributed Hierarchical Control , 1996, J. Parallel Distributed Comput..

[9]  Dror G. Feitelson,et al.  Paired Gang Scheduling , 2003, IEEE Trans. Parallel Distributed Syst..

[10]  Francisco Vilar Brasileiro,et al.  Exploiting Replication and Data Reuse to Efficiently Schedule Data-Intensive Applications on Grids , 2004, JSSPP.

[11]  Dror G. Feitelson,et al.  Metric and workload effects on computer systems evaluation , 2003, Computer.

[12]  Marco Danelutto,et al.  Euro-Par 2004 Parallel Processing , 2004, Lecture Notes in Computer Science.

[13]  Dror G. Feitelson,et al.  Flexible coscheduling: mitigating load imbalance and improving utilization of heterogeneous resources , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[14]  Anand Sivasubramaniam,et al.  Gang Scheduling Extensions for I/O Intensive Workloads , 2003, JSSPP.

[15]  Richard P. Brent,et al.  Gang scheduling with a queue for large jobs , 2001, Proceedings 15th International Parallel and Distributed Processing Symposium. IPDPS 2001.

[16]  Nazareno Andrade,et al.  OurGrid: An Approach to Easily Assemble Grids with Equitable Resource Sharing , 2003, JSSPP.

[17]  Hai Jin,et al.  Active Disks: Programming Model, Algorithms and Evaluation , 2002 .

[18]  Carla E. Brodley,et al.  An Incremental Method for Finding Multivariate Splits for Decision Trees , 1990, ML.

[19]  Joel H. Saltz,et al.  DataCutter: Middleware for Filtering Very Large Scientific Datasets on Archival Storage Systems , 2000, IEEE Symposium on Mass Storage Systems.

[20]  Luís Fabrício Wanderley Góes,et al.  Reconfigurable Gang Scheduling Algorithm , 2004, JSSPP.

[21]  J. Moreira,et al.  An Evaluation of Parallel Job Scheduling for ASCI Blue-Pacific , 1999, ACM/IEEE SC 1999 Conference (SC'99).

[22]  Fabrício Alves Barbosa da Silva,et al.  A Scheduling Algorithm for Running Bag-of-Tasks Data Mining Applications on the Grid , 2004, Euro-Par.

[23]  Hui Gao,et al.  Parallel and Distributed Processing and Applications , 2005 .

[24]  Larry Rudolph,et al.  Metrics and Benchmarking for Parallel Job Scheduling , 1998, JSSPP.

[25]  Warren Smith,et al.  Benchmarks and Standards for the Evaluation of Parallel Job Schedulers , 1999, JSSPP.

[26]  Lúcia Maria de A. Drummond,et al.  Anthill: a scalable run-time environment for data mining applications , 2005, 17th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD'05).