Cheap cycles from the desktop to the dedicated cluster: combining opportunistic and dedicated scheduling with Condor

Clusters of commodity PC hardware running Linux are becoming widely used as computational resources. Most software for controlling clusters relies on dedicated scheduling algorithms. These algorithms assume the constant availability of resources to compute fixed schedules. Unfortunately, due to hardware and software failures, dedicated resources are not always available over the long-term. Moreover, these dedicated scheduling solutions are only applicable to certain classes of jobs, and they can only manage clusters or large SMP machines. The Condor High Throughput Computing System overcomes these limitations by combining aspects of dedicated and opportunistic scheduling into a single system. Both parallel and serial jobs are managed at the same time, allowing a simpler interface for the user and better resource utilization. This paper describes the Condor system, defines opportunistic scheduling, explains how Condor supports MPI jobs with a combination of dedicated and opportunistic scheduling, and shows the advantages gained by such an approach. An exploration of future work in these areas concludes the paper. By using both desktop workstations and dedicated clusters, Condor harnesses all available computational power to enable the best possible science at a low cost.