Cluster Computing on the Fly : P 2 P Scheduling of Idle Cycles in the Internet

Peer-to-peer computing, the harnessing of idle compute cycles throughout the Internet, offers exciting new research challenges in the converging domains of networking and distributed computing. Our system, Cluster Computing on the Fly, seeks to harvest cycles from ordinary users in an open access, non-institutional environment. CCOF encompasses all activities involved in the management of idle cycles: overlay construction for hosts donating cycles, resource discovery within the overlay, application-based scheduling, local scheduling on the host node, and meta-level scheduling among a community of application-level schedulers. In this paper, we identify four important classes of cycle-sharing applications, each requiring its own scheduling strategy: workpile, workpile with deadlines, tree-based search, and point-of-presence. We describe a Wave Scheduler for workpile tasks that exploits idle night-time cycles using a geographic-based overlay. The scheduler incorporates a quizzing mechanism to check the correctness of results and determine trust ratings for the hosts. We also propose a Point-of-Presence Scheduler to discover and schedule hosts that meet application-specific requirements for location, topological distribution, and available resources.