Matchmaking frameworks for distributed resource management

Federated distributed systems present new challenges to resource management. Conventional resource managers are based on a relatively static resource model and a centralized allocator that assigns resources to customers. Distributed environments, particularly those built to support high-throughput computing (HTC), are often characterized by distributed management and distributed ownership. Distributed management introduces resource heterogeneity: Not only the set of available resources, but even the set of resource types is constantly changing. Distributed ownership introduces policy heterogeneity: Each resource may have its own idiosyncratic allocation policy. We propose a resource management framework based on a matchmaking paradigm to address these shortcomings. Matchmaking services enable discovery and exchange of goods and services in marketplaces. Agents that provide or require services advertise their presence by publishing constraints and preferences on the entities they would like to be matched with, as well as their own characteristics. A matchmaker uses a matching operation to discover pairings between compatible agents. Since the notion of “compatible” is completely determined by the content of agent classified advertisements (classads), a matchmaker can match classads from different kinds of entities in a general manner. Matched agents activate a separate claiming protocol to confirm the match and establish an allocation. The resulting framework is robust, scalable, flexible and evolvable, and has been demonstrated in Condor, a production-quality distributed high throughput computing system developed at the University of Wisconsin-Madison. The goal of this dissertation is to show the power, flexibility, desirability and feasibility of resource management through matchmaking. We detail the architecture and operation of matchmaking frameworks, and describe mechanisms to implement the components and interactions in such systems. We describe the architecture of a matchmaking framework that distinguishes itself by providing both bilateral and multilateral matchmaking (i.e., gangmatching) services. The classad language, a semi-structured agent specification language, is presented, and an indexing model for the classad data model is defined. The indexing solution tolerates the lax semantics of semi-structured data models, and indexes both classad attributes and constraints to efficiently identify compatible advertisements. Finally, algorithms that implement the proposed gangmatching model are described, and their performance characteristics analyzed.