Incentives in Resource Allocation under Dynamic Demands

Every computer system—from schedulers in public clouds (e.g., Amazon, Google, Microsoft etc.) to computer networks to hypervisors to operating systems—performs resource allocation across system users. The defacto allocation policy used in most of these systems, max-min fairness, guarantees many desirable properties like incentive compatibility and Pareto efficiency, assuming user demands are static (time-independent). However, in modern real-world production systems, user demands are dynamic, that is, vary over time. As a result, there is now a fundamental mismatch between the resource allocation goals of computer systems, and the properties enabled by classical resource allocation policies. This paper aims to bridge this mismatch. We consider a natural generalization of the classical algorithm for max-min fair resource allocation for the case of dynamic demands: this algorithm guarantees Pareto optimality, while ensuring that resources allocated to users are as max-min fair as possible up to any time instant, given the allocation in previous periods. While this dynamic allocation scheme remains Pareto optimal, unfortunately, it is not incentive compatible. Our results show that the possible increase in utility by misreporting demand is minimal and, since this misreporting can lead to significant decrease in overall useful allocation, this suggests that it is not a useful strategy. Our main result is to show that when user demands are independent random variables, increasing the total amount of resource by a (1+O(ε)) factor compared to the sum of expected instantaneous user demands, makes the algorithm (1+ ε)-incentive compatible, that is, no user can improve her allocation by more than a factor of (1 + ε) by misreporting her demands, where ε → 0 as n → ∞ and n is the number of users. In the adversarial setting, we show that this algorithm is 3/2-incentive compatible; we also show that this factor is nearly tight. We also present generalization of the above results for the case of multiple colluding users and multi-resource allocation.

[1]  Randy H. Katz,et al.  Heterogeneity and dynamicity of clouds at scale: Google trace analysis , 2012, SoCC '12.

[2]  Ali Anwar,et al.  Analyzing Alibaba’s Co-located Datacenter Workloads , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[3]  K. V. Rashmi,et al.  A large scale analysis of hundreds of in-memory cache clusters at Twitter , 2020, OSDI.

[4]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.

[5]  Eric Budish,et al.  Strategy-Proofness in the Large , 2017, The Review of Economic Studies.

[6]  Wei Lin,et al.  Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing , 2014, OSDI.

[7]  Albert G. Greenberg,et al.  Data center TCP (DCTCP) , 2010, SIGCOMM '10.

[8]  Sven Seuken,et al.  An axiomatic approach to characterizing and relaxing strategyproofness of one-sided matching mechanisms , 2014, EC.

[9]  Maria-Florina Balcan,et al.  Estimating Approximate Incentive Compatibility , 2019, EC.

[10]  W H I T E P A P E R,et al.  The Architecture of VMware ESXi , 2008 .

[11]  Ariel D. Procaccia,et al.  Incentive compatible regression learning , 2008, SODA '08.

[12]  Ridi Hossain,et al.  Sharing is Caring: Dynamic Mechanism for Shared Resource Ownership , 2019, AAMAS.

[13]  Srikanth Kandula,et al.  Multi-resource packing for cluster schedulers , 2014 .

[14]  Anurag Gupta,et al.  Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases , 2017, SIGMOD Conference.

[15]  Miguel Elias M. Campista,et al.  Stateful DRF: Considering the Past in a Multi-Resource Allocation , 2020, IEEE Transactions on Computers.

[16]  Éva Tardos,et al.  An approximate truthful mechanism for combinatorial auctions with single parameter agents , 2003, SODA '03.

[17]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[18]  David Zeng,et al.  Fairness-Efficiency Tradeoffs in Dynamic Fair Division , 2019, EC.

[19]  Benjamin Hindman,et al.  Dominant Resource Fairness: Fair Allocation of Multiple Resource Types , 2011, NSDI.

[20]  Michael I. Jordan,et al.  Online Learning Demands in Max-min Fairness , 2020, ArXiv.

[21]  Aditya Akella,et al.  Altruistic Scheduling in Multi-Resource Clusters , 2016, OSDI.

[22]  Toby Walsh,et al.  Strategy-Proofness, Envy-Freeness and Pareto Efficiency in Online Fair Division with Additive Utilities , 2019, PRICAI.

[23]  Ashish Motivala,et al.  Building An Elastic Query Engine on Disaggregated Storage , 2020, NSDI.

[24]  Subhash Suri,et al.  Approximately-strategyproof and tractable multi-unit auctions , 2003, EC '03.

[25]  Carlo Curino,et al.  Apache Hadoop YARN: yet another resource negotiator , 2013, SoCC.

[26]  Paul Dütting,et al.  Payment Rules through Discriminant-Based Classifiers , 2012, ACM Trans. Economics and Comput..

[27]  Anees Shaikh,et al.  Performance Isolation and Fairness for Multi-Tenant Cloud Storage , 2012, OSDI.

[28]  Vincent Conitzer,et al.  Dynamic Proportional Sharing: A Game-Theoretic Approach , 2018, SIGMETRICS.

[29]  Raj Jain,et al.  Analysis of the Increase and Decrease Algorithms for Congestion Avoidance in Computer Networks , 1989, Comput. Networks.

[30]  E. Maasland,et al.  Auction Theory , 2021, Springer Texts in Business and Economics.

[31]  Mor Harchol-Balter,et al.  The CacheLib Caching Engine: Design and Experiences at Scale , 2020, OSDI.

[32]  Abhishek Verma,et al.  Large-scale cluster management at Google with Borg , 2015, EuroSys.

[33]  Srikanth Kandula,et al.  This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Graphene: Packing and Dependency-aware Scheduling for Data-parallel Clusters G: Packing and Dependency-aware Scheduling for Data-parallel Clusters , 2022 .