Know Your Customer: Multi-armed Bandits with Capacity Constraints

A wide range of resource allocation and platform operation settings exhibit the following two simultaneous challenges: (1) service resources are capacity constrained; and (2) clients' preferences are not perfectly known. To study this pair of challenges, we consider a service system with heterogeneous servers and clients. Server types are known and there is fixed capacity of servers of each type. Clients arrive over time, with types initially unknown and drawn from some distribution. Each client sequentially brings $N$ jobs before leaving. The system operator assigns each job to some server type, resulting in a payoff whose distribution depends on the client and server types. Our main contribution is a complete characterization of the structure of the optimal policy for maximization of the rate of payoff accumulation. Such a policy must balance three goals: (i) earning immediate payoffs; (ii) learning client types to increase future payoffs; and (iii) satisfying the capacity constraints. We construct a policy that has provably optimal regret (to leading order as $N$ grows large). Our policy has an appealingly simple three-phase structure: a short type-"guessing" phase, a type-"confirmation" phase that balances payoffs with learning, and finally an "exploitation" phase that focuses on payoffs. Crucially, our approach employs the shadow prices of the capacity constraints in the assignment problem with known types as "externality prices" on the servers' capacity.