A Combinatorial-Bandit Algorithm for the Online Joint Bid/Budget Optimization of Pay-per-Click Advertising Campaigns

Pay-per-click advertising includes various formats (e.g., search, contextual, and social) with a total investment of more than 140 billion USD per year. An advertising campaign is composed of some subcampaigns—each with a different ad—and a cumulative daily budget. The allocation of the ads is ruled exploiting auction mechanisms. In this paper, we propose, for the first time to the best of our knowledge, an algorithm for the online joint bid/budget optimization of payper-click multi-channel advertising campaigns. We formulate the optimization problem as a combinatorial bandit problem, in which we use Gaussian Processes to estimate stochastic functions, Bayesian bandit techniques to address the exploration/exploitation problem, and a dynamic programming technique to solve a variation of the Multiple-Choice Knapsack problem. We experimentally evaluate our algorithm both in simulation—using a synthetic setting generated a Yahoo! dataset—and in a real-world application for two months.

[1]  W. R. Thompson ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[2]  T. Ibaraki,et al.  THE MULTIPLE-CHOICE KNAPSACK PROBLEM , 1978 .

[3]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[4]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[5]  Yi Gai,et al.  Learning Multiuser Channel Allocations in Cognitive Radio Networks: A Combinatorial Multi-Armed Bandit Formulation , 2010, 2010 IEEE Symposium on New Frontiers in Dynamic Spectrum (DySPAN).

[6]  Ole-Christoffer Granmo,et al.  Solving two-armed Bernoulli bandit problems using a Bayesian learning automaton , 2010, Int. J. Intell. Comput. Cybern..

[7]  Evangelos Markakis,et al.  Discrete Strategies in Keyword Auctions and Their Inefficiency for Locally Aware Bidders , 2010, WINE.

[8]  Benedict C. May Simulation Studies in Optimistic Bayesian Sampling in Contextual-Bandit Problems , 2011 .

[9]  Lihong Li,et al.  An Empirical Evaluation of Thompson Sampling , 2011, NIPS.

[10]  Tie-Yan Liu,et al.  Joint optimization of bid and budget allocation in sponsored search , 2012, KDD.

[11]  Aurélien Garivier,et al.  On Bayesian Upper Confidence Bounds for Bandit Problems , 2012, AISTATS.

[12]  David S. Leslie,et al.  Optimistic Bayesian Sampling in Contextual-Bandit Problems , 2012, J. Mach. Learn. Res..

[13]  Ali Jalali,et al.  Real time bid optimization with smooth budget delivery in online advertising , 2013, ADKDD '13.

[14]  Tao Qin,et al.  Multi-Armed Bandit with Budget Constraint and Variable Costs , 2013, AAAI.

[15]  Wei Chen,et al.  Combinatorial multi-armed bandit: general framework, results and applications , 2013, ICML 2013.

[16]  Michalis Vazirgiannis,et al.  Toward an integrated framework for automated development and optimization of online advertising campaigns , 2014, Intell. Data Anal..

[17]  Tao Qin,et al.  Sponsored Search Auctions , 2014, ACM Trans. Intell. Syst. Technol..

[18]  H. Varian,et al.  The VCG Auction in Theory and Practice , 2014 .

[19]  Sahin Cem Geyik,et al.  Multi-Touch Attribution Based Budget Allocation in Online Advertising , 2014, ADKDD'14.

[20]  Weinan Zhang,et al.  Optimal real-time bidding for display advertising , 2014, KDD.

[21]  Nicola Gatti,et al.  Truthful learning mechanisms for multi-slot sponsored search auctions with externalities , 2012, Artif. Intell..

[22]  Jian Xu,et al.  Smart Pacing for Effective Online Ad Campaign Optimization , 2015, KDD.

[23]  Nenghai Yu,et al.  Thompson Sampling for Budgeted Multi-Armed Bandits , 2015, IJCAI.

[24]  Jun Wang,et al.  Feedback Control of Real-Time Display Advertising , 2016, WSDM.

[25]  Pavel Kireyev,et al.  Do display ads influence search? Attribution and dynamics in online advertising , 2016 .

[26]  Jianqiang Shen,et al.  Joint Optimization of Multiple Performance Metrics in Online Video Advertising , 2016, KDD.

[27]  Marcello Restelli,et al.  Budgeted Multi-Armed Bandit in Continuous Action Space , 2016, ECAI.

[28]  Yajun Wang,et al.  Combinatorial Multi-Armed Bandit and Its Extension to Probabilistically Triggered Arms , 2014, J. Mach. Learn. Res..

[29]  Santiago Ontañón,et al.  Combinatorial Multi-armed Bandits for Real-Time Strategy Games , 2017, J. Artif. Intell. Res..

[30]  Marcello Restelli,et al.  Unimodal Thompson Sampling for Graph-Structured Arms , 2017, AAAI.

[31]  Nicola Gatti,et al.  Adopting the Cascade Model in Ad Auctions: Efficiency Bounds and Truthful Algorithmic Mechanisms , 2017, J. Artif. Intell. Res..

[32]  Jun Wang,et al.  Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting , 2016, Found. Trends Inf. Retr..