Robust Market Making via Adversarial Reinforcement Learning

We show that adversarial reinforcement learning (ARL) can be used to produce market marking agents that are robust to adversarial and adaptively-chosen market conditions. To apply ARL, we turn the well-studied single-agent model of Avellaneda and Stoikov [2008] into a discrete-time zero-sum game between a market maker and adversary. The adversary acts as a proxy for other market participants that would like to profit at the market maker's expense. We empirically compare two conventional single-agent RL agents with ARL, and show that our ARL approach leads to: 1) the emergence of risk-averse behaviour without constraints or domain-specific penalties; 2) significant improvements in performance across a set of standard metrics, evaluated with or without an adversary in the test environment, and; 3) improved robustness to model uncertainty. We empirically demonstrate that our ARL method consistently converges, and we prove for several special cases that the profiles that we converge to correspond to Nash equilibria in a simplified single-stage game.

[1]  Marc G. Bellemare,et al.  A Distributional Perspective on Reinforcement Learning , 2017, ICML.

[2]  Javier García,et al.  A comprehensive survey on safe reinforcement learning , 2015, J. Mach. Learn. Res..

[3]  Olivier Guéant,et al.  Dealing with the inventory risk: a solution to the market making problem , 2011, 1105.3115.

[4]  FernándezFernando,et al.  A comprehensive survey on safe reinforcement learning , 2015 .

[5]  Balaraman Ravindran,et al.  EPOpt: Learning Robust Neural Network Policies Using Model Ensembles , 2016, ICLR.

[6]  Svitlana Vyetrenko,et al.  Risk-Sensitive Compact Decision Trees for Autonomous Execution in Presence of Simulated Market Response , 2019, ArXiv.

[7]  M. Sion.  On general minimax theorems , 1958 .

[8]  John Fearnley,et al.  Market Making via Reinforcement Learning , 2018, AAMAS.

[9]  Jacob D. Abernethy,et al.  Adaptive Market Making via Online Learning , 2013, NIPS.

[10]  Olivier Guéant,et al.  Optimal market making , 2016, 1605.01862.

[11]  Olivier Pietquin,et al.  Actor-Critic Fictitious Play in Simultaneous Move Multistage Games , 2018, AISTATS.

[12]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[13]  Shie Mannor,et al.  Policy Gradients with Variance Related Risk Criteria , 2012, ICML.

[14]  Philip Thomas,et al.  Bias in Natural Actor-Critic Algorithms , 2014, ICML.

[15]  Marcos Lopez de Prado,et al.  Algorithmic and High Frequency Trading , 2016 .

[16]  T. Ho,et al.  Optimal dealer pricing under transactions and return uncertainty , 1981 .

[17]  Sebastian Jaimungal,et al.  Algorithmic Trading with Model Uncertainty , 2017, SIAM J. Financial Math..

[18]  M. Avellaneda,et al.  High-frequency trading in a limit order book , 2008 .

[19]  Stacy Williams,et al.  Limit order books , 2010, 1012.0349.

[20]  Yee Whye Teh,et al.  Particle Value Functions , 2017, ICLR.

[21]  Iuliia Manziuk,et al.  Deep Reinforcement Learning for Market Making in Corporate Bonds: Beating the Curse of Dimensionality , 2019, Applied Mathematical Finance.

[22]  Tuomas Sandholm,et al.  Automated Market Making: Theory and Practice , 2012 .

[23]  Sebastian Scherer,et al.  Improving Stochastic Policy Gradients in Continuous Control with Deep Reinforcement Learning using the Beta Distribution , 2017, ICML.

[24]  Pietro Fodra,et al.  High-frequency market-making with inventory constraints and directional bets , 2012, 1206.4810.

[25]  Christian R. Shelton,et al.  An Electronic Market-maker , 2001 .

[26]  Sebastian Jaimungal,et al.  Order-Flow and Liquidity Provision , 2015 .

[27]  Sanford J. Grossman,et al.  Liquidity and Market Structure , 1988 .

[28]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[29]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[30]  Michael Kearns,et al.  Reinforcement learning for optimized trade execution , 2006, ICML.

[31]  Mahesan Niranjan,et al.  On-line Q-learning using connectionist systems , 1994 .

[32]  Abhinav Gupta,et al.  Robust Adversarial Reinforcement Learning , 2017, ICML.

[33]  Rahul Savani,et al.  Robust Market Making via Adversarial Reinforcement Learning , 2020, IJCAI.

[34]  Paul R. Milgrom,et al.  Bid, ask and transaction prices in a specialist market with heterogeneously informed traders , 1985 .