ABIDES-gym: gym environments for multi-agent discrete event simulation and application to financial markets

Model-free Reinforcement Learning (RL) requires the ability to sample trajectories by taking actions in the original problem environment or a simulated version of it. Breakthroughs in the field of RL have been largely facilitated by the development of dedicated open source simulators with easy to use frameworks such as OpenAI Gym and its Atari environments. In this paper we propose to use the OpenAI Gym framework on discrete event time based Discrete Event Multi-Agent Simulation (DEMAS). We introduce a general technique to wrap a DEMAS simulator into the Gym framework. We expose the technique in detail and implement it using the simulator ABIDES as a base. We apply this work by specifically using the markets extension of ABIDES, ABIDES-Markets, and develop two benchmark financial markets OpenAI Gym environments for training daily investor and execution agents.1 As a result, these two environments describe classic financial problems with a complex interactive market behavior response to the experimental agent’s action. ACM Reference Format: Selim Amrouni, Aymeric Moulin, Jared Vann, Svitlana Vyetrenko, Tucker Balch, and Manuela Veloso. 2021. ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation andApplication to FinancialMarkets. In Proceedings of ICAIF’21. ACM, New York, NY, USA, 9 pages. https://doi. org/10.1145/1122445.1122456 ∗Both authors contributed equally to this research. 1ABIDES source code is open-sourced on https://github.com/jpmorganchase/abidesjpmc-public and available upon request. Please reach out to Selim Amrouni and Aymeric Moulin. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ICAIF’21, November 03–05, 2021, London, UK © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00 https://doi.org/10.1145/1122445.1122456

[1]  Michael Kearns,et al.  Reinforcement learning for optimized trade execution , 2006, ICML.

[2]  Ion Stoica,et al.  Tune: A Research Platform for Distributed Model Selection and Training , 2018, ArXiv.

[3]  Gabriel Dulac-Arnold,et al.  Challenges of Real-World Reinforcement Learning , 2019, ArXiv.

[4]  Nelson Minar,et al.  The Swarm Simulation System: A Toolkit for Building Multi-Agent Simulations , 1996 .

[5]  Maria Hybinette,et al.  ABIDES: Towards High-Fidelity Market Simulation for AI Research , 2019, ArXiv.

[6]  Sriram Srinivasan,et al.  OpenSpiel: A Framework for Reinforcement Learning in Games , 2019, ArXiv.

[7]  Maria Hybinette,et al.  How to Evaluate Trading Strategies: Single Agent Market Replay or Multiple Agent Interactive Simulation? , 2019, ArXiv.

[8]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[9]  Daniele Gianni,et al.  Bringing Discrete Event Simulation Concepts into Multi-agent Systems , 2008, Tenth International Conference on Computer Modeling and Simulation (uksim 2008).

[10]  M. Marchesi,et al.  Scaling and criticality in a stochastic multi-agent model of a financial market , 1999, Nature.

[11]  Sergey Levine,et al.  Guided Policy Search , 2013, ICML.

[12]  Michael I. Jordan,et al.  RLlib: Abstractions for Distributed Reinforcement Learning , 2017, ICML.

[13]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[14]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[15]  Harold William Kuhn,et al.  Contributions to the Theory of Games (AM-39), Volume III , 1951 .

[16]  Manuela Veloso,et al.  Get real: realism metrics for robust limit order book market simulations , 2019, ICAIF.

[17]  Michael I. Jordan,et al.  Ray: A Distributed Framework for Emerging AI Applications , 2017, OSDI.

[18]  Herb Schwetman,et al.  Discrete‐Event Simulation of Computer and Communication Systems , 2007 .

[19]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.