ModelicaGym: applying reinforcement learning to Modelica models

This paper presents ModelicaGym toolbox that was developed to employ Reinforcement Learning (RL) for solving optimization and control tasks in Modelica models. The developed tool allows connecting models using Functional Mock-up Interface (FMI) to OpenAI Gym toolkit in order to exploit Modelica equation-based modeling and co-simulation together with RL algorithms as a functionality of the tools correspondingly. Thus, ModelicaGym facilitates fast and convenient development of RL algorithms and their comparison when solving optimal control problem for Modelica dynamic models. Inheritance structure of ModelicaGym toolbox's classes and the implemented methods are discussed in details. The toolbox functionality validation is performed on Cart-Pole balancing problem. This includes physical system model description and its integration using the toolbox, experiments on selection and influence of the model parameters (i.e. force magnitude, Cart-pole mass ratio, reward ratio, and simulation time step) on the learning process of Q-learning algorithm supported with the discussion of the simulation results.