Benchmarking Q-Learning Methods for Intelligent Network Orchestration in the Edge

We benchmark Q-learning methods, with various action selection strategies, in intelligent orchestration of the network edge. Q-learning is a reinforcement learning technique that aims to find optimal action policies by taking advantage of the experiences in the past without utilizing a model that describes the dynamics of the environment. With experiences, we refer to the observed causality between the action and the corresponding impact to the environment. In this paper, the environment for Q-learning is composed of virtualized networking resources along with their dynamics that are monitored with Spindump, an in-network latency measurement tool with support for QUIC and TCP. We optimize the orchestration of these networking resources by introducing Q-learning as part of the machine learning driven, intelligent orchestration that is applicable in the edge. Based on the benchmarking results, we identify which action selection strategies support network orchestration that provides low latency and packet loss by considering network resource allocation in the edge.

[1]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[2]  Pieter Abbeel,et al.  Benchmarking Deep Reinforcement Learning for Continuous Control , 2016, ICML.

[3]  Leonard Barolli,et al.  Design and Implementation of a Simulation System Based on Deep Q-Network for Mobile Actor Node Control in Wireless Sensor and Actor Networks , 2017, 2017 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA).

[4]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[5]  Ryszard Kowalczyk,et al.  Dynamic analysis of multiagent Q-learning with ε-greedy exploration , 2009, ICML '09.

[6]  James Bergstra,et al.  Benchmarking Reinforcement Learning Algorithms on Real-World Robots , 2018, CoRL.

[7]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[8]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[10]  Jose Ordonez-Lucena,et al.  Automated Network Service Scaling in NFV: Concepts, Mechanisms and Scaling Workflow , 2018, IEEE Communications Magazine.

[11]  M. Kubát.  An Introduction to Machine Learning , 2017, Springer International Publishing.

[12]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[13]  Song Guo,et al.  Resource Management at the Network Edge: A Deep Reinforcement Learning Approach , 2019, IEEE Network.

[14]  Michael I. Jordan,et al.  Is Q-learning Provably Efficient? , 2018, NeurIPS.

[15]  Aram Galstyan,et al.  Dynamics of Boltzmann Q learning in two-player two-action games. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  Pieter Abbeel,et al.  Benchmarking Model-Based Reinforcement Learning , 2019, ArXiv.

[17]  Jason P. Jue,et al.  All One Needs to Know about Fog Computing and Related Edge Computing Paradigms , 2019 .

[18]  Tarik Taleb,et al.  On Multi-Access Edge Computing: A Survey of the Emerging 5G Network Edge Cloud Architecture and Orchestration , 2017, IEEE Communications Surveys & Tutorials.

[19]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.