Intelligent Routing based on Deep Reinforcement Learning in Software-Defined Data-Center Networks

In software-defined data-center networks, Elephant flow/Mice flow/Coflow coexist and multiple resources (bandwidth, cache and computing) coexist. However, the conventional routing methods cannot overcome the large gap between different performance requirements of flow and efficient resource allocation. Therefore, this paper proposes DRL-R (Deep Reinforcement Learning-based Routing) to bridge this gap. First, we recombine multiple resources (node’s cache, link’s bandwidth) by quantifying the contribution score of them reducing the delay. This actually converts the performance requirements of flow into resource requirements of this flow, hence, the routing problem can be converted into a job-scheduling problem in resource management. Second, a DRL agent deployed on an SDN controller continually interacts with the network for adaptively performing reasonable routing according to the network state, and optimally allocating network resources for traffic. We employ Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) to build the DRL-R. Finally, we demonstrate the effectiveness of DRL-R through extensive simulations. Benefitted from continually learning with a global view, DRL-R can improve throughput highest up to 40% and flow completion time highest up to 47% over OSPF. DRL-R can improve the load balance of link highest up to 18.8% over OSPF. Additionally, DDPG has better performance than DQN.

[1]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[2]  Raouf Boutaba,et al.  A comprehensive survey on machine learning for networking: evolution, applications and research opportunities , 2018, Journal of Internet Services and Applications.

[3]  Dilip Kumar Saikia,et al.  OpenFlow-Based Scalable Routing With Hybrid Addressing in Data Center Networks , 2017, IEEE Communications Letters.

[4]  Chi Harold Liu,et al.  Experience-driven Networking: A Deep Reinforcement Learning based Approach , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[5]  Matthew Roughan,et al.  Simplifying the synthesis of internet traffic matrices , 2005, CCRV.

[6]  Wai-Xi Liu,et al.  Content Popularity Prediction and Caching for ICN: A Deep Learning Approach With SDN , 2018, IEEE Access.

[7]  Fernando M. V. Ramos,et al.  Software-Defined Networking: A Comprehensive Survey , 2014, Proceedings of the IEEE.

[8]  Jian Song,et al.  SDN-Based Big Data Caching in ISP Networks , 2018, IEEE Transactions on Big Data.

[9]  Srikanth Kandula,et al.  Resource Management with Deep Reinforcement Learning , 2016, HotNets.

[10]  Jun Cai,et al.  Information-centric networking with built-in network coding to achieve multisource transmission at network-layer , 2017, Comput. Networks.

[11]  Kotaro Kataoka,et al.  AMPS: Application aware multipath flow routing using machine learning in SDN , 2017, 2017 Twenty-third National Conference on Communications (NCC).

[12]  Nei Kato,et al.  Routing or Computing? The Paradigm Shift Towards Intelligent Computer Network Packet Transmission Based on Deep Learning , 2017, IEEE Transactions on Computers.

[13]  V. Latora,et al.  Complex networks: Structure and dynamics , 2006 .

[14]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[15]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[16]  Nei Kato,et al.  State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems , 2017, IEEE Communications Surveys & Tutorials.

[17]  Claudio Castellano,et al.  Defining and identifying communities in networks. , 2003, Proceedings of the National Academy of Sciences of the United States of America.