Is Machine Learning Ready for Traffic Engineering Optimization?

Traffic Engineering (TE) is a basic building block of the Internet. In this paper, we analyze whether modern Machine Learning (ML) methods are ready to be used for TE optimization. We address this open question through a comparative analysis between the state of the art in ML and the state of the art in TE. To this end, we first present a novel distributed system for TE that leverages the latest advancements in ML. Our system implements a novel architecture that combines Multi-Agent Reinforcement Learning (MARL) and Graph Neural Networks (GNN) to minimize network congestion. In our evaluation, we compare our MARL+GNN system with DEFO, a network optimizer based on Constraint Programming that represents the state of the art in TE. Our experimental results show that the proposed MARL+GNN solution achieves equivalent performance to DEFO in a wide variety of network scenarios including three real-world network topologies. At the same time, we show that MARL+GNN can achieve significant reductions in execution time (from the scale of minutes with DEFO to a few seconds with our solution).

[1]  John Moy,et al.  OSPF Version 2 , 1998, RFC.

[2]  Toby Walsh,et al.  Handbook of Constraint Programming , 2006, Handbook of Constraint Programming.

[3]  Albert Cabellos-Aparicio,et al.  Unveiling the potential of Graph Neural Networks for network modeling and optimization in SDN , 2019, SOSR.

[4]  T. V. Lakshman,et al.  Optimized network traffic engineering using segment routing , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[5]  Georg Carle,et al.  Learning and Generating Distributed Routing Protocols Using Graph-Based Deep Learning , 2018, Big-DAMA@SIGCOMM.

[6]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[7]  L. Freeman,et al.  Centrality in valued graphs: A measure of betweenness based on network flow , 1991 .

[8]  Ming Zhang,et al.  MicroTE: fine grained traffic engineering for data centers , 2011, CoNEXT '11.

[9]  Viktor Prasanna,et al.  GraphACT: Accelerating GCN Training on CPU-FPGA Heterogeneous Platforms , 2019, FPGA.

[10]  Ning Wang,et al.  An overview of routing optimization for internet traffic engineering , 2008, IEEE Communications Surveys & Tutorials.

[11]  Pierre Schaus,et al.  REPETITA: Repeatable Experiments for Performance Evaluation of Traffic-Engineering Algorithms , 2017, ArXiv.

[12]  Xin Wang,et al.  Machine Learning for Networking: Workflow, Advances and Opportunities , 2017, IEEE Network.

[13]  Matthieu Geist,et al.  What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study , 2021, ICLR.

[14]  Marivi Higuero,et al.  A Survey on the Contributions of Software-Defined Networking to Traffic Engineering , 2017, IEEE Communications Surveys & Tutorials.

[15]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[16]  Nikos A. Vlassis,et al.  Optimal and Approximate Q-value Functions for Decentralized POMDPs , 2008, J. Artif. Intell. Res..

[17]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Mikkel Thorup,et al.  Increasing Internet Capacity Using Local Search , 2004, Comput. Optim. Appl..

[19]  Dafna Shahaf,et al.  Learning to Route , 2017, HotNets.

[20]  Mikkel Thorup,et al.  Internet traffic engineering by optimizing OSPF weights , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[21]  Jakob N. Foerster,et al.  Deep multi-agent reinforcement learning , 2018 .

[22]  Albert Cabellos-Aparicio,et al.  Challenging the generalization capabilities of Graph Neural Networks for network modeling , 2019, SIGCOMM Posters and Demos.

[23]  Mung Chiang,et al.  Link-State Routing with Hop-by-Hop Forwarding Can Achieve Optimal Traffic Engineering , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[24]  Albert Cabellos-Aparicio,et al.  A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization , 2017, ArXiv.

[25]  Olivier Bonaventure,et al.  A Declarative and Expressive Approach to Control Forwarding Paths in Carrier-Grade Networks , 2015, SIGCOMM.

[26]  Stefano Vissicchio,et al.  Expect the unexpected: Sub-second optimization for segment routing , 2017, IEEE INFOCOM 2017 - IEEE Conference on Computer Communications.

[27]  Kun Cao,et al.  A Survey of Deployment Solutions and Optimization Strategies for Hybrid SDN Networks , 2019, IEEE Communications Surveys & Tutorials.

[28]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[29]  Hongyan Li,et al.  Packet Routing Against Network Congestion: A Deep Multi-agent Reinforcement Learning Approach , 2020, 2020 International Conference on Computing, Networking and Communications (ICNC).

[30]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[31]  Albert Cabellos-Aparicio,et al.  Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case. , 2020 .

[32]  Tiejun Huang,et al.  Graph Convolutional Reinforcement Learning , 2020, ICLR.

[33]  Daniel O. Awduche,et al.  Requirements for Traffic Engineering Over MPLS , 1999, RFC.

[34]  Stephen C. Adams,et al.  Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication , 2020, ArXiv.

[35]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[36]  Chi Harold Liu,et al.  Experience-driven Networking: A Deep Reinforcement Learning based Approach , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[37]  A. Basu,et al.  Stability issues in OSPF routing , 2001, SIGCOMM '01.

[38]  Olivier Bonaventure,et al.  CG4SR: Near Optimal Traffic Engineering for Segment Routing with Column Generation , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[39]  Roch Guérin,et al.  Achieving near-optimal traffic engineering solutions for current OSPF/IS-IS networks , 2005, IEEE/ACM Transactions on Networking.

[40]  Mingwei Xu,et al.  A Multi-agent Reinforcement Learning Perspective on Distributed Traffic Engineering , 2020, 2020 IEEE 28th International Conference on Network Protocols (ICNP).

[41]  Clarence Filsfils,et al.  The Segment Routing Architecture , 2014, 2015 IEEE Global Communications Conference (GLOBECOM).

[42]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[43]  Edith Cohen,et al.  Optimal oblivious routing in polynomial time , 2003, STOC '03.

[44]  Matthew Roughan,et al.  Simplifying the synthesis of internet traffic matrices , 2005, CCRV.

[45]  Demis Hassabis,et al.  Mastering Atari, Go, chess and shogi by planning with a learned model , 2019, Nature.

[46]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[47]  Angela L. Chiu,et al.  Overview and Principles of Internet Traffic Engineering , 2002, RFC.