Evolutionary Actor-Multi-Critic Model for VNF-FG Embedding

The placement of Virtual Network Function - Forwarding Graphs (VNF-FGs) is one of the basic operations in the networks of the future. Being NP-hard, several heuristics and metaheuristics have been proposed. However, these approaches are inefficient due to the need to recalculate the solution at each service placement. In this paper, we adapt one of the most advanced approaches in Deep Reinforcement Learning (DRL), in order to improve exploration by generalizing the neural network calculating action values. We also propose an evolutionary algorithm to evolve these neural networks in order to discover better ones, which also avoids getting stuck in local minima. In order to avoid going through the almost innumerable number of infeasible solutions, we propose a heuristic, which combined with our DRL, makes it possible to guarantee the feasibility of the solutions and therefore to make the placement much more efficient. The simulation results we obtained confirm the quality of the solutions obtained as well as the superiority of the proposed solution over the existing one.

[1]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[2]  Djamal Zeghlache,et al.  Virtualized network functions chaining and routing algorithms , 2017, Comput. Networks.

[3]  Bin Liu,et al.  On data plane latency and pseudo-TCP congestion in Software-Defined Networking , 2016, 2016 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS).

[4]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[5]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[6]  Sangheon Pack,et al.  Joint Optimization of Service Function Placement and Flow Distribution for Service Function Chaining , 2017, IEEE Journal on Selected Areas in Communications.

[7]  Chi Harold Liu,et al.  Experience-driven Networking: A Deep Reinforcement Learning based Approach , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[8]  Matthew Roughan,et al.  The Internet Topology Zoo , 2011, IEEE Journal on Selected Areas in Communications.

[9]  B. Bollobás The evolution of random graphs , 1984 .

[10]  P. Erdos,et al.  On the evolution of random graphs , 1984 .

[11]  Peter Sunehag,et al.  Reinforcement Learning in Large Discrete Action Spaces , 2015, ArXiv.

[12]  Seungjoon Lee,et al.  Network function virtualization: Challenges and opportunities for innovations , 2015, IEEE Communications Magazine.

[13]  A. Varga,et al.  THE OMNET++ DISCRETE EVENT SIMULATION SYSTEM , 2003 .

[14]  Djamal Zeghlache,et al.  A multi-objective non-dominated sorting genetic algorithm for VNF chains placement , 2018, 2018 15th IEEE Annual Consumer Communications & Networking Conference (CCNC).

[15]  Gauthier Picard,et al.  Single and Multi-Domain Adaptive Allocation Algorithms for VNF Forwarding Graph Embedding , 2019, IEEE Transactions on Network and Service Management.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Richard Evans,et al.  Deep Reinforcement Learning in Large Discrete Action Spaces , 2015, 1512.07679.

[18]  Lazaros Gkatzikis,et al.  The Algorithmic Aspects of Network Slicing , 2017, IEEE Communications Magazine.

[19]  Tarik Taleb,et al.  Network Slicing and Softwarization: A Survey on Principles, Enabling Technologies, and Solutions , 2018, IEEE Communications Surveys & Tutorials.

[20]  Albert Cabellos-Aparicio,et al.  A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization , 2017, ArXiv.

[21]  Zhixiang Liu,et al.  Service Function Chaining Resource Allocation: A Survey , 2016, ArXiv.

[22]  Roberto Riggio,et al.  Scheduling Wireless Virtual Networks Functions , 2016, IEEE Transactions on Network and Service Management.

[23]  Ghasem Mirjalily,et al.  Optimal Network Function Virtualization and Service Function Chaining: A Survey , 2018, Chinese Journal of Electronics.

[24]  Djamal Zeghlache,et al.  Scalable and cost-efficient algorithms for VNF chaining and placement problem , 2017, 2017 20th Conference on Innovations in Clouds, Internet and Networks (ICIN).

[25]  Filip De Turck,et al.  Design and evaluation of learning algorithms for dynamic resource management in virtual networks , 2014, 2014 IEEE Network Operations and Management Symposium (NOMS).

[26]  Abdelkader Outtagarts,et al.  A Deep Reinforcement Learning Approach for VNF Forwarding Graph Embedding , 2019, IEEE Transactions on Network and Service Management.