EdgeDQN: Multiple SFC Placement in Edge Computing Environment

Network Function Virtualization (NFV) and service orchestration has simplified the Service Function Chain management (SFC) tasks, while the edge cloud infrastructure has reduced the latency. Due to these existing technological advantages, there is an urgent need for a dynamic and flexible service chain placement model that performs resource allocation of substrate network in a delay-sensitive and resource-efficient manner. We propose an off-policy Deep Reinforcement Learning algorithm EdgeDQN for efficient SFC placement in the edge cloud environment. The problem of edge resource scarcity is handled by designing a network model that allows worst-case resource renting from neighbors and data centers. This network model is integrated with EdgeDQN using several constraints. This paper aims to find the optimal placement by minimizing the underlying resource utilization and SFC end-to-end delay for multiple SFCs at the same time. To achieve that, an intuitive reward model is proposed. We compare the proposed EdgeDQN algorithm with DQN, Q-learning, and EdgeQL algorithms in terms of performance parameters such as cumulative reward, cumulative standard deviation, latency, and learning convergence time for 420 different test cases. Extensive test results on a simulated and physical (OpenStack) testbed demonstrate the effectiveness of the proposed EdgeDQN algorithm.

[1]  James W. Hong,et al.  Environment Aware Adaptive Q-Learning to Deploy SFC on Edge Computing , 2020, 2020 16th International Conference on Network and Service Management (CNSM).

[2]  Ahmad Khonsari,et al.  DeepViNE: Virtual Network Embedding with Deep Reinforcement Learning , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[3]  Hee-Gon Kim,et al.  A Network Intelligence Architecture for Efficient VNF Lifecycle Management , 2021, IEEE Transactions on Network and Service Management.

[4]  Luciana S. Buriol,et al.  Piecing together the NFV provisioning puzzle: Efficient placement and chaining of virtual network functions , 2015, 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM).

[5]  Peilin Hong,et al.  Efficiently Embedding Service Function Chains with Dynamic Virtual Network Function Placement in Geo-Distributed Cloud System , 2019, IEEE Transactions on Parallel and Distributed Systems.

[6]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[7]  Jae-Hyoung Yoo,et al.  Q-learning based Service Function Chaining using VNF Resource-aware Reward Model , 2020, 2020 21st Asia-Pacific Network Operations and Management Symposium (APNOMS).

[8]  Feng Liu,et al.  AuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization , 2018, SIGCOMM.

[9]  Tarik Taleb,et al.  Network Slicing and Softwarization: A Survey on Principles, Enabling Technologies, and Solutions , 2018, IEEE Communications Surveys & Tutorials.

[10]  Mathis Obadia,et al.  A graph approach to placement of Service Functions Chains , 2017, 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM).

[11]  Xavier Hesselbach,et al.  Virtual Network Embedding: A Survey , 2013, IEEE Communications Surveys & Tutorials.

[12]  Jae-Hyoung Yoo,et al.  Deep Q-Networks based Auto-scaling for Service Function Chaining , 2020, 2020 16th International Conference on Network and Service Management (CNSM).

[13]  Jae-Hyoung Yoo,et al.  Q-Learning based SFC deployment on Edge Computing Environment , 2020, 2020 21st Asia-Pacific Network Operations and Management Symposium (APNOMS).

[14]  Abdelkader Outtagarts,et al.  A Deep Reinforcement Learning Approach for VNF Forwarding Graph Embedding , 2019, IEEE Transactions on Network and Service Management.

[15]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[16]  Shangguang Wang,et al.  MVR: An Architecture for Computation Offloading in Mobile Edge Computing , 2017, 2017 IEEE International Conference on Edge Computing (EDGE).

[17]  Wenxin Qiao,et al.  SFC Embedding Meets Machine Learning: Deep Reinforcement Learning Approaches , 2021, IEEE Communications Letters.