Adaptive Online Decision Method for Initial Congestion Window in 5G Mobile Edge Computing Using Deep Reinforcement Learning

Mobile edge computing provides users with low response time and avoids unnecessary data transmission. Due to the deployment of 5G, the emerging edge systems can provide gigabit bandwidth. However, network protocols have not evolved together. In TCP, the initial congestion window (IW) is such a low value that most short flows still stay in slow start phase when finishing, and do not fully utilize available bandwidth. Naively increasing IW may result in congestion, which causes long latency. Moreover, since the network environment is dynamic, we have a challenging problem—how to adaptively adjust IW such that flow completion time is optimized, while congestion is minimized. In this paper, we propose an adaptive online decision method to solve the problem, which learns the best policy using deep reinforcement learning stably and fast. In addition, we propose an approach to further improve the performance by supervised learning, using data collected during online learning. We also propose to adopt SDN to address the challenges in implementing our method in MEC systems. To evaluate our method, we build an MEC simulator based on ns3. Our simulations demonstrate that our method performs better than existing methods. It can effectively reduce FCT with little congestion caused.

[1]  Dan Pei,et al.  Reducing Web Latency Through Dynamically Setting TCP Initial Window with Reinforcement Learning , 2018, 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS).

[2]  Sundeep Rangan,et al.  End-to-End Simulation of 5G mmWave Networks , 2017, IEEE Communications Surveys & Tutorials.

[3]  Mo Dong,et al.  PCC: Re-architecting Congestion Control for Consistent High Performance , 2014, NSDI.

[4]  Hari Balakrishnan,et al.  TCP ex machina: computer-generated congestion control , 2013, SIGCOMM.

[5]  Hongzi Mao,et al.  Neural Adaptive Video Streaming with Pensieve , 2017, SIGCOMM.

[6]  Christian Bonnet,et al.  Low latency MEC framework for SDN-based LTE/LTE-A networks , 2017, 2017 IEEE International Conference on Communications (ICC).

[7]  Monia Ghobadi,et al.  Rethinking end-to-end congestion control in software-defined networks , 2012, HotNets-XI.

[8]  Waleed Meleis,et al.  QTCP: Adaptive Congestion Control with Reinforcement Learning , 2019, IEEE Transactions on Network Science and Engineering.

[9]  Dan Pei,et al.  Dynamic TCP Initial Windows and Congestion Control Schemes Through Reinforcement Learning , 2019, IEEE Journal on Selected Areas in Communications.

[10]  Claudia Linnhoff-Popien,et al.  Mobile Edge Computing , 2016, Informatik-Spektrum.

[11]  Brighten Godfrey,et al.  A Deep Reinforcement Learning Perspective on Internet Congestion Control , 2019, ICML.

[12]  Marco Pavone,et al.  Cellular Network Traffic Scheduling With Deep Reinforcement Learning , 2018, AAAI.

[13]  Hans C. Woithe,et al.  Edge computing in the ePC: a reality check , 2017, SEC.

[14]  Yuan Zhang,et al.  A survey on software defined networking with multiple controllers , 2018, J. Netw. Comput. Appl..

[15]  Atay Ozgovde,et al.  How Can Edge Computing Benefit From Software-Defined Networking: A Survey, Use Cases, and Future Directions , 2017, IEEE Communications Surveys & Tutorials.

[16]  Zhiyuan Xu,et al.  Experience-Driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning , 2019, IEEE Journal on Selected Areas in Communications.

[17]  Mo Dong,et al.  PCC Vivace: Online-Learning Congestion Control , 2018, NSDI.

[18]  Xiqi Gao,et al.  Cellular architecture and key technologies for 5G wireless communication networks , 2014, IEEE Communications Magazine.

[19]  B. Liang,et al.  Mobile Edge Computing , 2020, Encyclopedia of Wireless Networks.

[20]  Xiaohua Jia,et al.  Energy Efficiency Enhancement for CNN-based Deep Mobile Sensing , 2019, IEEE Wireless Communications.

[21]  Jitendra K. Tugnait,et al.  TCP-Drinc: Smart Congestion Control Based on Deep Reinforcement Learning , 2019, IEEE Access.

[22]  Yan Li,et al.  CAPES: Unsupervised Storage Performance Tuning Using Neural Network-Based Deep Reinforcement Learning , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[23]  Feng Qian,et al.  An in-depth study of LTE: effect of network protocol and application behavior on performance , 2013, SIGCOMM.

[24]  Nick McKeown,et al.  Why flow-completion time is the right metric for congestion control , 2006, CCRV.

[25]  Dario Pompili,et al.  Collaborative Mobile Edge Computing in 5G Networks: New Paradigms, Scenarios, and Challenges , 2016, IEEE Communications Magazine.

[26]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[27]  Samy Bengio,et al.  Device Placement Optimization with Reinforcement Learning , 2017, ICML.

[28]  Amit Agarwal,et al.  An argument for increasing TCP's initial congestion window , 2010, CCRV.