QRED: A Q-Learning-based Active Queue Management Scheme

The Active Queue Management (AQM) algorithm is one of most important research fields in network congestion control. To adjust the maximum dropping probability (maxp) according to the network situation the maxp calculation based on the RED algorithm is improved using the Q-learning algorithm, and a new algorithm, known as QRED (Q-learning RED), is proposed. The self-adaptive adjustment for the maxp is achieved using the QRED algorithm and the queue length stability in a dynamic network environment is realized. In addition, the QRED algorithm not only avoids the sensitivity of the RED algorithm parameters, but also adapts the packet loss rate according to the specific network service type. Results based on the NS2 simulation show that the QRED algorithm has better stability in complex network environments, and hence, are superior to the RED active queue management algorithm.

[1]  Nen-Fu Huang,et al.  Application traffic classification at the early stage by characterizing application rounds , 2013, Inf. Sci..

[2]  Kang G. Shin,et al.  A self-configuring RED gateway , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[3]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[4]  Zhang Yan.  A Robust Active Queue Management Algorithm Based on Reinforcement Learning , 2004 .

[5]  Xiande Liu,et al.  Enhanced random early detection algorithm BACnet router for congestion avoidance , 2005, The 7th International Conference on Advanced Communication Technology, 2005, ICACT 2005..

[6]  S. S. Masoumzadeh,et al.  Deep Blue: A Fuzzy Q-Learning Enhanced Active Queue Management Scheme , 2009, 2009 International Conference on Adaptive and Intelligent Systems.

[7]  Bin Fan,et al.  Network slice access selection scheme in 5G , 2017, 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC).

[8]  Kao-Shing Hwang,et al.  Cooperative multiagent congestion control for high-speed networks , 2005, IEEE Trans. Syst. Man Cybern. Part B.

[9]  Han-Chieh Chao,et al.  A multilevel information fusion approach for road congestion detection in VANETs , 2013, Math. Comput. Model..

[10]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[11]  Victor C. M. Leung,et al.  Network Slicing Based 5G and Future Mobile Networks: Mobility, Resource Management, and Challenges , 2017, IEEE Communications Magazine.

[12]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[13]  Christophe Diot,et al.  Reasons not to deploy RED , 1999, 1999 Seventh International Workshop on Quality of Service. IWQoS'99. (Cat. No.98EX354).

[14]  Navid Nikaein,et al.  Network Slices toward 5G Communications: Slicing the LTE Network , 2017, IEEE Communications Magazine.

[15]  Wei Wu,et al.  Stability analysis on active queue management algorithms in routers , 2001, MASCOTS 2001, Proceedings Ninth International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[16]  Sally Floyd,et al.  Promoting the use of end-to-end congestion control in the Internet , 1999, TNET.

[17]  Yutaka Takahashi,et al.  A queue management algorithm for fair bandwidth allocation , 2007, Comput. Commun..

[18]  Thanasis Korakis,et al.  Network Store: Exploring Slicing in Future 5G Networks , 2015, MobiArch.

[19]  Ibrahim Habib,et al.  Reinforcement learning-based neural network congestion controller for ATM networks , 1995, Proceedings of MILCOM '95.

[20]  Kang G. Shin,et al.  The BLUE active queue management algorithms , 2002, TNET.

[21]  Kao-Shing Hwang,et al.  Reinforcement learning congestion controller for multimedia surveillance system , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[22]  Masayuki Murata,et al.  Comparisons of packet scheduling algorithms for fair service among connections on the Internet , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[23]  Liang Zhou,et al.  Multimedia traffic security architecture for the internet of things , 2011, IEEE Network.

[24]  V. Jacobson,et al.  Congestion avoidance and control , 1988, CCRV.

[25]  QUTdN QeO,et al.  Random early detection gateways for congestion avoidance , 1993, TNET.

[26]  Yao-Chung Chang.  Heterogeneous Wireless Sensor Network with EPC Network Architecture for U-Life Environment , 2014 .

[27]  Eitan Altman,et al.  NS Simulator for Beginners , 2012, NS Simulator for Beginners.

[28]  Kang Jun.  Study of congestion control algorithm based on neural network supervised control , 2010 .