Actor-Critic Learning Based QoS-Aware Scheduler for Reconfigurable Wireless Networks

The flexibility offered by reconfigurable wireless networks, provide new opportunities for various applications such as online AR/VR gaming, high-quality video streaming and autonomous vehicles, that desire high-bandwidth, reliable and low-latency communications. These applications come with very stringent Quality of Service (QoS) requirements and increase the burden over mobile networks. Currently, there is a huge spectrum scarcity due to the massive data explosion and this problem can be solved by helps of Reconfigurable Wireless Networks (RWNs) where nodes have reconfiguration and perception capabilities. Therefore, a necessity of AI-assisted algorithms for resource block allocation is observed. To tackle this challenge, in this paper, we propose an actor-critic learning-based scheduler for allocating resource blocks in a RWN. Various traffic types with different QoS levels are assigned to our agents to provide more realistic results. We also include mobility in our simulations to increase the dynamicity of networks. The proposed model is compared with another actor-critic model and with other traditional schedulers; proportional fair (PF) and Channel and QoS Aware (CQA) techniques. The proposed models are evaluated by considering the delay experienced by user equipment (UEs), successful transmissions and head-of-the-line delays. The results show that the proposed model noticeably outperforms other techniques in different aspects.

[1]  Marco Pavone,et al.  Cellular Network Traffic Scheduling With Deep Reinforcement Learning , 2018, AAAI.

[2]  Sijing Zhang,et al.  A Comparison of Reinforcement Learning Algorithms in Fairness-Oriented OFDMA Schedulers , 2019, Inf..

[3]  Dipak Ghosal,et al.  A Deep Deterministic Policy Gradient Based Network Scheduler For Deadline-Driven Data Transfers , 2020, 2020 IFIP Networking Conference (Networking).

[4]  Dale Schuurmans,et al.  Bridging the Gap Between Value and Policy Based Reinforcement Learning , 2017, NIPS.

[5]  Sandra Sendra,et al.  A Survey on 5G Usage Scenarios and Traffic Models , 2020, IEEE Communications Surveys & Tutorials.

[6]  Igor G. Olaizola,et al.  Network Resource Allocation System for QoE-Aware Delivery of Media Services in 5G Networks , 2018, IEEE Transactions on Broadcasting.

[7]  Melike Erol-Kantarci,et al.  AI-Enabled Radio Resource Allocation in 5G for URLLC and eMBB Users , 2019, 2019 IEEE 2nd 5G World Forum (5GWF).

[8]  Jonas Medbo,et al.  Numerology and frame structure for 5G radio access , 2016, 2016 IEEE 27th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC).

[9]  Chelsea C. White,et al.  A survey of solution techniques for the partially observed Markov decision process , 1991, Ann. Oper. Res..

[10]  Saied Abedi.  Efficient radio resource management for wireless multimedia communications: a multidimensional QoS-based packet scheduler , 2005, IEEE Transactions on Wireless Communications.

[11]  Xu Du,et al.  Balancing Queueing and Retransmission: Latency-Optimal Massive MIMO Design , 2020, IEEE Transactions on Wireless Communications.

[12]  Mate Boban,et al.  Reinforcement Learning Scheduler for Vehicle-to-Vehicle Communications Outside Coverage , 2018, 2018 IEEE Vehicular Networking Conference (VNC).

[13]  Anatolij Zubow,et al.  ns-3 meets OpenAI Gym: The Playground for Machine Learning in Networking Research , 2019, MSWiM.

[14]  Giuseppe Piro,et al.  Two-Level Downlink Scheduling for Real-Time Multimedia Services in LTE Networks , 2011, IEEE Transactions on Multimedia.

[15]  Zhu Han,et al.  User Scheduling and Resource Allocation in HetNets With Hybrid Energy Supply: An Actor-Critic Reinforcement Learning Approach , 2018, IEEE Transactions on Wireless Communications.

[16]  Ke Zhang,et al.  Machine Learning at the Edge: A Data-Driven Architecture With Applications to 5G Cellular Networks , 2018, IEEE Transactions on Mobile Computing.

[17]  Jeroen Wigard,et al.  Dynamic Packet Scheduling for Traffic Mixes of Best Effort and VoIP Users in E-UTRAN Downlink , 2010, 2010 IEEE 71st Vehicular Technology Conference.

[18]  Christos Verikoukis,et al.  Offline SLA-Constrained Deep Learning for 5G Networks Reliable and Dynamic End-to-End Slicing , 2020, IEEE Journal on Selected Areas in Communications.

[19]  Nicola Baldo,et al.  A new channel and QoS aware scheduler to enhance the capacity of voice over LTE systems , 2014, 2014 IEEE 11th International Multi-Conference on Systems, Signals & Devices (SSD14).

[20]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[21]  Mohit Sewak,et al.  Actor-Critic Models and the A3C , 2019, Deep Reinforcement Learning.

[22]  Ala I. Al-Fuqaha,et al.  Enabling Cognitive Smart Cities Using Big Data and Machine Learning: Approaches and Challenges , 2018, IEEE Communications Magazine.

[23]  Ying-Chang Liang,et al.  Applications of Deep Reinforcement Learning in Communications and Networking: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[24]  Sijing Zhang,et al.  Towards 5G: A Reinforcement Learning-Based Scheduling Solution for Data Traffic Management , 2018, IEEE Transactions on Network and Service Management.

[25]  Fatimah Audah Md. Zaki,et al.  Towards Efficient and Scalable Machine Learning-Based QoS Traffic Classification in Software-Defined Network , 2019, MobiWIS.

[26]  Ramona Trestian,et al.  An Innovative Machine-Learning-Based Scheduling Solution for Improving Live UHD Video Streaming Quality in Highly Dynamic Network Environments , 2020, IEEE Transactions on Broadcasting.

[27]  Pingzhi Fan,et al.  Multi-user Multi-channel Computation Offloading and Resource Allocation for Mobile Edge Computing , 2020, ICC 2020 - 2020 IEEE International Conference on Communications (ICC).

[28]  Melike Erol-Kantarci,et al.  Learning-Based Resource Allocation for Data-Intensive and Immersive Tactile Applications , 2018, 2018 IEEE 5G World Forum (5GWF).

[29]  Mohammad T. Kawser,et al.  Performance Comparison between Round Robin andProportional Fair Scheduling Methods for LTE , 2012 .

[30]  Bin Han,et al.  A Comprehensive Survey of RAN Architectures Toward 5G Mobile Communication System , 2019, IEEE Access.

[31]  Yan Chen,et al.  Intelligent 5G: When Cellular Networks Meet Artificial Intelligence , 2017, IEEE Wireless Communications.

[32]  Christos Verikoukis,et al.  Big Data for 5G Intelligent Network Slicing Management , 2020, IEEE Network.