A self-organizing resource allocation strategy based on Q-learning approach in ultra-dense networks

In ultra-dense heterogeneous cellular networks, with the density of low power base stations (BSs) increasing, the inter-cell interference (ICI) can be extremely strong when all BSs reuse the same time-frequency resources. In this paper, after proving that allocating orthogonal (frequency) sub-bands to adjacent cells can perform better on throughput than reusing the whole bandwidth, we propose a multi-agent Q-learning based resources allocation (QLRA) approach as an enhanced solution to maximize the system performance. For the QLRA, we operate two learning paradigms: the distributed Q-learning (DQL) algorithm and the centralized Q-learning (CQL) algorithm. In the DQL scenario, all small cells learn independently without sharing any information. While in the CQL scenario, interaction between different agents is taken into consideration and resources are scheduled in a centralized way. Simulation results show that both QLRA scenarios can study an ideal resource allocation strategy automatically and achieve better performance on system throughput. Moreover, by scheduling resources in a centralized way, the CQL scenario can improve the system throughput furtherly.

[1]  Tamer A. ElBatt,et al.  A cooperative Q-learning approach for distributed resource allocation in multi-user femtocell networks , 2014, 2014 IEEE Wireless Communications and Networking Conference (WCNC).

[2]  Ying Wang,et al.  A two-stage cluster-based resource management scheme in ultra-dense networks , 2014, 2014 IEEE/CIC International Conference on Communications in China (ICCC).

[3]  Hyung Seok Kim,et al.  A docitive Q-learning approach towards joint resource allocation and power control in self-organised femtocell networks , 2015, Trans. Emerg. Telecommun. Technol..

[4]  Meryem Simsek,et al.  Dynamic Inter-Cell Interference Coordination in HetNets: A reinforcement learning approach , 2012, 2012 IEEE Global Communications Conference (GLOBECOM).

[5]  Andreas Mitschele-Thiel,et al.  Multi-parameter Q-Learning for downlink Inter-Cell Interference Coordination in LTE SON , 2014, 2014 IEEE Symposium on Computers and Communications (ISCC).

[6]  Zhe Hao,et al.  A resource reserved PF algorithm for LTE-A with power allocation , 2015, 2015 International Conference on Estimation, Detection and Information Fusion (ICEDIF).

[7]  Rahim Tafazolli,et al.  Dynamic Clustering Framework for Multi-Cell Scheduling in Dense Small Cell Networks , 2013, IEEE Communications Letters.

[8]  Chenyang Yang,et al.  A graph-based interference topology control for ultra-dense networks , 2014, 2014 12th International Conference on Signal Processing (ICSP).

[9]  Xiaohu You,et al.  Improved MPSO based eICIC algorithm for LTE-a ultra dense HetNets , 2014, 2014 IEEE Global Communications Conference.

[10]  Tomoaki Ohtsuki,et al.  Cell Range Expansion Using Distributed Q-Learning in Heterogeneous Networks , 2013, VTC Fall.

[11]  Yuehong Gao,et al.  A Cooperative Resource Allocation Scheme Based on Self-Organized Network in Ultra-Dense Small Cell Deployment , 2015, 2015 IEEE 81st Vehicular Technology Conference (VTC Spring).