Self-organized femtocells: a Fuzzy Q-Learning approach

We introduce in this paper the innovative concept of self-organized femtocells for future generation broadband cellular networks. Since the home is the basic unit at which femtocells will be located, their deployment will be massive and their number and position unknown to the operator. This requires femtocells to be autonomous and self-organized, and able to work without human intervention. We propose self-organization to be implemented through Reinforcement Learning (RL) and femtocells to make transmission decisions as a multiagent system, with the objective of maximizing the system capacity and not generating additional interference to the traditional macrocell network. In particular, we manage the femto-to-macro aggregated interference, in realistic wireless settings, by means of Q-Learning (QL) techniques, which allow the femtocells to learn online and distributively the most appropriate resource allocation policy by continuous interactions with the environment. However, QL is based on discrete representation of state and action spaces, which makes the proposed approach not independent of the environment and designer criterion, since it requires a significant human intervention in the definition of the state and action spaces. As a result, we propose to optimize the self-organization capabilities of the proposed scheme by combining QL with the Fuzzy Inference System theory. We then propose a Fuzzy Q-Learning approach which allows avoiding the subjectivity of the QL design with continuous state and action representation, besides improving performance and convergence capabilities. We evaluate simulation results in a 3rd Generation Partnership Project (3GPP) compliant scenario and we compare them to heuristic approaches. Results will show the unique ability of these RL approaches to self-adapt to the dynamics of realistic wireless scenarios. Finally, we discuss the implementability of the proposed schemes in 3GPP systems, and in terms of memory and computational requirements.

[1]  Ana Galindo-Serrano,et al.  Managing Femto to Macro Interference without X2 Interface Support through POMDP , 2012, Mobile Networks and Applications.

[2]  Eitan Altman,et al.  A Nash-Stackelberg Fuzzy Q-Learning Decision Approach in Heterogeneous Cognitive Networks , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[3]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[4]  K. P. Sycara.  Multiagent systems : Special issue on agents , 1998 .

[5]  Ana Galindo-Serrano,et al.  Distributed Q-Learning for Aggregated Interference Control in Cognitive Radio Networks , 2010, IEEE Transactions on Vehicular Technology.

[6]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[7]  R. Bellman.  Dynamic programming. , 1957, Science.

[8]  Pekka Kyosti,et al.  MATLAB implementation of the 3GPP spatial channel model (3GPP TR 25.996) , 2005 .

[9]  Ana Galindo-Serrano,et al.  Distributed Q-Learning for Interference Control in OFDMA-Based Femtocell Networks , 2010, 2010 IEEE 71st Vehicular Technology Conference.

[10]  Chung-Ju Chang,et al.  Fuzzy Q-Learning Admission Control for WCDMA/WLAN Heterogeneous Networks with Multimedia Traffic , 2009, IEEE Transactions on Mobile Computing.

[11]  Rouzbeh Razavi,et al.  Self-optimization of capacity and coverage in LTE networks using a fuzzy reinforcement learning approach , 2010, 21st Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications.

[12]  Pierre Yves Glorennec,et al.  Reinforcement Learning: an Overview , 2000 .

[13]  Gerhard Weiss,et al.  Multiagent Systems , 1999 .

[14]  H. R. Berenji,et al.  Fuzzy Q-learning: a new approach for fuzzy dynamic programming , 1994, Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference.

[15]  Laurence Tianruo Yang,et al.  Fuzzy Logic with Engineering Applications , 1999 .

[16]  Zwi Altman,et al.  Fuzzy-Q-learning-based autonomic management of macro-diversity algorithm in UMTS networks , 2006, Ann. des Télécommunications.

[17]  김재현,et al.  Fuzzy-Q learning , 1996 .

[18]  Mariusz Bajger,et al.  Implementations of Square-Root and Exponential Functions for Large FPGAs , 2006, Asia-Pacific Computer Systems Architecture Conference.

[19]  Ana Galindo-Serrano,et al.  Downlink femto-to-macro interference management based on Fuzzy Q-Learning , 2011, 2011 International Symposium of Modeling and Optimization of Mobile, Ad Hoc, and Wireless Networks.

[20]  Ana Galindo-Serrano,et al.  Designing Time Difference Learning for Interference Management in Heterogeneous Networks , 2013, Dynamic Games and Applications.

[21]  Jeffrey G. Andrews,et al.  Femtocell networks: a survey , 2008, IEEE Communications Magazine.

[22]  Mance E. Harmon,et al.  Reinforcement Learning: A Tutorial. , 1997 .

[23]  Lionel Jouffe,et al.  Fuzzy inference system learning by reinforcement methods , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[24]  Agostino Poggi,et al.  Multiagent Systems , 2006, Intelligenza Artificiale.

[25]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.