New Channel Assignments in Cellular Networks: A reinforcement Learning Solution

The optimization of channel assignment in cellular networks is a very complex optimization problem and it becomes more difficult when the network handles different classes of traffic. The objective is that channel utility be maximized so as to maximize service in a stochastic caller environment. We address in this paper, the dynamic channel assignment (DCA) combined with call admission control (CAC) problem in a multimedia cellular network that handles several classes of traffic with different resource requirements. The problem is naturally formulated as a Semi-Markov Decision Process (SMDP) problem and we use an approach based on reinforcement learning (RL) [neuro-dynamic programming (NDP)] method to solving it. An additional solution to construct a call admission control (CAC) policy in an FCA (Fixed Channel Assignment) system through the same approach (RL) is proposed in this paper. We show that the policies obtained using our algorithms QCAC-FCA and Q-DCA provide a good solution and are able to earn significantly higher revenues than classical solutions. A broad set of experiments illustrates the robustness of our policies that improve the Quality of Service (QoS) and reduce call-blocking probabilities for handoff calls in spite of variations in the traffic conditions.

[1]  John N. Tsitsiklis,et al.  Reinforcement Learning for Call Admission Control and Routing in Integrated Service Networks , 1997, NIPS.

[2]  Guy Pujolle,et al.  NACR: A New Adaptive Channel Reservation in Cellular Communication Systems , 2001, Telecommun. Syst..

[3]  Wai Chen,et al.  Connection Admission Control for Mobile Multiple-Class Personal Communications Networks , 1997, IEEE J. Sel. Areas Commun..

[4]  D. O. Reudink,et al.  Dynamic channel assignment in two-dimensional large-scale mobile Radio systems , 1972 .

[5]  Michael L. Littman,et al.  Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach , 1993, NIPS.

[6]  Jie Wang,et al.  Robust dynamic admission control for unified cell and call QoS in statistical multiplexers , 1998, IEEE J. Sel. Areas Commun..

[7]  Ming Zhang,et al.  The nonuniform compact pattern allocation algorithm cellular mobile systems , 1991 .

[8]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[9]  S. Tekinay,et al.  Handover and channel assignment in mobile cellular networks , 1991, IEEE Communications Magazine.

[10]  Andrew G. Barto,et al.  Learning to Act Using Real-Time Dynamic Programming , 1995, Artif. Intell..

[12]  Timothy X. Brown,et al.  Adaptive call admission control under quality of service constraints: a reinforcement learning solution , 2000, IEEE Journal on Selected Areas in Communications.

[13]  Don Towsley,et al.  On optimal call admission control in cellular networks , 1996, Proceedings of IEEE INFOCOM '96. Conference on Computer Communications.

[14]  Sajal K. Das,et al.  Fast determination of QoS-based multicast routes in wireless networks using genetic algorithm , 2001, ICC 2001. IEEE International Conference on Communications. Conference Record (Cat. No.01CH37240).

[15]  E. Del Re,et al.  A dynamic channel allocation technique based on Hopfield neural networks , 1994, Proceedings of 1994 3rd IEEE International Conference on Universal Personal Communications.

[16]  Dimitri P. Bertsekas,et al.  Reinforcement Learning for Dynamic Channel Allocation in Cellular Telephone Systems , 1996, NIPS.

[17]  John N. Tsitsiklis,et al.  A neuro-dynamic programming approach to admission control in ATM networks: the single link case , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[18]  S. Haykin,et al.  A Q-learning-based dynamic channel assignment technique for mobile communication systems , 1999 .

[19]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[20]  John N. Tsitsiklis,et al.  Call admission control and routing in integrated services networks using neuro-dynamic programming , 2000, IEEE Journal on Selected Areas in Communications.

[21]  Guy Pujolle,et al.  Call Admission Control for Multimedia Cellular Networks Using Neuro-dynamic Programming , 2002, NETWORKING.

[22]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[23]  Kumar N. Sivarajan,et al.  Dynamic channel assignment in cellular radio , 1990, 40th IEEE Conference on Vehicular Technology.

[24]  Ming Zhang,et al.  Comparisons of channel assignment strategies in cellular mobile telephone systems , 1989, IEEE International Conference on Communications, World Prosperity Through Communications,.

[25]  Victor C. M. Leung,et al.  Mobility-based predictive call admission control and bandwidth reservation in wireless cellular networks , 2001, Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat. No.01CH37213).

[26]  Mahmoud Naghshineh,et al.  Channel assignment schemes for cellular mobile telecommunication systems: A comprehensive survey , 1996 .

[27]  Andre-Luc Beylot,et al.  A dynamic Q-learning-based call admission control for multimedia cellular networks , 2001 .

[28]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[29]  D. D. Dimitrijevic,et al.  Design and performance analysis of the algorithms for channel allocation in cellular networks , 1993 .

[30]  Timothy X. Brown,et al.  Low Power Wireless Communication via Reinforcement Learning , 1999, NIPS.

[31]  C. K. Un,et al.  Performance of personal portable radio telephone systems with and without guard channels , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[32]  Gerald Tesauro,et al.  Pricing in Agent Economies Using Neural Networks and Multi-agent Q-Learning , 2001, Sequence Learning.

[33]  Manu Sridharan,et al.  Multi-agent Q-learning and regression trees for automated pricing decisions , 2000, Proceedings Fourth International Conference on MultiAgent Systems.

[34]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.