Joint Cache Placement and Delivery Design using Reinforcement Learning for Cellular Networks

We consider a reinforcement learning (RL) based joint cache placement and delivery (CPD) policy for cellular networks with limited caching capacity at both Base Stations (BSs) and User Equipments (UEs). The dynamics of file preferences of users is modeled by a Markov process. User requests are based on current preferences, and on the content of the user’s cache. We assume probabilistic models for the cache placement at both the UEs and the BSs. When the network receives a request for an un-cached file, it fetches the file from the core network via a backhaul link. File delivery is based on network-level orthogonal multipoint multicasting transmissions. For this, all BSs caching a specific file transmit collaboratively in a dedicated resource. File reception depends on the state of the wireless channels. We design the CPD policy while taking into account the user Quality of Service and the backhaul load, and using an Actor-Critic RL framework with two neural networks. Simulation results are used to show the merits of the devised CPD policy.

[1]  Mugen Peng,et al.  Deep Reinforcement Learning Based Coded Caching Scheme in Fog Radio Access Networks , 2018, 2018 IEEE/CIC International Conference on Communications in China (ICCC Workshops).

[2]  Cellular Network Caching Based on Multipoint Multicast Transmissions , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[3]  Urs Niesen,et al.  Online Coded Caching , 2013, IEEE/ACM Transactions on Networking.

[4]  Jasper Goseling,et al.  On Optimal Geographical Caching in Heterogeneous Cellular Networks , 2016, 2017 IEEE Wireless Communications and Networking Conference (WCNC).

[5]  Alireza Sadeghi,et al.  Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-Time Popularities , 2017, IEEE Journal of Selected Topics in Signal Processing.

[6]  Mehdi Bennis,et al.  Living on the edge: The role of proactive caching in 5G wireless networks , 2014, IEEE Communications Magazine.

[7]  Xiaohu You,et al.  Distributed Edge Caching via Reinforcement Learning in Fog Radio Access Networks , 2019, 2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring).

[8]  Xiaofei Wang,et al.  Deep Reinforcement Learning for Cooperative Edge Caching in Future Mobile Networks , 2019, 2019 IEEE Wireless Communications and Networking Conference (WCNC).

[9]  Georgios B. Giannakis,et al.  Deep Reinforcement Learning for Adaptive Caching in Hierarchical Content Delivery Networks , 2019, IEEE Transactions on Cognitive Communications and Networking.

[10]  Bartlomiej Blaszczyszyn,et al.  Optimal geographic caching in cellular networks , 2014, 2015 IEEE International Conference on Communications (ICC).

[11]  Li Fan,et al.  Web caching and Zipf-like distributions: evidence and implications , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[12]  Zhu Han,et al.  Joint User Scheduling and Content Caching Strategy for Mobile Edge Networks Using Deep Reinforcement Learning , 2018, 2018 IEEE International Conference on Communications Workshops (ICC Workshops).

[13]  Zhisheng Niu,et al.  Optimal base station density for energy-efficient heterogeneous cellular networks , 2012, 2012 IEEE International Conference on Communications (ICC).

[14]  Paolo Giaccone,et al.  Analyzing the Performance of LRU Caches under Non-Stationary Traffic Patterns , 2013, ArXiv.

[15]  Tharmalingam Ratnarajah,et al.  Content Placement Learning for Success Probability Maximization in Wireless Edge Caching Networks , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[16]  Pascal Vincent,et al.  Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives , 2012, ArXiv.

[17]  Hikmet Sari,et al.  Transmission techniques for digital terrestrial TV broadcasting , 1995, IEEE Commun. Mag..

[18]  Robert Babuska,et al.  A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[19]  Jing Peng,et al.  Function Optimization using Connectionist Reinforcement Learning Algorithms , 1991 .

[20]  Jeffrey G. Andrews,et al.  A Primer on Cellular Network Analysis Using Stochastic Geometry , 2016, ArXiv.