Function Approximation Based Reinforcement Learning for Edge Caching in Massive MIMO Networks

Caching popular contents in advance is an important technique to achieve low latency and reduced backhaul congestion in future wireless communication systems. In this article, a multi-cell massive multi-input-multi-output system is considered, where locations of base stations are distributed as a Poisson point process. Assuming probabilistic caching, average success probability (ASP) of the system is derived for a known content popularity (CP) profile, which in practice is time-varying and unknown in advance. Further, modeling CP variations across time as a Markov process, reinforcement <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning is employed to learn the optimal content placement strategy to optimize the long-term-discounted ASP and average cache refresh rate. In the <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning, the number of <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-updates are large and proportional to the number of states and actions. To reduce the space complexity and update requirements towards scalable <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning, two novel (linear and non-linear) function approximations-based <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning approaches are proposed, where only a constant (4 and 3 respectively) number of variables need updation, irrespective of the number of states and actions. Convergence of these approximation-based approaches are analyzed. Simulations verify that these approaches converge and successfully learn the similar best content placement, which shows the successful applicability and scalability of the proposed approximated <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning schemes.