A Reinforcement Learning Approach to Age of Information in Multi-User Networks

Scheduling the transmission of time-sensitive data to multiple users over error-prone communication channels is studied with the goal of minimizing the longterm average age of information (AoI) at the users under a constraint on the average number of transmissions. The source can transmit only to a single user at each time slot, and after each transmission, it receives an instantaneous ACK/NACK feedback from the intended receiver, and decides on when and to which user to transmit the next update. The optimal scheduling policy is first studied under different feedback mechanisms when the channel statistics are known; in particular, the standard automatic repeat request (ARQ) and hybrid ARQ (HARQ) protocols are considered. Then a reinforcement learning (RL) approach is introduced, which does not assume any a priori information on the random processes governing the channel states. Different RL methods are applied and compared through numerical simulations.

[1]  Eitan Altman,et al.  Forever Young: Aging Control In DTNs , 2010, ArXiv.

[2]  Eytan Modiano,et al.  Scheduling Policies for Minimizing Age of Information in Broadcast Wireless Networks , 2018, IEEE/ACM Transactions on Networking.

[3]  Michael L. Honig,et al.  Reliability-based type II hybrid ARQ schemes , 2003, IEEE International Conference on Communications, 2003. ICC '03..

[4]  Roy D. Yates,et al.  Status updates through multicast networks , 2017, 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[5]  Sanjit Krishnan Kaul,et al.  Minimizing age of information in vehicular networks , 2011, 2011 8th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks.

[6]  Roy D. Yates,et al.  Status updates through M/G/1/1 queues with HARQ , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[7]  Roy D. Yates,et al.  Status updates over unreliable multiaccess channels , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[8]  Deniz Gündüz,et al.  A Reinforcement Learning Approach to Age of Information in Multi-User Networks , 2018, 2018 IEEE 29th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC).

[9]  Anthony Ephremides,et al.  Optimal Link Scheduling for Age Minimization in Wireless Systems , 2018, IEEE Transactions on Information Theory.

[10]  Roy D. Yates,et al.  Update or wait: How to keep your data fresh , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[11]  Elif Uysal-Biyikoglu,et al.  Age of information under energy replenishment constraints , 2015, 2015 Information Theory and Applications Workshop (ITA).

[12]  Deniz Gündüz,et al.  Designing intelligent energy harvesting communication systems , 2014, IEEE Communications Magazine.

[13]  Eytan Modiano,et al.  Age of information: Design and analysis of optimal scheduling algorithms , 2016, 2017 IEEE International Symposium on Information Theory (ISIT).

[14]  Roy D. Yates,et al.  Real-time status: How often should one update? , 2012, 2012 Proceedings IEEE INFOCOM.

[15]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[16]  Deniz Gündüz,et al.  Average age of information with hybrid ARQ under a resource constraint , 2017, 2018 IEEE Wireless Communications and Networking Conference (WCNC).

[17]  Roy D. Yates,et al.  Timely updates over an erasure channel , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[18]  Peter Auer,et al.  Near-optimal Regret Bounds for Reinforcement Learning , 2008, J. Mach. Learn. Res..