Exploiting Propagation Delay in Underwater Acoustic Communication Networks via Deep Reinforcement Learning

This article proposes a novel deep-reinforcement learning-based medium access control (DL-MAC) protocol for underwater acoustic networks (UANs) where one agent node employing the proposed DL-MAC protocol coexists with other nodes employing traditional protocols, such as time division multiple access (TDMA) or q-Aloha. The DL-MAC agent learns to exploit the large propagation delays inherent in underwater acoustic communications to improve system throughput by either a synchronous or an asynchronous transmission mode. In the sync-DL-MAC protocol, the agent action space is transmission or no transmission, while in the async-DL-MAC, the agent can also vary the start time in each transmission time slot to further exploit the spatiotemporal uncertainty of the UANs. The deep Q-learning algorithm is applied to both sync-DL-MAC and async-DL-MAC agents to learn the optimal policies. A theoretical analysis and computer simulations demonstrate the performance gain obtained by both DL-MAC protocols. The async-DL-MAC protocol outperforms the sync-DL-MAC protocol significantly in sum throughput and packet success rate by adjusting the transmission start time and reducing the length of time slot.