Distributed Q-Learning Aided Heterogeneous Network Association for Energy-Efficient IIoT

To achieve the goal of “Industrial 4.0,” cellular network with wide coverage has gradually become an intensely important carrier for industrial Internet of Things (IIoT). The fifth generation cellular network is expected to be a unifying network that may connect billions of IIoT devices for the sake of supporting advanced IIoT business. In order to realize wide and seamless information coverage, heterogeneous network architecture becomes a beneficial method, which can also improve the near-ceiling network capacity. In order to guarantee the quality of service (QoS) as well as the fairness of different IIoT devices with limited network resources, the network association in IIoT should be performed in a more intelligent manner. In this article, we propose a distributed $Q$-learning aided power allocation algorithm for two-layer heterogeneous IIoT networks. Moreover, we discuss the spirit of designing reward functions, followed by four delicately defined reward functions considering both the QoS of femtocell IoT user equipments and macrocell IoT user equipments and their fairness. Also, both fixed and dynamic learning rates and different kinds of multiagent cooperation modes are investigated. Finally, simulation results show the effectiveness and superiority of our proposed $Q$-learning based power allocation algorithm.