Delay-Optimal Dynamic Mode Selection and Resource Allocation in Device-to-Device Communications—Part II: Practical Algorithm

In Part I of this paper (“Delay-Optimal Dynamic Mode Selection and Resource Allocation in Device-to-Device Communications-Part I: Optimal Policy”), we investigated dynamic mode selection and subchannel allocation for an orthogonal frequency-division multiple access (OFDMA) cellular network with device-to-device (D2D) communications to minimize the average end-to-end delay performance under the dropping probability constraint. We formulated the optimal resource control problem into an infinite-horizon average-reward constrained Markov decision process (CMDP), and the optimal control policy derived in Part I using the brute-force offline value iteration algorithm based on the reduced-state equivalent Bellman equation still faces the well-known curse-of-dimensionality problem, which limits its practical application in realistic scenarios with multiple D2D users and cellular users. In Part II of this paper, we use linear value approximation techniques to further reduce the state space. Moreover, an online stochastic learning algorithm with two timescales is applied to update the value functions and Lagrangian multipliers (LMs) based on the real-time observations of channel state information (CSI) and queue state information (QSI). The combined online stochastic learning solution converges almost surely to a global optimal solution under some realistic conditions. Simulation results show that the proposed approach achieves nearly the same performance as the offline value iteration algorithm and outperforms the conventional CSI-only scheme and throughput-optimal scheme in a stability sense.