Intelligent spectrum management based on reinforcement learning schemes in cooperative cognitive radio networks

Abstract Cognitive Radio (CR) and Cooperative Communication provide key technologies for efficient utilization of available unused spectrum bands (called resources) to achieve a spectral efficient system with high throughput. But to achieve its full potential, it is essential to empower the brain of CR that is Cognitive Engine (CE), using machine learning algorithms to control the operation and adapt parameters according to the dynamic environment. However, in practical scenarios, it is difficult to formulate network model beforehand due to complex network dynamics. To address this issue, the most favorable machine learning scheme, Reinforcement Learning (RL) based schemes are proposed to empower CE without forming an explicit network model. The proposed schemes, Comparison based Cooperative Q-Learning (CCopQL) and Comparison based Cooperative State-Action-Reward-(next) State-(next) Action (CCopSARSA) for resource allocation, allows each CR to learn cooperatively. The cooperation among CRs is in the form of comparing and then exchanging Q-values to obtain an optimal policy. Though these schemes involve information exchange among CRs as compared to independent Q-Leaning and SARSA but it provides improved system performance with high system throughput. Numerical results reveal the significant benefits from exploiting the cooperative feature with RL, both proposed schemes outperform other existing schemes in terms of system throughput and expedite the convergence than individual CR learning with CCopSARSA and CCopQL respectively.