Achieving Context Awareness and Intelligence in Cognitive Radio Networks using Reinforcement Learning for Stateful Applications

The tremendous growth in ubiquitous low-cost wireless applications that utilize the unlicensed spectrum bands has laid increasing stress on the limited and scarce radio spectrum resources. Given that the licensed or Primary Users (PUs) are oblivious to the presence of unlicensed or Secondary Users (SUs), Cognitive Radio (CR) is a new paradigm in wireless communication that allows the SUs to detect and use the underutilized licensed spectrum opportunistically and temporarily. Context awareness and intelligence are key characteristics of CR to enable the SU to sense for and use the underutilized licensed spectrum in an efficient manner. In this technical report, we advocate the application of Reinforcement Learning (RL) for achieving context awareness and intelligence, including application schemes that require state representation, or stateful application schemes. In RL, the state encompasses the condition of the operating environment that are relevant to decision making in the application scheme. We investigate the use of RL for stateful applications with respect to Dynamic Channel Selection (DCS) scheme that helps SU Base Station (BS) to select channel adaptively for data transmission to different SU hosts in centralized static and mobile CR networks. The purpose is to enhance Quality of Service (QoS), particularly throughput and delay, and in terms of minimising number of channel switchings. Channel heterogeneity is considered in this paper. Our contribution in this paper, in comparison to our previous work, is the extension of state representation into the DCS application scheme so that the DCS is aware of the changes in the operating environment. Simulation results reveal that the proposed scheme achieves very good performance, and similar trends are observed in our previous work. Keywords–Cognitive radio networks; dynamic channel selection; context awareness; intelligence; reinforcement learning