Optimal Channel Selection Based on Online Decision and Offline Learning in Multichannel Wireless Sensor Networks

We propose a channel selection strategy with hybrid architecture, which combines the centralized method and the distributed method to alleviate the overhead of access point and at the same time provide more flexibility in network deployment. By this architecture, we make use of game theory and reinforcement learning to fulfill the optimal channel selection under different communication scenarios. Particularly, when the network can satisfy the requirements of energy and computational costs, the online decision algorithm based on noncooperative game can help each individual sensor node immediately select the optimal channel. Alternatively, when the network cannot satisfy the requirements of energy and computational costs, the offline learning algorithm based on reinforcement learning can help each individual sensor node to learn from its experience and iteratively adjust its behavior toward the expected target. Extensive simulation results validate the effectiveness of our proposal and also prove that higher system throughput can be achieved by our channel selection strategy over the conventional off-policy channel selection approaches.

[1]  Chao Wang,et al.  A Novel Dynamic Spectrum Access Framework Based on Reinforcement Learning for Cognitive Radio Sensor Networks , 2016, Sensors.

[2]  L. Shapley,et al.  Potential Games , 1994 .

[3]  Lei Tang,et al.  EM-MAC: a dynamic multichannel energy-efficient MAC protocol for wireless sensor networks , 2011, MobiHoc '11.

[4]  Liqiang Zhao,et al.  Using incompletely cooperative game theory in wireless LANs , 2007 .

[5]  Yusun Chang,et al.  Learning through Reinforcement for Repeated Power Control Game in Cognitive Radio Networks , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[6]  Yide Wang,et al.  Distributed Interference-Aware Cooperative MAC Based on Stackelberg Pricing Game , 2015, IEEE Transactions on Vehicular Technology.

[7]  S. Haykin,et al.  A Q-learning-based dynamic channel assignment technique for mobile communication systems , 1999 .

[8]  Tian He,et al.  Realistic and Efficient Multi-Channel Communications in Wireless Sensor Networks , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[9]  Shalinee Kishore,et al.  A game-theoretic analysis of decode-and-forward user cooperation , 2008, IEEE Transactions on Wireless Communications.

[10]  Ann Nowé,et al.  Reinforcement Learning for Self-organizing Wake-Up Scheduling in Wireless Sensor Networks , 2011, ICAART.

[11]  Yueming Cai,et al.  MAC-layer interference mitigation in dynamic and distributed environment: dynamic graphic game with stochastic learning , 2014, The 2014 5th International Conference on Game Theory for Networks.

[12]  Zhu Han,et al.  Data Collection and Wireless Communication in Internet of Things (IoT) Using Economic Analysis and Pricing Models: A Survey , 2016, IEEE Communications Surveys & Tutorials.

[13]  Fouad A. Tobagi,et al.  Cooperative and Non-Cooperative Aloha Games with Channel Capture , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[14]  Bin Han,et al.  Using game theory to investigate stochastic channel selection for multi-channel MAC protocol , 2012, 2012 IEEE International Conference on Communication Systems (ICCS).

[15]  Debdeep Chatterjee,et al.  Resource allocation and cooperative behavior in fading multiple-access channels under uncertainty , 2009, MILCOM 2009 - 2009 IEEE Military Communications Conference.

[16]  Zhenzhen Liu,et al.  RL-MAC: a reinforcement learning based MAC protocol for wireless sensor networks , 2006, Int. J. Sens. Networks.

[17]  Hui Liu,et al.  Resource Allocation for OFDMA Relay Networks With Fairness Constraints , 2006, IEEE Journal on Selected Areas in Communications.

[18]  Chen-Khong Tham,et al.  Distributed Reinforcement Learning Frameworks for Cooperative Retransmission in Wireless Networks , 2010, IEEE Transactions on Vehicular Technology.

[19]  Peijian Ju,et al.  Repeated Game Analysis for Cooperative MAC With Incentive Design for Wireless Networks , 2016, IEEE Transactions on Vehicular Technology.

[20]  Jean-Pierre Hubaux,et al.  Efficient MAC in cognitive radio systems: A game-theoretic approach , 2009, IEEE Transactions on Wireless Communications.

[21]  L. Shapley,et al.  REGULAR ARTICLEPotential Games , 1996 .

[22]  Mihaela van der Schaar,et al.  Game theoretic design of MAC protocols: Pricing and intervention in slotted-Aloha , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[23]  Mohammad Reza Pakravan,et al.  A Game-Theoretic Approach for Power Allocation in Bidirectional Cooperative Communication , 2010, 2010 IEEE Wireless Communication and Networking Conference.

[24]  Mihaela van der Schaar,et al.  Cooperative Multi-Agent Learning and Coordination for Cognitive Radio Networks , 2014, IEEE Journal on Selected Areas in Communications.

[25]  Zhu Han,et al.  Wireless Access in Vehicular Environments Using BitTorrent and Bargaining , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[26]  Pavan Nuggehalli,et al.  A Game-Theoretic Analysis of QoS in Wireless MAC , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[27]  Zhu Han,et al.  Distributed Relay Selection and Power Control for Multiuser Cooperative Communication Networks Using Stackelberg Game , 2009, IEEE Transactions on Mobile Computing.

[28]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[29]  Didem Kivanc-Tureli,et al.  Computationally efficient bandwidth allocation and power control for OFDMA , 2003, IEEE Trans. Wirel. Commun..

[30]  Mei Song,et al.  Reinforcement Learning Based Auction Algorithm for Dynamic Spectrum Access in Cognitive Radio Networks , 2010, 2010 IEEE 72nd Vehicular Technology Conference - Fall.

[31]  Yueming Cai,et al.  Stochastic Game-Theoretic Spectrum Access in Distributed and Dynamic Environment , 2015, IEEE Transactions on Vehicular Technology.

[32]  Yasir Saleem,et al.  Joint channel selection and cluster-based routing scheme based on reinforcement learning for cognitive radio networks , 2015, 2015 International Conference on Computer, Communications, and Control Technology (I4CT).

[33]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[34]  A. Girotra,et al.  Performance Analysis of the IEEE 802 . 11 Distributed Coordination Function , 2005 .

[35]  Tim Clarke,et al.  Distributed Heuristically Accelerated Q-Learning for Robust Cognitive Spectrum Management in LTE Cellular Systems , 2016, IEEE Transactions on Mobile Computing.