Application of reinforcement learning for security enhancement in cognitive radio networks

Cognitive radio leverages on reinforcement learning (RL) to enhance network security.There is lack of reviews on the application of RL to based security schemes.We cover the challenges, characteristics, performance enhancements, and others. Cognitive radio network (CRN) enables unlicensed users (or secondary users, SUs) to sense for and opportunistically operate in underutilized licensed channels, which are owned by the licensed users (or primary users, PUs). Cognitive radio network (CRN) has been regarded as the next-generation wireless network centered on the application of artificial intelligence, which helps the SUs to learn about, as well as to adaptively and dynamically reconfigure its operating parameters, including the sensing and transmission channels, for network performance enhancement. This motivates the use of artificial intelligence to enhance security schemes for CRNs. Provisioning security in CRNs is challenging since existing techniques, such as entity authentication, are not feasible in the dynamic environment that CRN presents since they require pre-registration. In addition these techniques cannot prevent an authenticated node from acting maliciously. In this article, we advocate the use of reinforcement learning (RL) to achieve optimal or near-optimal solutions for security enhancement through the detection of various malicious nodes and their attacks in CRNs. RL, which is an artificial intelligence technique, has the ability to learn new attacks and to detect previously learned ones. RL has been perceived as a promising approach to enhance the overall security aspect of CRNs. RL, which has been applied to address the dynamic aspect of security schemes in other wireless networks, such as wireless sensor networks and wireless mesh networks can be leveraged to design security schemes in CRNs. We believe that these RL solutions will complement and enhance existing security solutions applied to CRN To the best of our knowledge, this is the first survey article that focuses on the use of RL-based techniques for security enhancement in CRNs.

[1]  Zhu Han,et al.  Defeating Primary User Emulation Attacks Using Belief Propagation in Cognitive Radio Networks , 2012, IEEE Journal on Selected Areas in Communications.

[2]  Alejandro Betancourt,et al.  Intelligent cognitive radio jamming - a game-theoretical approach , 2014, EURASIP J. Adv. Signal Process..

[3]  Ian F. Akyildiz,et al.  Multiagent jamming-resilient control channel game for cognitive radio ad hoc networks , 2012, 2012 IEEE International Conference on Communications (ICC).

[4]  Chee Peng Lim,et al.  Reinforcement learning of competitive and cooperative skills in soccer agents , 2011, Appl. Soft Comput..

[5]  Gary G. Yen,et al.  Coordination of exploration and exploitation in a dynamic environment , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[6]  Kok-Lim Alvin Yau,et al.  Security aspects in the cognition cycle of distributed cognitive radio networks: a survey from a multi-agent perspective , 2013, Int. J. Ad Hoc Ubiquitous Comput..

[7]  Peter L. Bartlett,et al.  Infinite-Horizon Policy-Gradient Estimation , 2001, J. Artif. Intell. Res..

[8]  Youki Kadobayashi,et al.  Toward cost-sensitive self-optimizing anomaly detection and response in autonomic networks , 2011, Comput. Secur..

[9]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[10]  Mihaela van der Schaar,et al.  Spectrum Access Games and Strategic Learning in Cognitive Radio Networks for Delay-Critical Applications , 2009, Proceedings of the IEEE.

[11]  Youyun Xu,et al.  A Q-Learning based sensing task selection scheme for cognitive radio networks , 2009, 2009 International Conference on Wireless Communications & Signal Processing.

[12]  Philippe Ciblat,et al.  Attack and surveillance strategies for selfish primary user emulator in cognitive radio network , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[13]  Ana Galindo-Serrano,et al.  Distributed Q-Learning for Aggregated Interference Control in Cognitive Radio Networks , 2010, IEEE Transactions on Vehicular Technology.

[14]  Yishay Mansour,et al.  Learning Rates for Q-learning , 2004, J. Mach. Learn. Res..

[15]  J.A. Stankovic,et al.  Denial of Service in Sensor Networks , 2002, Computer.

[16]  Francois Chin,et al.  CrownCom 2008 General Co-Chairs' Message , 2008 .

[17]  Sudharman K. Jayaweera,et al.  A Survey on Machine-Learning Techniques in Cognitive Radios , 2013, IEEE Communications Surveys & Tutorials.

[18]  Brandon F. Lo.  A survey of common control channel design in cognitive radio networks , 2011, Phys. Commun..

[19]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[20]  W. Usaha,et al.  Preventing malicious nodes in ad hoc networks using reinforcement learning , 2005, 2005 2nd International Symposium on Wireless Communication Systems.

[21]  T. Charles Clancy,et al.  Security in Cognitive Radio Networks: Threats and Mitigation , 2008, 2008 3rd International Conference on Cognitive Radio Oriented Wireless Networks and Communications (CrownCom 2008).

[22]  Alexandros G. Fragkiadakis,et al.  A Survey on Security Threats and Detection Techniques in Cognitive Radio Networks , 2013, IEEE Communications Surveys & Tutorials.

[23]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[24]  Song Han,et al.  Cognitive radio network security: A survey , 2012, J. Netw. Comput. Appl..

[25]  Michail G. Lagoudakis,et al.  Least-Squares Policy Iteration , 2003, J. Mach. Learn. Res..

[26]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[27]  Alexandros G. Fragkiadakis,et al.  Securing Cognitive Wireless Sensor Networks: A Survey , 2014, Int. J. Distributed Sens. Networks.

[28]  Joseph Mitola,et al.  Cognitive radio: making software radios more personal , 1999, IEEE Wirel. Commun..

[29]  H. Vincent Poor,et al.  Reinforcement learning based distributed multiagent sensing policy for cognitive radio networks , 2011, 2011 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN).

[30]  Shamik Sengupta,et al.  Vulnerabilities in cognitive radio networks: A survey , 2013, Comput. Commun..

[31]  Ali Hamzeh,et al.  Using reinforcement learning to find an optimal set of features , 2013, Comput. Math. Appl..

[32]  Ian F. Akyildiz,et al.  NeXt generation/dynamic spectrum access/cognitive radio wireless networks: A survey , 2006, Comput. Networks.

[33]  Parag Kulkarni.  Reinforcement and Systemic Machine Learning for Decision Making , 2012 .

[34]  Alborz Geramifard,et al.  A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning , 2013, Found. Trends Mach. Learn..

[35]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[36]  D. Vengerov,et al.  An Empirical Model of Factor Adjustment Dynamics , 2006 .

[37]  Sandip Sen,et al.  Comparing trust mechanisms for monitoring aggregator nodes in sensor networks , 2009, AAMAS.

[38]  Daniel Kudenko,et al.  Reinforcement Learning Approaches to Coordination in Cooperative Multi-agent Systems , 2002, Adaptive Agents and Multi-Agents Systems.

[39]  Martin A. Riedmiller,et al.  Batch Reinforcement Learning , 2012, Reinforcement Learning.

[40]  David Charypar,et al.  Q-Learning for Flexible Learning of Daily Activity Plans , 2005 .

[41]  Reza Curtmola,et al.  Mitigating Byzantine Attacks in Ad Hoc Wireless Networks , 2004 .

[42]  Mohsen Guizani,et al.  Opportunistic Bandwidth Sharing Through Reinforcement Learning , 2010, IEEE Transactions on Vehicular Technology.

[43]  Davy Janssens,et al.  Calibrating a New Reinforcement Learning Mechanism for Modeling Dynamic Activity-Travel Behavior and Key Events , 2007 .

[44]  Christian P. Janssen,et al.  When, What, and How Much to Reward in Reinforcement Learning-Based Models of Cognition , 2012, Cogn. Sci..

[45]  Michèle Sebag,et al.  Feature Selection as a One-Player Game , 2010, ICML.

[46]  Chunsheng Xin,et al.  A game-theoretical anti-jamming scheme for cognitive radio networks , 2013, IEEE Network.

[47]  Amir Ghasemi,et al.  Spectrum sensing in cognitive radio networks: requirements, challenges and design trade-offs , 2008, IEEE Communications Magazine.

[48]  Sandip Sen,et al.  Comparing Reputation Schemes for Detecting Malicious Nodes in Sensor Networks , 2011, Comput. J..

[49]  Shamik Sengupta,et al.  Sustenance against RL-Based Sybil Attacks in Cognitive Radio Networks Using Dynamic Reputation System , 2013, MILCOM 2013 - 2013 IEEE Military Communications Conference.

[50]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[51]  Zhiyong Feng,et al.  Centralized channel and power allocation for cognitive radio networks: A Q-learning solution , 2010, 2010 Future Network & Mobile Summit.

[52]  Jordi Pérez-Romero,et al.  Dynamic cooperator selection in cognitive radio networks , 2012, Ad Hoc Networks.

[53]  Ian F. Akyildiz,et al.  Efficient Recovery Control Channel Design in Cognitive Radio Ad Hoc Networks , 2010, IEEE Transactions on Vehicular Technology.

[54]  An He,et al.  A Survey of Artificial Intelligence for Cognitive Radios , 2010, IEEE Transactions on Vehicular Technology.

[55]  Maryam Shokri,et al.  Knowledge of opposite actions for reinforcement learning , 2011, Appl. Soft Comput..

[56]  Dan Rubenstein,et al.  Using Channel Hopping to Increase 802.11 Resilience to Jamming Attacks , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[57]  David Grace,et al.  Efficient exploration in reinforcement learning-based cognitive radio spectrum sharing , 2011, IET Commun..

[58]  Shabnam Sodagari,et al.  An Anti-jamming Strategy for Channel Access in Cognitive Radio Networks , 2011, GameSec.

[59]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[60]  Mohsen Guizani,et al.  Opportunistic Exploitation of Bandwidth Resources through Reinforcement Learning , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[61]  Roman Zajdel.  Epoch-incremental reinforcement learning algorithms , 2013, Int. J. Appl. Math. Comput. Sci..

[62]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[63]  Su Fong Chien,et al.  Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms , 2014, TheScientificWorldJournal.

[64]  Danda B. Rawat,et al.  Advances on Security Threats and Countermeasures for Cognitive Radio Networks: A Survey , 2015, IEEE Communications Surveys & Tutorials.

[65]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[66]  Mee Hong Ling,et al.  Trust and reputation management in cognitive radio networks: a survey , 2014, Secur. Commun. Networks.

[67]  K. J. Ray Liu,et al.  Evolutionary cooperative spectrum sensing game: how to collaborate? , 2010, IEEE Transactions on Communications.

[68]  K. J. Ray Liu,et al.  An anti-jamming stochastic game for cognitive radio networks , 2011, IEEE Journal on Selected Areas in Communications.

[69]  Yenumula B. Reddy.  Detecting Primary Signals for Efficient Utilization of Spectrum Using Q-Learning , 2008, Fifth International Conference on Information Technology: New Generations (itng 2008).

[70]  Roberto Di Pietro,et al.  Jamming mitigation in cognitive radio networks , 2013, IEEE Network.

[71]  Chuang Lin,et al.  Survey on transport control in data center networks , 2013, IEEE Network.

[72]  Ian F. Akyildiz,et al.  Sensor Networks , 2002, Encyclopedia of GIS.

[73]  E. Postma,et al.  Evolutionary Learning Outperforms Reinforcement Learning on Non-Markovian Tasks , 2005 .

[74]  Shamik Sengupta,et al.  Primary user emulation attack in dynamic spectrum access networks: a game-theoretic approach , 2012, IET Commun..

[75]  Jeng-Yih Chiou,et al.  Reinforcement learning in zero-sum Markov games for robot soccer systems , 2004, IEEE International Conference on Networking, Sensing and Control, 2004.

[76]  Manuela M. Veloso,et al.  Multiagent learning using a variable learning rate , 2002, Artif. Intell..

[77]  K. J. Ray Liu,et al.  Anti-Jamming Games in Multi-Channel Cognitive Radio Networks , 2012, IEEE Journal on Selected Areas in Communications.

[78]  Kok-Lim Alvin Yau,et al.  Applications of Reinforcement Learning to Cognitive Radio Networks , 2010, 2010 IEEE International Conference on Communications Workshops.