Assessing the appropriateness of using markov decision processes for RF spectrum management

The stochastic nature of wireless communication suggests a Markov Decision Process (MDP) as a formalism for identifying and evaluating spectrum control policies. However, in practice numerous factors influence the success or failure of a transmission, so that the applicability of particular MDP models to real spectrum management problems must itself be examined. This paper presents a series of model validation studies in which correspondence between an MDP model and a discrete-event simulation (DES) model is evaluated. We test several hypotheses that together provide a foundation and an exemplar for the idea of using MDPs to guide management of shared spectrum. We conclude that there is sufficient similarity between the performance predictions made by the MDP model and the DES model that MDPs can be used effectively to determine spectrum control policies.

[1]  I. Babuska,et al.  Finite Element Analysis , 2021 .

[2]  Marco Abundo,et al.  An MDP-based admission control for a QoS-aware service-oriented system , 2011, 2011 IEEE Nineteenth IEEE International Workshop on Quality of Service.

[3]  David M. Brooks,et al.  Roughness of microarchitectural design topologies and its implications for optimization , 2008, 2008 IEEE 14th International Symposium on High Performance Computer Architecture.

[4]  Mohsen Guizani,et al.  Cognitive Radio Technology , 2006 .

[5]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[6]  Jonathan P. How,et al.  Experimental Demonstration of Adaptive MDP-Based Planning with Model Uncertainty , 2008 .

[7]  Roger D. Chamberlain,et al.  Use of simple analytic performance models for streaming data applications deployed on diverse architectures , 2013, 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[8]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[9]  Qing Zhao,et al.  Optimality and Complexity of Opportunistic Spectrum Access: A Truncated Markov Decision Process Formulation , 2007, 2007 IEEE International Conference on Communications.

[10]  Winfried K. Grassmann,et al.  Transient solutions in Markovian Queueing Systems , 1977, Comput. Oper. Res..

[11]  Vivek S. Borkar,et al.  A Theory of QoS for Wireless , 2009, IEEE INFOCOM 2009.

[12]  Brian M. Sadler,et al.  A Survey of Dynamic Spectrum Access , 2007, IEEE Signal Processing Magazine.

[13]  Fumiyuki Adachi,et al.  Modeling and Analysis for Reactive-Decision Spectrum Handoff in Cognitive Radio Networks , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[14]  Winfried K. Grassmann Transient solutions in markovian queueing systems , 1977, Comput. Oper. Res..

[15]  Guy Shani,et al.  An MDP-Based Recommender System , 2002, J. Mach. Learn. Res..

[16]  G. Staple,et al.  The end of spectrum scarcity [spectrum allocation and utilization] , 2004, IEEE Spectrum.

[17]  Lang Tong,et al.  A Measurement-Based Model for Dynamic Spectrum Access in WLAN Channels , 2006, MILCOM 2006 - 2006 IEEE Military Communications conference.

[18]  Christopher D. Gill,et al.  Optimizing Expected Time Utility in Cyber-Physical Systems Schedulers , 2010, 2010 31st IEEE Real-Time Systems Symposium.

[19]  Ananthram Swami,et al.  Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: A POMDP framework , 2007, IEEE Journal on Selected Areas in Communications.

[20]  Leonardo Badia,et al.  Simulation models for the performance evaluation of spectrum sharing techniques in OFDMA networks , 2011, MSWiM '11.

[21]  W.H. Tranter,et al.  Dynamic spectrum allocation in cognitive radio using hidden Markov models: Poisson distributed case , 2007, Proceedings 2007 IEEE SoutheastCon.

[22]  Christopher D. Gill,et al.  Towards More Effective Spectrum Use Based on Memory Allocation Models , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference.

[23]  B. A. Sevast'yanov An Ergodic Theorem for Markov Processes and Its Application to Telephone Systems with Refusals , 1957 .

[24]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[25]  Laura Marie Feeney Towards trustworthy simulation of wireless MAC/PHY layers: a comparison framework , 2012, MSWiM '12.