Performance Evaluation of Cognitive Radios: Metrics, Utility Functions, and Methodology

Performance evaluation of cognitive radio (CR) networks is an important problem but has received relatively limited attention from the CR community. Unlike traditional radios, a cognitive radio may change its objectives as radio scenarios vary. Because of the dynamic pairing of objectives and contexts, it is imperative for cognitive radio network designers to have a firm understanding of the interrelationships among goals, performance metrics, utility functions, link/network performance, and operating environments. In this paper, we first overview various performance metrics at the node, network, and application levels. From a game-theoretic viewpoint, we then show that the performance evaluation of cognitive radio networks exhibits the interdependent nature of actions, goals, decisions, observations, and context. We discuss the interrelationships among metrics, utility functions, cognitive engine algorithms, and achieved performance, as well as various testing scenarios. We propose the radio environment map-based scenario-driven testing (REM-SDT) for thorough performance evaluation of cognitive radios. An IEEE 802.22 WRAN cognitive engine testbed is presented to provide further insights into this important problem area.

[1]  Brian M. Sadler,et al.  COGNITIVE RADIOS FOR DYNAMIC SPECTRUM ACCESS - Dynamic Spectrum Access in the Time Domain: Modeling and Exploiting White Space , 2007, IEEE Communications Magazine.

[2]  Jeffrey H. Reed,et al.  A new approach to signal classification using spectral correlation and neural networks , 2005, First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, 2005. DySPAN 2005..

[3]  Stephen B. Wicker,et al.  Game theory in communications: motivation, explanation, and application to power control , 2001, GLOBECOM'01. IEEE Global Telecommunications Conference (Cat. No.01CH37270).

[4]  Jeffrey H. Reed,et al.  A Low Complexity Dynamic Frequency Selection Algorithm for Cognitive Radio Networks , 2007 .

[5]  Thomas W. Rondeau,et al.  Cognitive Techniques: Physical and Link Layers , 2009 .

[6]  Shiwen Mao,et al.  Overhead Analysis for Radio Environment Mapenabled Cognitive Radio Networks , 2006, 2006 1st IEEE Workshop on Networking Technologies for Software Defined Radio Networks.

[7]  Yunnan Wu,et al.  KNOWS: Kognitiv Networking Over White Spaces , 2007 .

[8]  Bhaskar Krishnamachari,et al.  Low-Complexity Approaches to Spectrum Opportunity Tracking , 2007, 2007 2nd International Conference on Cognitive Radio Oriented Wireless Networks and Communications.

[9]  Ching-Lai Hwang,et al.  Fuzzy Multiple Attribute Decision Making - Methods and Applications , 1992, Lecture Notes in Economics and Mathematical Systems.

[10]  Margaret H. Pinson,et al.  Comparing subjective video quality testing methodologies , 2003, Visual Communications and Image Processing.

[11]  Y.T. Hou,et al.  On Path Selection and Rate Allocation for Video in Wireless Mesh Networks , 2009, IEEE/ACM Transactions on Networking.

[12]  C.W. Bostian,et al.  Online modeling of wireless channels with hidden Markov models and channel impulse responses for cognitive radios , 2004, 2004 IEEE MTT-S International Microwave Symposium Digest (IEEE Cat. No.04CH37535).

[13]  Jeffrey H. Reed.  Software Radio , 2002 .

[14]  J. Baina,et al.  Objective methods for assessment of video quality : state of the art , 1997, IEEE Trans. Broadcast..

[15]  Jeffrey H. Reed,et al.  Network Support: The Radio Environment Map , 2009 .

[16]  Youping Zhao,et al.  Enabling Cognitive Radios through Radio Environment Maps , 2007 .

[17]  Jung-Min Park,et al.  Ensuring Trustworthy Spectrum Sensing in Cognitive Radio Networks , 2006, 2006 1st IEEE Workshop on Networking Technologies for Software Defined Radio Networks.

[18]  Theodore S. Rappaport,et al.  Wireless communications - principles and practice , 1996 .

[19]  Simon Haykin,et al.  Cognitive radio: brain-empowered wireless communications , 2005, IEEE Journal on Selected Areas in Communications.

[20]  Allen B. MacKenzie,et al.  Cognitive Radio Performance Analysis , 2006 .

[21]  Jeffrey H. Reed,et al.  Software design issues in networks with software-defined-radio nodes , 2001, Proceedings Tenth IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. WET ICE 2001.

[22]  Eitan Altman,et al.  S-modular games and power control in wireless networks , 2003, IEEE Trans. Autom. Control..

[23]  Joseph Mitola,et al.  Cognitive Radio An Integrated Agent Architecture for Software Defined Radio , 2000 .

[24]  Per Enge,et al.  Multi-Fault Tolerant RAIM Algorithm for Hybrid GPS/TV Positioning , 2007 .

[25]  Fernando Paganini,et al.  Mechanism-based resource allocation for multimedia transmission over spectrum agile wireless networks , 2007, IEEE Journal on Selected Areas in Communications.

[26]  A. Ghasemi,et al.  Collaborative spectrum sensing for opportunistic access in fading environments , 2005, First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, 2005. DySPAN 2005..

[27]  C. Hwang.  Multiple Objective Decision Making - Methods and Applications: A State-of-the-Art Survey , 1979 .

[28]  Joseph Mitola,et al.  Cognitive radio: making software radios more personal , 1999, IEEE Wirel. Commun..

[29]  Thao Nguyen,et al.  XG Dynamic Spectrum Access Field Test Results , 2007 .

[30]  Luciano Lenzini,et al.  Performance Evaluation of the IEEE 802.16 MAC for QoS Support , 2007, IEEE Transactions on Mobile Computing.

[31]  Geoffrey Ye Li,et al.  Soft Combination and Detection for Cooperative Spectrum Sensing in Cognitive Radio Networks , 2007, IEEE Transactions on Wireless Communications.

[32]  Jeffrey H. Reed,et al.  Performance Evaluation of Radio Environment Map-Enabled Cognitive Spectrum-Sharing Networks , 2007, MILCOM 2007 - IEEE Military Communications Conference.

[33]  Ananthram Swami,et al.  Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: A POMDP framework , 2007, IEEE Journal on Selected Areas in Communications.

[34]  James O'Daniell Neel,et al.  Analysis and Design of Cognitive Radio Networks and Distributed Radio Resource Management Algorithms , 2006 .

[35]  Tong Heng Lee,et al.  Multiobjective Evolutionary Algorithms and Applications , 2005, Advanced Information and Knowledge Processing.

[36]  Ying-Chang Liang,et al.  Optimization for Cooperative Sensing in Cognitive Radio Networks , 2007, 2007 IEEE Wireless Communications and Networking Conference.

[37]  Ness B. Shroff,et al.  A utility-based power-control scheme in wireless cellular systems , 2003, TNET.

[38]  Limin Xiao,et al.  Optimization of Detection Time for Channel Efficiency in Cognitive Radio Systems , 2007, 2007 IEEE Wireless Communications and Networking Conference.

[39]  Bernd Girod,et al.  Analysis of packet loss for compressed video: does burst-length matter? , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[40]  Guocong Song,et al.  Utility-based resource allocation and scheduling in OFDM-based wireless broadband networks , 2005, IEEE Communications Magazine.

[41]  IPTV QoE : Understanding and interpreting MDI values White Paper , .

[42]  Roy D. Yates,et al.  A Framework for Uplink Power Control in Cellular Radio Systems , 1995, IEEE J. Sel. Areas Commun..

[43]  Kenneth Baclawski,et al.  The Role of Ontologies in Cognitive Radios , 2009 .

[44]  Ram Ramanathan,et al.  Challenges: a radically new architecture for next generation mobile ad hoc networks , 2005, MobiCom '05.

[45]  C.W. Bostian,et al.  Analog to Digital Converters , 2020, Embedded Systems Design using the MSP430FR2355 LaunchPad™.

[46]  Katta G. Murty,et al.  Nonlinear Programming Theory and Algorithms , 2007, Technometrics.

[47]  Jeffrey H. Reed,et al.  Development of Radio Environment Map Enabled Case- and Knowledge-Based Learning Algorithms for IEEE 802.22 WRAN Cognitive Engines , 2007, 2007 2nd International Conference on Cognitive Radio Oriented Wireless Networks and Communications.

[48]  Guo-Ping Liu,et al.  Multiobjective Optimisation And Control , 2008 .

[49]  Jeffrey H. Reed,et al.  Convergence of cognitive radio networks , 2004, 2004 IEEE Wireless Communications and Networking Conference (IEEE Cat. No.04TH8733).

[50]  Kang G. Shin,et al.  What and how much to gain by spectrum agility? , 2007, IEEE Journal on Selected Areas in Communications.

[51]  Zhi Ding,et al.  Non-Intrusive Cognitive Radio Networks Based on Smart Antenna Technology , 2007, IEEE GLOBECOM 2007 - IEEE Global Telecommunications Conference.

[52]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[53]  Definition of spectrum use and efficiency of a radio system SM Series Spectrum management , 2011 .

[54]  Joseph B. Evans,et al.  COGNITIVE RADIOS FOR DYNAMIC SPECTRUM ACCESS , 2007 .

[55]  Methods for objective and subjective assessment of quality Perceptual evaluation of speech quality ( PESQ ) : An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs , 2002 .

[56]  Thao Nguyen,et al.  XG dynamic spectrum access field test results [Topics in Radio Communications] , 2007, IEEE Communications Magazine.

[57]  K. Moessner,et al.  Techno - economic of collaborative based secondary spectrum usage - E/sup 2/R research project outcomes overview , 2005, First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, 2005. DySPAN 2005..

[58]  Brian M. Sadler,et al.  A Survey of Dynamic Spectrum Access , 2007, IEEE Signal Processing Magazine.

[59]  Pat Langley,et al.  Artificial Intelligence and Intelligent Systems , 2006 .

[60]  Joseph Mitola.  An Integrated Agent Architecture for Software Defined Radio , 2000 .

[61]  J.H. Reed,et al.  Performance of Distributed Dynamic Frequency Selection Schemes for Interference Reducing Networks , 2006, MILCOM 2006 - 2006 IEEE Military Communications conference.

[62]  F. Perich.  Policy-Based Network Management for NeXt Generation Spectrum Access Control , 2007, 2007 2nd IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks.

[63]  Tihao Chiang,et al.  A new rate control scheme using quadratic rate distortion model , 1997, IEEE Trans. Circuits Syst. Video Technol..

[64]  Brian M. Sadler,et al.  Dynamic Spectrum Access: Signal Processing, Networking, and Regulatory Policy , 2006, ArXiv.

[65]  Geoffrey Ye Li,et al.  Soft Combination and Detection for Cooperative Spectrum Sensing in Cognitive Radio Networks , 2008, IEEE GLOBECOM 2007 - IEEE Global Telecommunications Conference.

[66]  Bernd Girod,et al.  Analysis of video transmission over lossy channels , 2000, IEEE Journal on Selected Areas in Communications.

[67]  Mohsen Guizani,et al.  Cognitive Radio Technology , 2006 .

[68]  Linda Doyle,et al.  GUEST EDITORIAL - COGNITIVE RADIOS FOR DYNAMIC SPECTRUM ACCESS , 2007 .

[69]  V.W.S. Chan,et al.  Principles of Digital Communication and Coding , 1979 .

[70]  Andrew J. Viterbi,et al.  Principles of Digital Communication and Coding , 1979 .

[71]  James Clark,et al.  A Proposed Media Delivery Index (MDI) , 2006, RFC.

[72]  P. Strevens.  Iii , 1985 .

[73]  Hanif D. Sherali,et al.  Linear Programming and Network Flows , 1977 .