Routing, power control and rate adaptation: A Q-learning-based cross-layer design

Mobile ad hoc networks (MANETs) contain individual nodes exchanging data in a distributed manner. Although each node is capable of determining its own transmission power level, rate adaptation method and routing strategy, few existing routing protocols fully explore the potential performance improvement by the joint optimization of power, rate and routing. This paper proposes and evaluates a cross-layer routing protocol considering power control and rate adaptation by using a delay-based non-selfish cost function with a multi-agent Q-learning coordination mechanism. A simulation-based evaluation, considering node density, node mobility, traffic load and flow number, shows that our algorithm improves the average end-to-end latency and throughput with acceptable power consumption level.

[1]  Wendi B. Heinzelman,et al.  A Survey of Routing Protocols that Support QoS in Mobile Ad Hoc Networks , 2007, IEEE Network.

[2]  Leo Monteban,et al.  WaveLAN®-II: A high-performance wireless LAN for the unlicensed band , 1997, Bell Labs Technical Journal.

[3]  Edward W. Knightly,et al.  OAR: An Opportunistic Auto-Rate Media Access Protocol for Ad Hoc Networks , 2005, Wirel. Networks.

[4]  Lawrence Wai-Choong Wong,et al.  An on-demand rate-aware joint power control and routing scheme , 2014, 2014 IEEE Wireless Communications and Networking Conference (WCNC).

[5]  Celimuge Wu,et al.  A MANET protocol considering link stability and bandwidth efficiency , 2009, 2009 International Conference on Ultra Modern Telecommunications & Workshops.

[6]  Izhak Rubin,et al.  High transmission power increases the capacity of ad hoc wireless networks , 2004, 2004 IEEE International Conference on Communications (IEEE Cat. No.04CH37577).

[7]  Natarajan Meghanathan.  On-Demand Maximum Battery Life Routing with Power Sensitive Power Control in Ad Hoc Networks , 2006, International Conference on Networking, International Conference on Systems and International Conference on Mobile Communications and Learning Technologies (ICNICONSMCL'06).

[8]  Guy Pujolle,et al.  Transmission power and data rate aware routing on wireless networks , 2010, Comput. Networks.

[9]  Thierry Turletti,et al.  IEEE 802.11 rate adaptation: a practical approach , 2004, MSWiM '04.

[10]  Gunter Bolch,et al.  Queueing Networks and Markov Chains - Modeling and Performance Evaluation with Computer Science Applications, Second Edition , 1998 .

[11]  Ying Zhang,et al.  Parallel and distributed algorithms for finite constraint satisfaction problems , 1991, Proceedings of the Third IEEE Symposium on Parallel and Distributed Processing.

[12]  Haojun Huang,et al.  A routing algorithm based on Cross-layer Power Control in wireless ad hoc networks , 2010, 2010 5th International ICST Conference on Communications and Networking in China.

[13]  Vikas Kawadia,et al.  Power control and clustering in ad hoc networks , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[14]  Alamelu Nachiappan,et al.  Q-learning based adaptive QoS routing protocol for MANETs , 2011, 2011 International Conference on Recent Trends in Information Technology (ICRTIT).

[15]  Tarek N. Saadawi,et al.  Adaptive Distributed Power Management Algorithm for Interference-Aware Topology Control in Mobile Ad Hoc Networks , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[16]  Victor R. Lesser,et al.  Solving distributed constraint optimization problems using cooperative mediation , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[17]  Antonio Alfredo Ferreira Loureiro,et al.  Transmission power control techniques for wireless sensor networks , 2007, Comput. Networks.

[18]  Paramvir Bahl,et al.  A rate-adaptive MAC protocol for multi-Hop wireless networks , 2001, MobiCom '01.

[19]  Azzedine Boukerche,et al.  Routing protocols in ad hoc networks: A survey , 2011, Comput. Networks.

[20]  Anastasios A. Economides,et al.  Performance evaluation of power control routing for ad-hoc networks , 2006 .

[21]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[22]  Hisashi Kobayashi,et al.  Application of the Diffusion Approximation to Queueing Networks I: Equilibrium Queue Distributions , 1974, JACM.

[23]  Chai-Keong Toh.  Maximum battery life routing to support ubiquitous mobile computing in wireless ad hoc networks , 2001 .

[24]  Nicholas R. Jennings,et al.  Decentralised coordination of low-power embedded devices using the max-sum algorithm , 2008, AAMAS.

[25]  Amnon Meisels,et al.  Asynchronous Forward Bounding for Distributed COPs , 2014, J. Artif. Intell. Res..

[26]  Katia P. Sycara,et al.  No-commitment branch and bound search for distributed constraint optimization , 2006, AAMAS '06.

[27]  Kyung Sup Kwak,et al.  On PHY and MAC Performance in Body Sensor Networks , 2009, EURASIP J. Wirel. Commun. Netw..

[28]  Chen-Khong Tham,et al.  Reinforcement learning and CMAC-based adaptive routing for MANETs , 2004, Proceedings. 2004 12th IEEE International Conference on Networks (ICON 2004) (IEEE Cat. No.04EX955).

[29]  Archie C. Chapman,et al.  A unifying framework for iterative approximate best-response algorithms for distributed constraint optimization problems1 , 2011, The Knowledge Engineering Review.

[30]  Zubair Khalid,et al.  GRAdient Cost Establishment (GRACE) for an Energy-Aware Routing in Wireless Sensor Networks , 2009, EURASIP J. Wirel. Commun. Netw..

[31]  P. R. Kumar,et al.  Power Control in Ad-Hoc Networks: Theory, Architecture, Algorithm and Implementation of the COMPOW Protocol , 2002 .

[32]  Hyuk Lim,et al.  Improving spatial reuse through tuning transmit power, carrier sense threshold, and data rate in multihop wireless networks , 2006, MobiCom '06.

[33]  Charles E. Perkins,et al.  Ad-hoc on-demand distance vector routing , 1999, Proceedings WMCSA'99. Second IEEE Workshop on Mobile Computing Systems and Applications.

[34]  Cheng-Fu Chou,et al.  Topology-Control-based QoS Routing (TLQR) in Wireless Ad Hoc Networks , 2006, 2006 IEEE 17th International Symposium on Personal, Indoor and Mobile Radio Communications.

[35]  Ness B. Shroff,et al.  Energy-Efficient Interference-Based Routing for Multi-Hop Wireless Networks , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[36]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[37]  Vaduvur Bharghavan,et al.  Robust rate adaptation for 802.11 wireless networks , 2006, MobiCom '06.

[38]  Charles E. Perkins,et al.  Highly dynamic Destination-Sequenced Distance-Vector routing (DSDV) for mobile computers , 1994, SIGCOMM.

[39]  Dietrich Braess,et al.  Über ein Paradoxon aus der Verkehrsplanung , 1968, Unternehmensforschung.

[40]  Ece Guran Schmidt,et al.  Machine learning algorithms for accurate flow-based network traffic classification: Evaluation and comparison , 2010, Perform. Evaluation.

[41]  Nicholas R. Jennings,et al.  Bounded approximate decentralised coordination via the max-sum algorithm , 2009, Artif. Intell..

[42]  Boi Faltings,et al.  A Scalable Method for Multiagent Constraint Optimization , 2005, IJCAI.

[43]  M. Yokoo,et al.  Distributed Breakout Algorithm for Solving Distributed Constraint Satisfaction Problems , 1996 .

[44]  Sunghyun Choi,et al.  Interference Analysis and Transmit Power Control in IEEE 802.11a/h Wireless LANs , 2007, IEEE/ACM Transactions on Networking.

[45]  Jie Wu,et al.  PCAR: A power controlled routing protocol for wireless ad hoc networks , 2010, 2010 IEEE International Symposium on "A World of Wireless, Mobile and Multimedia Networks" (WoWMoM).

[46]  Makoto Yokoo,et al.  Adopt: asynchronous distributed constraint optimization with quality guarantees , 2005, Artif. Intell..

[47]  Leslie Pack Kaelbling,et al.  Mobilized ad-hoc networks: a reinforcement learning approach , 2004 .

[48]  Kang G. Shin,et al.  Interference Analysis and Transmit Power Control in IEEE 802.11a/h Wireless LANs , 2007, IEEE/ACM Transactions on Networking.

[49]  Seongkwan Kim,et al.  CARA: Collision-Aware Rate Adaptation for IEEE 802.11 WLANs , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[50]  Chiu-Kuo Liang,et al.  An ad hoc on-demand routing protocol with high packet delivery fraction , 2004, 2004 IEEE International Conference on Mobile Ad-hoc and Sensor Systems (IEEE Cat. No.04EX975).

[51]  Hiromi Okada,et al.  Power control routing for multi hop wireless ad-hoc network , 2001, GLOBECOM'01. IEEE Global Telecommunications Conference (Cat. No.01CH37270).

[52]  Alhussein A. Abouzeid,et al.  Queuing network models for delay analysis of multihop wireless ad hoc networks , 2006, IWCMC '06.

[53]  Jan M. Rabaey,et al.  Distributed algorithms for transmission power control in wireless sensor networks , 2003, 2003 IEEE Wireless Communications and Networking, 2003. WCNC 2003..

[54]  Eric I. Pas,et al.  Braess' paradox: Some new insights , 1997 .