Learning-based route management in wireless ad hoc networks

The nodes in a wireless ad hoc network must act as routers in a self-configuring network without infrastructure. An application running on nodes in the ad hoc network may require that intermediate nodes act as routers, receiving and forwarding data packets to other nodes to overcome limitations of noise, router congestion and limited transmission power. In existing routing protocols, the "self-configuring" aspect of network construction has generally been limited to route selection using a shortest-path routing metric as a predictor of routing efficiency. This limited, network-layer predictor fails to consider the effects of existing traffic on router loads and fails to consider the effects of noise experienced at the MAC layer. Not all network topologies are suited to efficient routing using a shortest-path metric. The location of the nodes and physical characteristics of the network environment can create topologies where shortest-path routing overloads some routers and underutilizes others. Similarly, noise sources can undermine the quality of wireless links depending on the relative distance between the noise sources and the receiving nodes. This dissertation presents a cross-layer predictor that combines the effects of noise and router congestion into a single time-based routing metric based on statistical estimation from recent experience. Also presented is a new cross-layer, adaptive routing protocol, called Warp-5, that not only uses the new routing metric to make better initial routing decisions in a noisy or congested network, but can also adjust previously existing routes as new routes or new noise sources are added to the network. Simulation results for Warp-5 are presented and compared to the existing shortest-path routing protocol AODV. The results show the cross-layer approach of Warp-5 to be superior to shortest-path routing protocols for managing router congestion and noise in wireless ad hoc networks.

[1]  Charles E. Perkins,et al.  Highly dynamic Destination-Sequenced Distance-Vector routing (DSDV) for mobile computers , 1994, SIGCOMM.

[2]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[3]  Thierry Turletti,et al.  IEEE 802.11 rate adaptation: a practical approach , 2004, MSWiM '04.

[4]  Philippe Jacquet,et al.  Optimized Link State Routing Protocol (OLSR) , 2003, RFC.

[5]  R. Bellman.  Dynamic programming. , 1957, Science.

[6]  King Ngi Ngan,et al.  Video Coding for Wireless Communication Systems , 2001 .

[7]  David J. Goodman,et al.  Wireless Personal Communications Systems , 1997 .

[8]  Charles Elkan,et al.  Expectation Maximization Algorithm , 2010, Encyclopedia of Machine Learning.

[9]  Ting Wang,et al.  Adaptive Routing for Sensor Networks using Reinforcement Learning , 2006, The Sixth IEEE International Conference on Computer and Information Technology (CIT'06).

[10]  Hisashi Kobayashi,et al.  Modeling and analysis , 1978 .

[11]  Witold Pedrycz,et al.  Autonomic Communication , 2009 .

[12]  Andrew S. Tanenbaum,et al.  Distributed operating systems , 2009, CSUR.

[13]  David A. Maltz,et al.  The Dynamic Source Routing Protocol (DSR) for Mobile Ad Hoc Networks for IPv4 , 2007, RFC.

[14]  Brian D. Bunday,et al.  Basic queueing theory , 1986 .

[15]  Kristof Van Laerhoven,et al.  How to build smart appliances? , 2001, IEEE Personal Communications.

[16]  Chen-Khong Tham,et al.  Reinforcement learning and CMAC-based adaptive routing for MANETs , 2004, Proceedings. 2004 12th IEEE International Conference on Networks (ICON 2004) (IEEE Cat. No.04EX955).

[17]  Paramvir Bahl,et al.  A rate-adaptive MAC protocol for multi-Hop wireless networks , 2001, MobiCom '01.

[18]  K. V. Prasad.  Principles of Digital Communication Systems and Computer Networks (Electrical and Computer Engineering Series) , 2004 .

[19]  Ram Ramanathan,et al.  Topology control of multihop wireless networks using transmit power adjustment , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[20]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[21]  Barry M. Leiner,et al.  Goals and challenges of the DARPA GloMo program [global mobile information systems] , 1996, IEEE Wirel. Commun..

[22]  Joseph Davis,et al.  Understanding IPv6 , 2002 .

[23]  Dit-Yan Yeung,et al.  Predictive Q-Routing: A Memory-based Reinforcement Learning Approach to Adaptive Traffic Control , 1995, NIPS.

[24]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[25]  Stephen G. Wilson,et al.  Digital Modulation and Coding , 1995 .

[26]  Serge Fdida,et al.  Some Requirements for Autonomic Routing in Self-organizing Networks , 2004, WAC.

[27]  Robert M. Hinden,et al.  IP next generation overview , 1996, CACM.

[28]  A. Laouiti,et al.  Optimized link state routing protocol for ad hoc networks , 2001, Proceedings. IEEE International Multi Topic Conference, 2001. IEEE INMIC 2001. Technology for the 21st Century..

[29]  Jim Kurose,et al.  Study companion, Computer networking, a top-down approach featuring the Internet, third edition, James F. Kurose, Keith W. Ross , 2007 .

[30]  Charles E. Perkins,et al.  Highly Dynamic Destination-Sequenced Distance-Vector Routing (DSDV) for mobile computers , 1994, SIGCOMM.

[31]  David B. Johnson,et al.  The Dynamic Source Routing Protocol for Mobile Ad Hoc Networks , 2003 .

[32]  Charles E. Perkins,et al.  Ad-hoc on-demand distance vector routing , 1999, Proceedings WMCSA'99. Second IEEE Workshop on Mobile Computing Systems and Applications.

[33]  H. Frazier.  The 802.3z Gigabit Ethernet Standard , 1998 .

[34]  T. Moon.  The expectation-maximization algorithm , 1996, IEEE Signal Process. Mag..

[35]  Frank Dellaert,et al.  The Expectation Maximization Algorithm , 2002 .

[36]  Devika Subramanian,et al.  Ants and Reinforcement Learning: A Case Study in Routing in Dynamic Networks , 1997, IJCAI.

[37]  Shailesh Kumar and Risto Miikkulainen.  Dual Reinforcement Q-Routing: An On-Line Adaptive Routing Algorithm , 1997 .

[38]  Leo Monteban,et al.  WaveLAN®-II: A high-performance wireless LAN for the unlicensed band , 1997, Bell Labs Technical Journal.

[39]  Michael L. Littman,et al.  Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach , 1993, NIPS.

[40]  Leonid Peshkin,et al.  Reinforcement learning for adaptive routing , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[41]  Andrea Goldsmith,et al.  Wireless Communications , 2005, 2021 15th International Conference on Advanced Technologies, Systems and Services in Telecommunications (TELSIKS).

[42]  Zygmunt J. Haas,et al.  A new routing protocol for the reconfigurable wireless networks , 1997, Proceedings of ICUPC 97 - 6th International Conference on Universal Personal Communications.

[43]  Michael L. Littman,et al.  A Distributed Reinforcement Learning Scheme for Network Routing , 1993 .

[44]  Simon Haykin,et al.  Correlative Learning: A Basis for Brain and Adaptive Systems (Adaptive and Learning Systems for Signal Processing, Communications and Control Series) , 2007 .

[45]  D. R. Fulkerson,et al.  Flows in Networks. , 1964 .

[46]  Michail G. Lagoudakis,et al.  Model-Free Least-Squares Policy Iteration , 2001, NIPS.

[47]  Risto Miikkulainen,et al.  On-Line Adaptation of a Signal Predistorter through Dual Reinforcement Learning , 1996, ICML.

[48]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[49]  E. Gelenbe,et al.  Genetic Algorithms for Autonomic Route Discovery , 2006, IEEE Workshop on Distributed Intelligent Systems: Collective Intelligence and Its Applications (DIS'06).

[50]  Charles E. Perkins,et al.  Ad hoc On-Demand Distance Vector (AODV) Routing , 2001, RFC.

[51]  Jim Lansford,et al.  HomeRF: wireless networking for the connected home , 2000, IEEE Wirel. Commun..

[52]  Emin Gün Sirer,et al.  SHARP: a hybrid adaptive routing protocol for mobile ad hoc networks , 2003, MobiHoc '03.

[53]  Keith W. Ross,et al.  Computer networking - a top-down approach featuring the internet , 2000 .

[54]  John C. Bicket,et al.  Bit-rate selection in wireless networks , 2005 .

[55]  Leslie Pack Kaelbling,et al.  Mobilized ad-hoc networks: a reinforcement learning approach , 2004 .

[56]  Lex Weaver,et al.  A Multi-Agent Policy-Gradient Approach to Network Routing , 2001, ICML.