Congestion-Aware Routing in Dynamic IoT Networks: A Reinforcement Learning Approach

The innovative services empowered by the Internet of Things (IoT) require a seamless and reliable wireless infrastructure that enables communications within heterogeneous and dynamic low-power and lossy networks (LLNs). The Routing Protocol for LLNs (RPL) was designed to meet the communication requirements of a wide range of IoT application domains. However, a load balancing problem exists in RPL under heavy trafficload scenarios, degrading the network performance in terms of delay and packet delivery. In this paper, we tackle the problem of load-balancing in RPL networks using a reinforcement-learning framework. The proposed method adopts Q-learning at each node to learn an optimal parent selection policy based on the dynamic network conditions. Each node maintains the routing information of its neighbours as Q-values that represent a composite routing cost as a function of the congestion level, the link-quality and the hop-distance. The Q-values are updated continuously exploiting the existing RPL signalling mechanism. The performance of the proposed approach is evaluated through extensive simulations and compared with the existing work to demonstrate its effectiveness. The results show that the proposed method substantially improves network performance in terms of packet delivery and average delay with a marginal increase in the signalling frequency.

[1]  D. PraveenKumar,et al.  Machine learning algorithms for wireless sensor networks: A survey , 2019, Inf. Fusion.

[2]  Jorge Cabral,et al.  Path loss exponent analysis in Wireless Sensor Networks: Experimental evaluation , 2013, 2013 11th IEEE International Conference on Industrial Informatics (INDIN).

[3]  Azzedine Boukerche,et al.  A Survey of Limitations and Enhancements of the IPv6 Routing Protocol for Low-Power and Lossy Networks: A Focus on Core Operations , 2019, IEEE Communications Surveys & Tutorials.

[4]  Abderrezak Rachedi,et al.  Fuzzy-Based Objective Function for Routing Protocol in the Internet of Things , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[5]  Jae-Hyun Ham,et al.  Q-LBR: Q-Learning Based Load Balancing Routing for UAV-Assisted VANET , 2020, Sensors.

[6]  Pascal Thubert,et al.  Objective Function Zero for the Routing Protocol for Low-Power and Lossy Networks (RPL) , 2012, RFC.

[7]  Philip Levis,et al.  RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks , 2012, RFC.

[8]  Michael L. Littman,et al.  Reinforcement learning improves behaviour from evaluative feedback , 2015, Nature.

[9]  Philip Levis,et al.  The Minimum Rank with Hysteresis Objective Function , 2012, RFC.

[10]  Dominique Barthel,et al.  Routing Metrics Used for Path Calculation in Low-Power and Lossy Networks , 2012, RFC.

[11]  Halima Elbiaze,et al.  CLRPL: Context-Aware and Load Balancing RPL for Iot Networks Under Heavy and Highly Dynamic Load , 2018, IEEE Access.

[12]  Julien Montavont,et al.  LoBaPS: Load Balancing Parent Selection for RPL Using Wake-Up Radios , 2019, 2019 IEEE Symposium on Computers and Communications (ISCC).

[13]  Patrick Charpentier,et al.  Toward the Internet of Things for Physical Internet: Perspectives and Challenges , 2020, IEEE Internet of Things Journal.

[14]  Chandan Giri,et al.  Noncooperative Gaming for Energy-Efficient Congestion Control in 6LoWPAN , 2020, IEEE Internet of Things Journal.

[15]  Xianbin Wang,et al.  Toward Massive Machine Type Communications in Ultra-Dense Cellular IoT Networks: Current Issues and Machine Learning-Assisted Solutions , 2018, IEEE Communications Surveys & Tutorials.

[16]  Xin Yuan,et al.  Machine Learning Aided Load Balance Routing Scheme Considering Queue Utilization , 2019, IEEE Transactions on Vehicular Technology.

[17]  Jadhav Rahul,et al.  Optimization of Parent-node Selection in RPL-based Networks , 2017 .

[18]  Jonathan W. Hui,et al.  The Trickle Algorithm , 2011, RFC.

[19]  Ricardo A. L. Rabêlo,et al.  Routing Protocols for Low Power and Lossy Networks in Internet of Things Applications , 2019, Sensors.

[20]  Rashid Ali,et al.  Reinforcement Learning-Enabled Cross-Layer Optimization for Low-Power and Lossy Networks under Heterogeneous Traffic Patterns , 2020, Sensors.

[21]  Saewoong Bahk,et al.  Load Balancing Under Heavy Traffic in RPL Routing Protocol for Low Power and Lossy Networks , 2017, IEEE Transactions on Mobile Computing.