Energy-Efficient Depth-Based Opportunistic Routing with Q-Learning for Underwater Wireless Sensor Networks

Underwater Wireless Sensor Networks (UWSNs) have aroused increasing interest of many researchers in industry, military, commerce and academe recently. Due to the harsh underwater environment, energy efficiency is a significant theme should be considered for routing in UWSNs. Underwater positioning is also a particularly tricky task since the high attenuation of radio-frequency signals in UWSNs. In this paper, we propose an energy-efficient depth-based opportunistic routing algorithm with Q-learning (EDORQ) for UWSNs to guarantee the energy-saving and reliable data transmission. It combines the respective advantages of Q-learning technique and opportunistic routing (OR) algorithm without the full-dimensional location information to improve the network performance in terms of energy consumption, average network overhead and packet delivery ratio. In EDORQ, the void detection factor, residual energy and depth information of candidate nodes are jointly considered when defining the Q-value function, which contributes to proactively detecting void nodes in advance, meanwhile, reducing energy consumption. In addition, a simple and scalable void node recovery mode is proposed for the selection of candidate set so as to rescue packets that are stuck in void nodes unfortunately. Furthermore, we design a novel method to set the holding time for the schedule of packet forwarding base on Q-value so as to alleviate the packet collision and redundant transmission. We conduct extensive simulations to evaluate the performance of our proposed algorithm and compare it with other three routing algorithms on Aqua-sim platform (NS2). The results show that the proposed algorithm significantly improve the performance in terms of energy efficiency, packet delivery ratio and average network overhead without sacrificing too much average packet delay.

[1]  Mark Coates,et al.  Optimal Forwarding in Opportunistic Delay Tolerant Networks With Meeting Rate Estimations , 2015, IEEE Transactions on Signal and Information Processing over Networks.

[2]  Luiz Filipe M. Vieira,et al.  Performance and trade-offs of opportunistic routing in underwater networks , 2012, 2012 IEEE Wireless Communications and Networking Conference (WCNC).

[3]  Seyed Mohammad Ghoreyshi,et al.  Void-Handling Techniques for Routing Protocols in Underwater Sensor Networks: Survey and Challenges , 2017, IEEE Communications Surveys & Tutorials.

[4]  Zhigang Jin,et al.  A Q-Learning-Based Delay-Aware Routing Algorithm to Extend the Lifetime of Underwater Sensor Networks , 2017, Sensors.

[5]  Hasan Mahmood,et al.  Routing Protocols for Underwater Wireless Sensor Networks: Taxonomy, Research Challenges, Routing Strategies and Future Directions , 2018, Sensors.

[6]  Tim Brys,et al.  A Gentle Introduction to Reinforcement Learning , 2016, SUM.

[7]  Yiming Pi,et al.  TDoA for Passive Localization: Underwater versus Terrestrial Environment , 2013, IEEE Transactions on Parallel and Distributed Systems.

[8]  Junaid Qadir,et al.  Energy Balanced Localization-Free Cooperative Noise-Aware Routing Protocols for Underwater Wireless Sensor Networks , 2019, Energies.

[9]  Jun-Hong Cui,et al.  Improving the Robustness of Location-Based Routing for Underwater Sensor Networks , 2007, OCEANS 2007 - Europe.

[10]  Michele Zorzi,et al.  Energy-Efficient Routing Schemes for Underwater Acoustic Networks , 2008, IEEE Journal on Selected Areas in Communications.

[11]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[12]  Yunsi Fei,et al.  QELAR: A Machine-Learning-Based Adaptive Routing Protocol for Energy-Efficient and Lifetime-Extended Underwater Sensor Networks , 2010, IEEE Transactions on Mobile Computing.

[13]  Milica Stojanovic,et al.  Focused beam routing protocol for underwater acoustic networks , 2008, Underwater Networks.

[14]  Azzedine Boukerche,et al.  Modeling and Analysis of Opportunistic Routing in Low Duty-Cycle Underwater Sensor Networks , 2015, MSWiM.

[15]  Mohsen Guizani,et al.  Routing protocols for underwater wireless sensor networks , 2015, IEEE Communications Magazine.

[16]  José-Fernán Martínez,et al.  A Survey on Underwater Acoustic Sensor Network Routing Protocols , 2016, Sensors.

[17]  Nadeem Javaid,et al.  DEADS: Depth and Energy Aware Dominating Set Based Algorithm for Cooperative Routing along with Sink Mobility in Underwater WSNs , 2015, Sensors.

[18]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[19]  Azzedine Boukerche,et al.  Underwater sensor networks: a new challenge for opportunistic routing protocols , 2015, IEEE Communications Magazine.

[20]  Tie Qiu,et al.  Survey on high reliability wireless communication for underwater sensor networks , 2019, J. Netw. Comput. Appl..

[21]  Ning Sun,et al.  A reliable and evenly energy consumed routing protocol for underwater acoustic sensor networks , 2015, 2015 IEEE 20th International Workshop on Computer Aided Modelling and Design of Communication Links and Networks (CAMAD).

[22]  Seyed Mohammad Ghoreyshi,et al.  A Novel Cooperative Opportunistic Routing Scheme for Underwater Sensor Networks , 2016, Sensors.

[23]  Chao Li,et al.  Improving Both Energy and Time Efficiency of Depth-Based Routing for Underwater Sensor Networks , 2015, Int. J. Distributed Sens. Networks.

[24]  Cherry Wakayama,et al.  Utilizing kinematics and selective sweeping in reinforcement learning-based routing algorithms for underwater networks , 2015, Ad Hoc Networks.

[25]  Jun-Hong Cui,et al.  DBR: Depth-Based Routing for Underwater Sensor Networks , 2008, Networking.

[26]  Jing Feng,et al.  Reinforcement Learning-Based Data Forwarding in Underwater Wireless Sensor Networks with Passive Mobility , 2019, Sensors.

[27]  Marília Curado,et al.  A reinforcement learning-based routing for delay tolerant networks , 2013, Eng. Appl. Artif. Intell..

[28]  Yasir Saleem,et al.  Network Simulator NS-2 , 2015 .

[29]  Nirvana Meratnia,et al.  Underwater Acoustic Wireless Sensor Networks: Advances and Future Trends in Physical, MAC and Routing Layers , 2014, Sensors.

[30]  Ruiqin Zhao,et al.  Path Diversity Improved Opportunistic Routing for Underwater Sensor Networks , 2018, Sensors.

[31]  Imran Baig,et al.  A survey on routing techniques in underwater wireless sensor networks , 2011, J. Netw. Comput. Appl..

[32]  Bernhard Plattner,et al.  A Decade of Research in Opportunistic Networks: Challenges, Relevance, and Future Directions , 2017, IEEE Communications Magazine.

[33]  Yuhan Dong,et al.  A Survey of Underwater Optical Wireless Communications , 2017, IEEE Communications Surveys & Tutorials.

[34]  Kiseon Kim,et al.  HydroCast: Pressure Routing for Underwater Sensor Networks , 2016, IEEE Transactions on Vehicular Technology.

[35]  Peng Xie,et al.  VBF: Vector-Based Forwarding Protocol for Underwater Sensor Networks , 2006, Networking.

[36]  Ikram Ud Din,et al.  Energy-Effective Cooperative and Reliable Delivery Routing Protocols for Underwater Wireless Sensor Networks , 2019, Energies.

[37]  Yunyoung Nam,et al.  Underwater Wireless Sensor Networks: A Review of Recent Issues and Challenges , 2019, Wirel. Commun. Mob. Comput..

[38]  Azzedine Boukerche,et al.  Design guidelines for opportunistic routing in underwater networks , 2016, IEEE Communications Magazine.

[39]  Azzedine Boukerche,et al.  Routing protocols in ad hoc networks: A survey , 2011, Comput. Networks.

[40]  Xiaoxiong Zhong,et al.  A Novel Routing Scheme for Resource-Constraint Opportunistic Networks: A Cooperative Multiplayer Bargaining Game Approach , 2016, IEEE Transactions on Vehicular Technology.

[41]  Mario Gerla,et al.  VAPR: Void-Aware Pressure Routing for Underwater Sensor Networks , 2013, IEEE Transactions on Mobile Computing.

[42]  Azzedine Boukerche,et al.  Underwater Wireless Sensor Networks , 2018, ACM Comput. Surv..