Optimal Cooperative Relaying and Power Control for IoUT Networks With Reinforcement Learning

Internet of Underwater Things (IoUT) consists of numerous sensor nodes distributed in an underwater area for sensing, collecting, processing information, and sending related messages to the data processing center. However, the characteristics of the underwater environment will bring strict limitations on communication coverage and power scarcity to IoUT networks. Applying cooperative communications to IoUT networks can expand the communication range and alleviate power shortages. In this article, we investigate the cooperative communication problem in a power-limited cooperative IoUT system and propose a reinforcement learning-based underwater relay selection strategy. Specifically, we first determine the optimal transmit powers of the source node and the selected underwater relay to maximize the end-to-end signal-to-noise ratio of the system. Then, we formulate the underwater cooperative relaying process as a Markov process and apply reinforcement learning to obtain an effective underwater relay selection strategy. The simulation results show that the performance of the proposed scheme outperforms that of the equal transmit power settings under the same conditions. In addition, the proposed deep Q-network-based underwater relay selection strategy improves the communication efficiency compared with the Q-learning-based strategy, and the number of iterations needed for convergence can be effectively reduced.

[1]  Xiaodong Yan,et al.  Optimal Node Selection for Hybrid Attack in Underwater Acoustic Sensor Networks: A Virtual Expert-Guided Bandit Algorithm , 2020, IEEE Sensors Journal.

[2]  Nasir Saeed,et al.  End-to-End Performance Analysis of Underwater Optical Wireless Relaying and Routing Techniques Under Location Uncertainty , 2019, IEEE Transactions on Wireless Communications.

[3]  Arafet Ben Makhlouf,et al.  Practical Rate Adaptation for Very High Throughput WLANs , 2013, IEEE Transactions on Wireless Communications.

[4]  Ian F. Akyildiz,et al.  Wireless sensor networks: a survey , 2002, Comput. Networks.

[5]  Mohsen Guizani,et al.  A Routing-Driven Key Management Scheme for Heterogeneous Sensor Networks , 2007, 2007 IEEE International Conference on Communications.

[6]  Maode Ma,et al.  Providing deterministic quality-of-service guarantees on WDM optical networks , 2000, IEEE Journal on Selected Areas in Communications.

[7]  Kaishun Wu,et al.  Opportunistic Cooperative Transmission for Underwater Communication Based on the Water’s Key Physical Variables , 2020, IEEE Sensors Journal.

[8]  Jun Cai,et al.  Semi-Distributed User Relaying Algorithm for Amplify-and-Forward Wireless Relay Networks , 2008, IEEE Transactions on Wireless Communications.

[9]  Mohsen Guizani,et al.  An effective key management scheme for heterogeneous sensor networks , 2007, Ad Hoc Networks.

[10]  Mohsen Guizani,et al.  LTE-U and Wi-Fi Coexistence Algorithm Based on Q-Learning in Multi-Channel , 2018, IEEE Access.

[11]  Mohsen Guizani,et al.  Tac-U: A traffic balancing scheme over licensed and unlicensed bands for Tactile Internet , 2019, Future Gener. Comput. Syst..

[12]  Shen Xiaohong,et al.  Cooperative power-frequency joint allocation based game theory for mobile UAN , 2014, OCEANS 2014 - TAIPEI.

[13]  Milica Stojanovic,et al.  Underwater acoustic communication channels: Propagation models and statistical characterization , 2009, IEEE Communications Magazine.

[14]  Qian Zhang,et al.  Enabling the Femtocells: A Cooperation Framework for Mobile and Fixed-Line Operators , 2013, IEEE Transactions on Wireless Communications.

[15]  Lianfen Huang,et al.  QRED: A Q-Learning-based Active Queue Management Scheme , 2018 .

[16]  Chiara Petrioli,et al.  CARMA: Channel-Aware Reinforcement Learning-Based Multi-Path Adaptive Routing for Underwater Wireless Sensor Networks , 2019, IEEE Journal on Selected Areas in Communications.

[17]  Alexander M. Haimovich,et al.  Decode-and-Forward Cooperative Diversity with Power Allocation in Wireless Networks , 2007, IEEE Transactions on Wireless Communications.

[18]  M. Stojanovic,et al.  Underwater acoustic networks , 2000, IEEE Journal of Oceanic Engineering.

[19]  Huseyin Ugur Yildiz,et al.  On the Lifetime of Compressive Sensing Based Energy Harvesting in Underwater Sensor Networks , 2019, IEEE Sensors Journal.

[20]  Muhammad Imran,et al.  Co-EEORS: Cooperative Energy Efficient Optimal Relay Selection Protocol for Underwater Wireless Sensor Networks , 2018, IEEE Access.

[21]  Bernhard Rinner,et al.  Resource coordination in wireless sensor networks by cooperative reinforcement learning , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications Workshops.

[22]  Yunsi Fei,et al.  QELAR: A Machine-Learning-Based Adaptive Routing Protocol for Energy-Efficient and Lifetime-Extended Underwater Sensor Networks , 2010, IEEE Transactions on Mobile Computing.

[23]  Tolga M. Duman,et al.  Cooperative underwater acoustic communications [Accepted From Open Call] , 2013, IEEE Communications Magazine.

[24]  Xiaojiang Du,et al.  Cooperative Communications With Relay Selection Based on Deep Reinforcement Learning in Wireless Sensor Networks , 2019, IEEE Sensors Journal.

[25]  Aria Nosratinia,et al.  Cooperative communication in wireless networks , 2004, IEEE Communications Magazine.

[26]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[27]  Michele Zorzi,et al.  Protocol design issues in underwater acoustic networks , 2011, Comput. Commun..

[28]  Hyundong Shin,et al.  Outage optimality of opportunistic amplify-and-forward relaying , 2007, IEEE Communications Letters.

[29]  Qian Zhang,et al.  Cooperation among wireless service providers: opportunity, challenge, and solution [Dynamic Spectrum Management] , 2010, IEEE Wireless Communications.

[30]  Gregory W. Wornell,et al.  Cooperative diversity in wireless networks: Efficient protocols and outage behavior , 2004, IEEE Transactions on Information Theory.

[31]  Raviraj S. Adve,et al.  Improving amplify-and-forward relay networks: optimal power allocation versus selection , 2006, IEEE Transactions on Wireless Communications.

[32]  Yalew Zelalem Jembre,et al.  REMEDY: Receiver-Initiated MAC Based on Energy-Efficient Duty-Cycling in the IoUT , 2019, IEEE Access.

[33]  A. Belbachir,et al.  Smart Communication for cooperative Wireless Sensor Network , 2016, 2016 International Conference on Applied Electronics (AE).

[34]  Jun Zhang,et al.  Power control for underwater acoustic MC-CDMA communication networks using the improved PSO algorithm , 2016, OCEANS 2016 - Shanghai.

[35]  Lei Yan,et al.  Relay Selection in Underwater Acoustic Cooperative Networks: A Contextual Bandit Approach , 2017, IEEE Communications Letters.

[36]  Tao Jiang,et al.  To Relay or Not to Relay: Open Distance and Optimal Deployment for Linear Underwater Acoustic Networks , 2018, IEEE Transactions on Communications.

[37]  Xi Zhang,et al.  Power-efficient resource allocation for QoS provisioning in underwater MIMO-OFDM acoustic cooperative wireless networks , 2013, 2013 IEEE Globecom Workshops (GC Wkshps).

[38]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[39]  Ashraf Hossain,et al.  Joint Optimal Design for Outage Minimization in DF Relay-Assisted Underwater Acoustic Networks , 2018, IEEE Communications Letters.

[40]  Dario Pompili,et al.  Underwater acoustic sensor networks: research challenges , 2005, Ad Hoc Networks.