Wi-Fi Assisted Contextual Multi-Armed Bandit for Neighbor Discovery and Selection in Millimeter Wave Device to Device Communications

The unique features of millimeter waves (mmWaves) motivate its leveraging to future, beyond-fifth-generation/sixth-generation (B5G/6G)-based device-to-device (D2D) communications. However, the neighborhood discovery and selection (NDS) problem still needs intelligent solutions due to the trade-off of investigating adjacent devices for the optimum device choice against the crucial beamform training (BT) overhead. In this paper, by making use of multiband (μW/mmWave) standard devices, the mmWave NDS problem is addressed using machine-learning-based contextual multi-armed bandit (CMAB) algorithms. This is done by leveraging the context information of Wi-Fi signal characteristics, i.e., received signal strength (RSS), mean, and variance, to further improve the NDS method. In this setup, the transmitting device acts as the player, the arms are the candidate mmWave D2D links between that device and its neighbors, while the reward is the average throughput. We examine the NDS’s primary trade-off and the impacts of the contextual information on the total performance. Furthermore, modified energy-aware linear upper confidence bound (EA-LinUCB) and contextual Thomson sampling (EA-CTS) algorithms are proposed to handle the problem through reflecting the nearby devices’ withstanding battery levels, which simulate real scenarios. Simulation results ensure the superior efficiency of the proposed algorithms over the single band (mmWave) energy-aware noncontextual MAB algorithms (EA-UCB and EA-TS) and traditional schemes regarding energy efficiency and average throughput with a reasonable convergence rate.

[1]  Djallel Bouneffouf,et al.  Survey on Applications of Multi-Armed and Contextual Bandits , 2020, 2020 IEEE Congress on Evolutionary Computation (CEC).

[2]  Jiguo Yu,et al.  Multi-Armed-Bandit-Based Spectrum Scheduling Algorithms in Wireless Networks: A Survey , 2020, IEEE Wireless Communications.

[3]  Andreas F. Molisch,et al.  Directional neighbor discovery in dual-band systems , 2015, 2015 49th Asilomar Conference on Signals, Systems and Computers.

[4]  Moez Draief,et al.  Parallel Contextual Bandits in Wireless Handover Optimization , 2018, ICDM Workshops.

[5]  Antonio Capone,et al.  Facing the Millimeter-Wave Cell Discovery Challenge in 5G Networks With Context-Awareness , 2016, IEEE Access.

[6]  Kei Sakaguchi,et al.  Millimeter wave beamforming based on WiFi fingerprinting in indoor environment , 2015, 2015 IEEE International Conference on Communication Workshop (ICCW).

[7]  Guihai Chen,et al.  Millimeter Wave Communication: A Comprehensive Survey , 2018, IEEE Communications Surveys & Tutorials.

[8]  Abdollah Homaifar,et al.  Device-to-device communications in the millimeter wave band: A novel distributed mechanism , 2018, 2018 Wireless Telecommunications Symposium (WTS).

[9]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[10]  Akihito Taya,et al.  Communication-Efficient Cooperative Contextual Bandit and Its Application to Wi-Fi BSS Selection , 2020, 2020 IEEE 17th Annual Consumer Communications & Networking Conference (CCNC).

[11]  Ehab Mahmoud Mohamed,et al.  Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit , 2020, Sensors.

[12]  Rémi Munos,et al.  Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis , 2012, ALT.

[13]  Kaishun Wu,et al.  Leveraging Machine-Learning for D2D Communications in 5G/Beyond 5G Networks , 2021 .

[14]  Mohsen Guizani,et al.  5G D2D Networks: Techniques, Challenges, and Future Prospects , 2018, IEEE Systems Journal.

[15]  Walid Saad,et al.  Contextual Bandit Learning for Machine Type Communications in the Null Space of Multi-Antenna Systems , 2020, IEEE Transactions on Communications.

[16]  Xuemin Shen,et al.  Enabling device-to-device communications in millimeter-wave 5G cellular networks , 2015, IEEE Communications Magazine.

[17]  Ignas G. Niemegeers,et al.  CogCell: cognitive interplay between 60 GHz picocells and 2.4/5 GHz hotspots in the 5G era , 2015, IEEE Communications Magazine.

[18]  Kei Sakaguchi,et al.  Wi-Fi Coordinated WiGig Concurrent Transmissions in Random Access Scenarios , 2017, IEEE Transactions on Vehicular Technology.

[19]  E. M. Mohamed,et al.  Minimax Optimal Stochastic Strategy (MOSS) For Neighbor Discovery and Selection In Millimeter Wave D2D Networks , 2020, 2020 23rd International Symposium on Wireless Personal Multimedia Communications (WPMC).

[20]  Robert W. Heath,et al.  Analysis of Blockage Effects on Urban Cellular Networks , 2013, IEEE Transactions on Wireless Communications.

[21]  Jiao Wu,et al.  Millimeter Wave Cell Discovery Based on Out-of-Band Information and Design of Beamforming , 2019, IEEE Access.

[22]  Raphaël Féraud,et al.  A Neural Networks Committee for the Contextual Bandit Problem , 2014, ICONIP.

[23]  Setareh Maghsudi,et al.  Multi-armed bandits with application to 5G small cells , 2015, IEEE Wireless Communications.

[24]  Zhijun Li,et al.  Neighbor Discovery Based on Cross-Technology Communication for Mobile Applications , 2020, IEEE Transactions on Vehicular Technology.

[25]  John Langford,et al.  Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits , 2014, ICML.

[26]  Yu Wang,et al.  Directional neighbor discovery in mmWave wireless networks , 2020, Digit. Commun. Networks.

[27]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[28]  Wansu Lim,et al.  Machine Learning for 5G/B5G Mobile and Wireless Communications: Potential, Limitations, and Future Directions , 2019, IEEE Access.

[29]  Syed Ali Hassan,et al.  Energy Efficient Neighbor Discovery for mmWave D2D Networks Using Polya's Necklaces , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[30]  Aleksandrs Slivkins,et al.  Introduction to Multi-Armed Bandits , 2019, Found. Trends Mach. Learn..

[31]  Teruyuki Miyajima,et al.  Feasibility of RSSI based access network detection for multi-band WLAN using 2.4/5GHz and 60GHz , 2014, 2014 International Symposium on Wireless Personal Multimedia Communications (WPMC).

[32]  Zhu Han,et al.  Machine Learning Paradigms for Next-Generation Wireless Networks , 2017, IEEE Wireless Communications.

[33]  Marwan Krunz,et al.  MAMBA: A Multi-armed Bandit Framework for Beam Tracking in Millimeter-wave Systems , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[34]  Xinyu Zhang,et al.  FastND: Accelerating Directional Neighbor Discovery for 60-GHz Millimeter-Wave Wireless Networks , 2018, IEEE/ACM Transactions on Networking.

[35]  Ming Cheng,et al.  Bandit Inspired Beam Searching Scheme for mmWave High-Speed Train Communications , 2018, ArXiv.

[36]  Ehab Mahmoud Mohamed,et al.  An Efficient Paradigm for Multiband WiGig D2D Networks , 2019, IEEE Access.

[37]  Antonio Capone,et al.  Context Information for Fast Cell Discovery in mm-wave 5G Networks , 2015 .

[38]  José Ferreira de Rezende,et al.  A Clustering Approach for Multiband Neighbor Discovery on 60 GHz WLAN , 2019, Wirel. Commun. Mob. Comput..

[39]  Shipra Agrawal,et al.  Thompson Sampling for Contextual Bandits with Linear Payoffs , 2012, ICML.

[40]  Yuguang Fang,et al.  IEEE 802.11ay-Based mmWave WLANs: Design Challenges and Solutions , 2018, IEEE Communications Surveys & Tutorials.

[41]  Ehab Mahmoud Mohamed,et al.  Relay Probing for Millimeter Wave Multi-Hop D2D Networks , 2020, IEEE Access.

[42]  Omar Hayat,et al.  Device Discovery in D2D Communication: A Survey , 2019, IEEE Access.

[43]  Mats Bengtsson,et al.  Contextual Multi-Armed Bandits for Link Adaptation in Cellular Networks , 2019, NetAI@SIGCOMM.

[44]  Rosa Maria Valdovinos,et al.  A comparison between UCB and UCB-Tuned as selection policies in GGP , 2019, J. Intell. Fuzzy Syst..

[45]  Ehab Mahmoud Mohamed,et al.  Neighbor Discovery and Selection in Millimeter Wave D2D Networks Using Stochastic MAB , 2020, IEEE Communications Letters.

[46]  Ahmed S. Mubarak,et al.  LTE/Wi-Fi/mmWave RAN-Level Interworking Using 2C/U Plane Splitting for Future 5G Networks , 2018, IEEE Access.

[47]  Tassadit Amghar,et al.  Context Enhancement for Linear Contextual Multi-Armed Bandits , 2018, 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI).

[48]  Nor Muzlifah Mahyuddin,et al.  A Comprehensive Survey on Millimeter Wave Communications for Fifth-Generation Wireless Networks: Feasibility and Challenges , 2020, IEEE Access.