Multipath Communication With Deep Q-Network for Industry 4.0 Automation and Orchestration

In this article, we design a novel multipath communication framework for Industry 4.0 using deep Q-network [1] to achieve human-level intelligence in networking automation and orchestration. To elaborate, we first investigate the challenges and approaches in exploiting heterogeneous networks and multipath communication [e.g., using multipath transmission control protocol (MPTCP)] for the information technology cum operation technology (IT/OT) convergence in Industry 4.0. Based on the novel idea of intelligent and flexible manufacturing, we analyze the technical challenges of IT/OT convergence and then model network data traffics using MPTCP over the converged frameworks. It quantifies the adverse impact of network convergence on the performance for flexible manufacturing. We provide a few proof-of-concepts solutions; however, after a clear understanding of the tradeoffs, we discover the need for experience-driven MPTCP. The simulation result demonstrates that the proposed scheme significantly outperforms the baseline schemes.

[1]  Yang Lu,et al.  Industry 4.0: A survey on technologies, applications and open research issues , 2017, J. Ind. Inf. Integr..

[2]  Carey Williamson,et al.  Modeling Compound TCP Over WiFi for IoT , 2018, IEEE/ACM Transactions on Networking.

[3]  Kavosh Asadi,et al.  An Alternative Softmax Operator for Reinforcement Learning , 2016, ICML.

[4]  Hai Le Vu,et al.  TCP Performance over Wi-Fi: Joint Impact of Buffer and Channel Losses , 2016, IEEE Transactions on Mobile Computing.

[5]  Weihai Chen,et al.  Industrial IoT in 5G environment towards smart manufacturing , 2018, J. Ind. Inf. Integr..

[6]  Jinho Choi,et al.  Federated Learning With Blockchain for Autonomous Vehicles: Analysis and Design Challenges , 2020, IEEE Transactions on Communications.

[7]  Shiva Raj Pokhrel,et al.  Adaptive Admission Control for IoT Applications in Home WiFi Networks , 2020, IEEE Transactions on Mobile Computing.

[8]  Jinho Choi,et al.  Low-Delay Scheduling for Internet of Vehicles: Load-Balanced Multipath Communication With FEC , 2019, IEEE Transactions on Communications.

[9]  Keoma Brun-Laguna,et al.  Deterministic Networking for the Industrial IoT , 2018 .

[10]  Zhiyuan Xu,et al.  Experience-Driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning , 2019, IEEE Journal on Selected Areas in Communications.

[11]  Lihong Li,et al.  PAC model-free reinforcement learning , 2006, ICML.

[12]  Mark Handley,et al.  Coupled Congestion Control for Multipath Transport Protocols , 2011, RFC.

[13]  Jinho Choi,et al.  Improving TCP Performance Over WiFi for Internet of Vehicles: A Federated Learning Approach , 2020, IEEE Transactions on Vehicular Technology.

[14]  Shiva Raj Pokhrel,et al.  Analytical Modeling of Multipath TCP Over Last-Mile Wireless , 2017, IEEE/ACM Transactions on Networking.

[15]  Kaishun Wu,et al.  Adaptive Online Decision Method for Initial Congestion Window in 5G Mobile Edge Computing Using Deep Reinforcement Learning , 2020, IEEE Journal on Selected Areas in Communications.

[16]  Chaojing Xue,et al.  SmartCC: A Reinforcement Learning Approach for Multipath TCP Congestion Control in Heterogeneous Networks , 2019, IEEE Journal on Selected Areas in Communications.

[17]  Shiva Raj Pokhrel,et al.  Compound TCP Performance for Industry 4.0 WiFi: A Cognitive Federated Learning Approach , 2021, IEEE Transactions on Industrial Informatics.

[18]  Michel Mandjes,et al.  Improving Multipath TCP Performance over WiFi and Cellular Networks: An Analytical Approach , 2019, IEEE Transactions on Mobile Computing.

[19]  Hyunseok Chang,et al.  SDN-Based Multi-Protocol Edge Switching for IoT Service Automation , 2018, IEEE Journal on Selected Areas in Communications.

[20]  Erich M. Nahum,et al.  A measurement-based study of MultiPath TCP performance over wireless networks , 2013, Internet Measurement Conference.

[21]  Shiva Raj Pokhrel,et al.  Fair Coexistence of Regular and Multipath TCP over Wireless Last-Miles , 2019, IEEE Transactions on Mobile Computing.

[22]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[23]  Miroslav Popovic,et al.  MPTCP Is Not Pareto-Optimal: Performance Issues and a Possible Solution , 2013, IEEE/ACM Transactions on Networking.

[24]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[25]  Kavosh Asadi,et al.  DeepMellow: Removing the Need for a Target Network in Deep Q-Learning , 2019, IJCAI.

[26]  Jiafu Wan,et al.  Adaptive Transmission Optimization in SDN-Based Industrial Internet of Things With Edge Computing , 2018, IEEE Internet of Things Journal.

[27]  Dan Pei,et al.  Dynamic TCP Initial Windows and Congestion Control Schemes Through Reinforcement Learning , 2019, IEEE Journal on Selected Areas in Communications.

[28]  Steven H. Low,et al.  Multipath TCP: Analysis, Design, and Implementation , 2013, IEEE/ACM Transactions on Networking.