A New Deep-Q-Learning-Based Transmission Scheduling Mechanism for the Cognitive Internet of Things

Cognitive networks (CNs) are one of the key enablers for the Internet of Things (IoT), where CNs will play an important role in the future Internet in several application scenarios, such as healthcare, agriculture, environment monitoring, and smart metering. However, the current low packet transmission efficiency of IoT faces a problem of the crowded spectrum for the rapidly increasing popularities of various wireless applications. Hence, the IoT that uses the advantages of cognitive technology, namely the cognitive radio-based IoT (CIoT), is a promising solution for IoT applications. A major challenge in CIoT is the packet transmission efficiency using CNs. Therefore, a new Q-learning-based transmission scheduling mechanism using deep learning for the CIoT is proposed to solve the problem of how to achieve the appropriate strategy to transmit packets of different buffers through multiple channels to maximize the system throughput. A Markov decision process-based model is formulated to describe the state transformation of the system. A relay is used to transmit packets to the sink for the other nodes. To maximize the system utility in different system states, the reinforcement learning method, i.e., the Q learning algorithm, is introduced to help the relay to find the optimal strategy. In addition, the stacked auto-encoders deep learning model is used to establish the mapping between the state and the action to accelerate the solution of the problem. Finally, the experimental results demonstrate that the new action selection method can converge after a certain number of iterations. Compared with other algorithms, the proposed method can better transmit packets with less power consumption and packet loss.

[1]  Vincent K. N. Lau.  Performance of variable rate bit interleaved coding for high bandwidth efficiency , 2000, VTC2000-Spring. 2000 IEEE 51st Vehicular Technology Conference Proceedings (Cat. No.00CH37026).

[2]  Ah-Hwee Tan,et al.  Fast Reinforcement Learning under Uncertainties with Self-Organizing Neural Networks , 2015, 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT).

[3]  Jingyuan Zhang,et al.  Application of Artificial Neural Network Based on Q-learning for Mobile Robot Path Planning , 2006, 2006 IEEE International Conference on Information Acquisition.

[4]  Qihui Wu,et al.  Cognitive Internet of Things: A New Paradigm Beyond Connection , 2014, IEEE Internet of Things Journal.

[5]  Jianxiong Zhou,et al.  A Low-Power and Portable Biomedical Device for Respiratory Monitoring with a Stable Power Source , 2015, Sensors.

[6]  Setareh Maghsudi,et al.  Hybrid Centralized–Distributed Resource Allocation for Device-to-Device Communication Underlaying Cellular Networks , 2015, IEEE Transactions on Vehicular Technology.

[7]  Yonghui Song,et al.  Multi-Armed Bandit Channel Access Scheme With Cognitive Radio Technology in Wireless Sensor Networks for the Internet of Things , 2016, IEEE Access.

[8]  Andrey Somov,et al.  Supporting smart-city mobility with cognitive Internet of Things , 2013, 2013 Future Network & Mobile Summit.

[9]  M. Nitti,et al.  Exploiting Social Internet of Things Features in Cognitive Radio , 2016, IEEE Access.

[10]  Michael Gerndt,et al.  Wireless sensors networks for Internet of Things , 2016, 2014 IEEE Ninth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP).

[11]  Deji Chen,et al.  Building a Large Scale Wireless Sensor Network for the Industrial Environment , 2016, 2016 IEEE 22nd International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA).

[12]  T. Praveena,et al.  A novel Scheduling Algorithm emphasizing fairness for Cross Layer Design in wireless networks , 2016, 2016 International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS).

[13]  Sijing Zhang,et al.  Cognitive radio networks for Internet of Things: Applications, challenges and future , 2013, 2013 19th International Conference on Automation and Computing.

[14]  Zhihan Lv,et al.  A Self-Assessment Stereo Capture Model Applicable to the Internet of Things , 2015, Sensors.

[15]  Hong Shen Wang,et al.  Finite-state Markov channel-a useful model for radio communication channels , 1995 .

[16]  Tiejun Lv,et al.  Joint cross-layer design for wireless QoS content delivery , 2004, 2004 IEEE International Conference on Communications (IEEE Cat. No.04CH37577).

[17]  Gang Zhu,et al.  Robust QoS-Aware Cross-layer Design of Adaptive Modulation Transmission on OFDM Systems in High-Speed Railway , 2016, IEEE Access.

[18]  Zhenzhen Peng,et al.  A transmission and scheduling scheme based on W-learning algorithm in wireless networks , 2013, 2013 8th International Conference on Communications and Networking in China (CHINACOM).

[19]  Ivica Kostanic,et al.  Analysis of the FM Radio Spectrum for Secondary Licensing of Low-Power Short-Range Cognitive Internet of Things Devices , 2016, IEEE Access.

[20]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[21]  Wenhui Zhao,et al.  An optimization-based robust routing algorithm to energy-efficient networks for cloud computing , 2016, Telecommun. Syst..

[22]  Mubashir Husain Rehmani,et al.  When Cognitive Radio meets the Internet of Things? , 2016, 2016 International Wireless Communications and Mobile Computing Conference (IWCMC).

[23]  Ghassane Aniba,et al.  Cross-Layer Designed Adaptive Modulation Algorithm with Packet Combining and Truncated ARQ over MIMO Nakagami Fading Channels , 2011, IEEE Transactions on Wireless Communications.

[24]  Bruce Christianson,et al.  BTG-AC: Break-the-Glass Access Control Model for Medical Data in Wireless Sensor Networks , 2016, IEEE Journal of Biomedical and Health Informatics.

[25]  Jianzhong Li,et al.  A Study on Application-Aware Scheduling in Wireless Networks , 2017, IEEE Transactions on Mobile Computing.

[26]  Jianzhong Li,et al.  An Application-Aware Scheduling Policy for Real-Time Traffic , 2015, 2015 IEEE 35th International Conference on Distributed Computing Systems.

[27]  Zheng Ma,et al.  Agricultural environment information collection system based on wireless sensor network , 2012, 2012 IEEE Global High Tech Congress on Electronics.

[28]  Richard Demo Souza,et al.  Rate and Energy Efficient Power Control in a Cognitive Radio Ad Hoc Network , 2013, IEEE Signal Processing Letters.

[29]  Derong Liu,et al.  A Novel Dual Iterative $Q$-Learning Method for Optimal Battery Management in Smart Residential Environments , 2015, IEEE Transactions on Industrial Electronics.

[30]  Li Chen,et al.  Image recognition based on deep learning , 2015, 2015 Chinese Automation Congress (CAC).

[31]  Konstantin Mikhaylov,et al.  Cognitive Internet-of-Things solutions enabled by wireless sensor and actuator networks , 2014, 2014 5th IEEE Conference on Cognitive Infocommunications (CogInfoCom).

[32]  Andrea J. Goldsmith,et al.  Degrees of freedom in adaptive modulation: a unified view , 2001, IEEE Trans. Commun..

[33]  Slawomir Stanczak,et al.  Cognitive Wireless Communications – A paradigm shift in dealing with radio resources as a prerequisite for the wireless network of the future – An overview on the topic of cognitive wireless technologies , 2016 .

[34]  Takaaki KOBAYASHI,et al.  Q-Learning in Continuous State-Action Space by Using a Selective Desensitization Neural Network , 2015 .

[35]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[36]  Mark Humphrys.  W-learning: Competition among selfish Q-learners , 1995 .

[37]  Wei Li,et al.  Distributed Auctions for Task Assignment and Scheduling in Mobile Crowdsensing Systems , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[38]  Zhihan Lv,et al.  Multimedia cloud transmission and storage system based on internet of things , 2017, Multimedia Tools and Applications.

[39]  Min Chen,et al.  Disease Prediction by Machine Learning Over Big Data From Healthcare Communities , 2017, IEEE Access.

[40]  Jaime Lloret Mauri,et al.  Cognitive Networks: Applications and Deployments , 2014 .

[41]  Lin Xiao-hu.  MDP-based energy efficient policy for wireless transmission , 2014 .

[42]  S. Shanmugavel,et al.  Mobility Adaptive Cross Layer Design for Reliable Route Discovery in Ad-hoc Networks , 2007, 2007 Third International Conference on Wireless Communication and Sensor Networks.

[43]  Wenhao Huang,et al.  Deep Architecture for Traffic Flow Prediction: Deep Belief Networks With Multitask Learning , 2014, IEEE Transactions on Intelligent Transportation Systems.

[44]  Jiang,et al.  Optimal and Suboptimal Access and Transmission Polices for Dynamic Spectrum Access over Fading Channels in Cognitive Radio Networks , 2008 .

[45]  Jianzhong Li,et al.  Scheduling Flows With Multiple Service Frequency Constraints , 2017, IEEE Internet of Things Journal.

[46]  Sijing Zhang,et al.  A novel dynamic Q-learning-based scheduler technique for LTE-advanced technologies using neural networks , 2012, 37th Annual IEEE Conference on Local Computer Networks.

[47]  Hefeng Dong,et al.  Simulation study on cross-layer design for energy conservation in underwater acoustic networks , 2013, 2013 OCEANS - San Diego.

[48]  Jianlin Cheng,et al.  A Deep Learning Network Approach to ab initio Protein Secondary Structure Prediction , 2015, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[49]  Min Chen,et al.  A 5G Cognitive System for Healthcare , 2017, Big Data Cogn. Comput..

[50]  Yan Dong,et al.  Dispatching algorithm design for elevator group control system with Q-learning based on a recurrent neural network , 2013, 2013 25th Chinese Control and Decision Conference (CCDC).

[51]  Li Ren,et al.  A Multiagent Q-Learning-Based Optimal Allocation Approach for Urban Water Resource Management System , 2014, IEEE Transactions on Automation Science and Engineering.

[52]  Luca Mainetti,et al.  Evolution of wireless sensor networks towards the Internet of Things: A survey , 2011, SoftCOM 2011, 19th International Conference on Software, Telecommunications and Computer Networks.