Green Resource Allocation Based on Deep Reinforcement Learning in Content-Centric IoT

In the era of information, the green services of content-centric IoT are expected to offer users the better satisfaction of Quality of Experience (QoE) than that in a conventional IoT. Nevertheless, the network traffic and new demands from IoT users increase along with the promising of the content-centric computing system. Therefore, the satisfaction of QoE will become the major challenge in the content-centric computing system for IoT users. In this article, to enhance the satisfaction of QoE, we propose QoE models to evaluate the qualities of the IoT concerning both network and users. The value of QoE does not only refer to the network cost, but also the Mean Opinion Score (MOS) of users. Therefore, our models could capture the influence factors from network cost and services for IoT users based on IoT conditions. Specially, we mainly focus on the issues of cache allocation and transmission rate. Under this content-centric IoT, aiming to allocate the cache capacity among content-centric computing nodes and handle the transmission rates under a constrained total network cost and MOS for the whole IoT, we devote our efforts to the following two aspects. First, we formulate the QoE as a green resource allocation problem under the different transmission rate to acquire the best QoE. Then, in the basis of the node centrality, we will propose a suboptimal dynamic approach, which is suitable for IoT with content delivery frequently. Furthermore, we present a green resource allocation algorithm based on Deep Reinforcement Learning (DRL) to improve accuracy of QoE adaptively. Simulation results reveal that our proposals could achieve high QoE performance for content-centric IoT.

[1]  Zhou Su,et al.  Content distribution over content centric mobile social networks in 5G , 2015, IEEE Communications Magazine.

[2]  Seungjoon Lee,et al.  Optimal Content Placement for a Large-Scale VoD System , 2010, IEEE/ACM Transactions on Networking.

[3]  Jacob Chakareski,et al.  Joint Source-Channel Rate Allocation and Client Clustering for Scalable Multistream IPTV , 2015, IEEE Transactions on Image Processing.

[4]  Peter Han Joo Chong,et al.  Fundamentals of Cluster-Centric Content Placement in Cache-Enabled Device-to-Device Networks , 2015, IEEE Transactions on Communications.

[5]  Arkady B. Zaslavsky,et al.  Context-Aware QoE Modelling, Measurement, and Prediction in Mobile Computing Systems , 2015, IEEE Transactions on Mobile Computing.

[6]  Kun Wang,et al.  Intelligent Resource Management in Blockchain-Based Cloud Datacenters , 2018, IEEE Cloud Computing.

[7]  Thrasyvoulos Spyropoulos,et al.  Effects of Content Popularity on the Performance of Content-Centric Opportunistic Networking: An Analytical Approach and Applications , 2016, IEEE/ACM Transactions on Networking.

[8]  Jianhua Li,et al.  Towards QoE named content-centric wireless multimedia sensor networks with mobile sinks , 2017, 2017 IEEE International Conference on Communications (ICC).

[9]  Shigeng Zhang,et al.  A locality-based range-free localization algorithm for anisotropic wireless sensor networks , 2016, Telecommun. Syst..

[10]  Tao Jiang,et al.  QoE-aware resource allocation for adaptive device-to-device video streaming , 2015, IEEE Network.

[11]  Min Chen,et al.  Fairness Resource Allocation in Blind Wireless Multimedia Communications , 2013, IEEE Transactions on Multimedia.

[12]  Guihai Chen,et al.  Dynamic virtual machine management via approximate Markov decision process , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[13]  Long-Ji Lin,et al.  Reinforcement learning for robots using neural networks , 1992 .

[14]  Qinru Qiu,et al.  A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[15]  B. John Oommen,et al.  Learning Automata-Based Solutions to the Nonlinear Fractional Knapsack Problem With Applications to Optimal Resource Allocation , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Steve Uhlig,et al.  Design and Evaluation of the Optimal Cache Allocation for Content-Centric Networking , 2016, IEEE Transactions on Computers.

[17]  Sherali Zeadally,et al.  QoE-Based Server Selection for Content Distribution Networks , 2014, IEEE Transactions on Computers.

[18]  Meixia Tao,et al.  Stochastic Content-Centric Multicast Scheduling for Cache-Enabled Heterogeneous Cellular Networks , 2015, IEEE Transactions on Wireless Communications.

[19]  Kwangsue Chung,et al.  A client side buffer management algorithm to improve QoE , 2016, IEEE Transactions on Consumer Electronics.

[20]  Song Guo,et al.  Pinpointing Anomaly RFID Tags: Situation and Opportunities , 2017, IEEE Network.

[21]  H. Vincent Poor,et al.  A Secure Mobile Crowdsensing Game With Deep Reinforcement Learning , 2018, IEEE Transactions on Information Forensics and Security.

[22]  Rahul Vaze Online knapsack problem and budgeted truthful bipartite matching , 2017, IEEE INFOCOM 2017 - IEEE Conference on Computer Communications.

[23]  Jie Wu,et al.  Top- $k$ Queries for Categorized RFID Systems , 2017, IEEE/ACM Transactions on Networking.

[24]  Laurence T. Yang,et al.  Energy-Efficient Resource Allocation for D2D Communications Underlaying Cloud-RAN-Based LTE-A Networks , 2016, IEEE Internet of Things Journal.

[25]  Xu Zhou,et al.  QoE-Oriented Rate Allocation for Multipath High-Definition Video Streaming Over Heterogeneous Wireless Access Networks , 2017, IEEE Systems Journal.

[26]  Song Guo,et al.  A Survey on Energy Internet: Architecture, Approach, and Emerging Technologies , 2018, IEEE Systems Journal.

[27]  Yonggang Wen,et al.  How Much to Coordinate? Optimizing In-Network Caching in Content-Centric Networks , 2015, IEEE Transactions on Network and Service Management.

[28]  Der-Jiunn Deng,et al.  Toward trustworthy crowdsourcing in the social internet of things , 2016, IEEE Wireless Communications.

[29]  Hai Dong,et al.  Long-Term QoS-Aware Cloud Service Composition Using Multivariate Time Series Analysis , 2016, IEEE Transactions on Services Computing.

[30]  Jie Wu,et al.  RFID Estimation With Blocker Tags , 2017, IEEE/ACM Transactions on Networking.

[31]  Mianxiong Dong,et al.  Energy-Efficient Matching for Resource Allocation in D2D Enabled Cellular Networks , 2017, IEEE Transactions on Vehicular Technology.

[32]  Mainak Chatterjee,et al.  Effects of Internet Path Selection on Video-QoE: Analysis and Improvements , 2011, IEEE/ACM Transactions on Networking.

[33]  Jianzhao Zhang,et al.  QoE-Aware Downlink User-Cell Association in Small Cell Networks: A Transfer-matching Game Theoretic Solution With Peer Effects , 2016, IEEE Access.

[34]  Ying Wang,et al.  A Data-Driven Architecture for Personalized QoE Management in 5G Wireless Networks , 2017, IEEE Wireless Communications.

[35]  Mehmet Akar,et al.  Fastest Mixing Reversible Markov Chains on Graphs With Degree Proportional Stationary Distributions , 2015, IEEE Transactions on Automatic Control.

[36]  Xiaofei Wang,et al.  Energy Efficiency Optimization: Joint Antenna-Subcarrier-Power Allocation in OFDM-DASs , 2016, IEEE Transactions on Wireless Communications.

[37]  Song Guo,et al.  Green Industrial Internet of Things Architecture: An Energy-Efficient Perspective , 2016, IEEE Communications Standards.

[38]  Gaogang Xie,et al.  CodingCache: multipath-aware CCN cache with network coding , 2013, ICN '13.

[39]  Sherali Zeadally,et al.  QoS-Aware Hierarchical Web Caching Scheme for Online Video Streaming Applications in Internet-Based Vehicular Ad Hoc Networks , 2015, IEEE Transactions on Industrial Electronics.

[40]  Liang Qian,et al.  A QoE-Driven Encoder Adaptation Scheme for Multi-User Video Streaming in Wireless Networks , 2017, IEEE Transactions on Broadcasting.

[41]  Keke Gai,et al.  Dynamic energy-aware cloudlet-based mobile cloud computing model for green computing , 2016, J. Netw. Comput. Appl..

[42]  Yicong Zhou,et al.  QoE Evaluation of Multimedia Services Based on Audiovisual Quality and User Interest , 2016, IEEE Transactions on Multimedia.

[43]  Robbert van Renesse,et al.  Proactive Cache Placement on Cooperative Client Caches for Online Social Networks , 2016, IEEE Transactions on Parallel and Distributed Systems.

[44]  Minyi Guo,et al.  Reinforcement learning-based adaptive resource management of differentiated services in geo-distributed data centers , 2017, 2017 IEEE/ACM 25th International Symposium on Quality of Service (IWQoS).

[45]  Kyoung-Rok Cho,et al.  Memristor MOS Content Addressable Memory (MCAM): Hybrid Architecture for Future High Performance Search Engines , 2010, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[46]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[47]  Weichao Li,et al.  Detecting Low-Quality Workers in QoE Crowdtesting: A Worker Behavior-Based Approach , 2017, IEEE Transactions on Multimedia.

[48]  Mehul Motani,et al.  A QoE-Aware Resource Distribution Framework Incentivizing Context Sharing and Moderate Competition , 2016, IEEE/ACM Transactions on Networking.

[49]  Valentino Pacifici,et al.  Coordinated Selfish Distributed Caching for Peering Content-Centric Networks , 2016, IEEE/ACM Transactions on Networking.

[50]  Song Guo,et al.  Crowdsourcing-Based Content-Centric Network: A Social Perspective , 2017, IEEE Network.

[51]  Bertrand Mathieu,et al.  Economic incentives for deploying local-aware ICN-based content delivery , 2016, 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[52]  Yue Yu,et al.  A query-matching mechanism over out-of-order event stream in IOT , 2013, Int. J. Ad Hoc Ubiquitous Comput..

[53]  Toshiaki Miyazaki,et al.  QoE-Based Big Data Analysis with Deep Learning in Pervasive Edge Environment , 2018, 2018 IEEE International Conference on Communications (ICC).

[54]  Haiying Shen,et al.  Distributed Autonomous Virtual Resource Management in Datacenters Using Finite-Markov Decision Process , 2014, IEEE/ACM Transactions on Networking.

[55]  Shigeng Zhang,et al.  Flexible and Time-Efficient Tag Scanning with Handheld Readers , 2016, IEEE Transactions on Mobile Computing.

[56]  Min Gao,et al.  Probabilistic Model Checking and Scheduling Implementation of an Energy Router System in Energy Internet for Green Cities , 2018, IEEE Transactions on Industrial Informatics.

[57]  Keke Gai,et al.  Energy-aware task assignment for mobile cyber-enabled applications in heterogeneous cloud computing , 2018, J. Parallel Distributed Comput..