POMDP-Based Energy Cooperative Transmission Policy for Multiple Access Model Powered by Energy Harvesting

To efficiently use harvested energy, we often consider energy cooperation between transmitter nodes in a multiple access wireless network. However the imperfect observations of current network states will degrade the energy transfer policy. In addition, it is difficult to derive an optimal data transmission policy based on the imperfect observation states. In order to solve these problems, we design a partially observed Markov decision process based energy cooperative transmission policy to maximize long-term system throughput. This policy optimizes the energy transfer and data transmission when we cannot observe energy harvesting states and channel states in a current time interval. We adopt the piecewise linearity of the value function and incremental pruning algorithm to obtain the optimal policy for all states in three energy harvesting scenarios. Simulation results show that the proposed cooperative optimal policy converges fast and produces higher sum throughput compared with other policies. Specifically, this policy could be used in device-to-device communication and vehicular networks.

[1]  Nei Kato,et al.  Postdisaster User Location Maneuvering Method for Improving the QoE Guaranteed Service Time in Energy Harvesting Small Cell Networks , 2017, IEEE Transactions on Vehicular Technology.

[2]  Chen Chen,et al.  A GTS Allocation Scheme to Improve Multiple-Access Performance in Vehicular Sensor Networks , 2016, IEEE Transactions on Vehicular Technology.

[3]  Nei Kato,et al.  Device-to-device communications achieve efficient load balancing in LTE-advanced networks , 2014, IEEE Wireless Communications.

[4]  Sungsoo Park,et al.  Optimal Spectrum Access for Energy Harvesting Cognitive Radio Networks , 2013, IEEE Transactions on Wireless Communications.

[5]  Ankita Singh,et al.  Security in vehicular ad hoc network by using multiple operating channels , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[6]  Mengistu Abera Mulatu.  Energy cooperation in communication of energy harvesting tags , 2017 .

[7]  Faisal Karim Shaikh,et al.  Energy harvesting in wireless sensor networks: A comprehensive review , 2016 .

[8]  Behrouz Maham,et al.  Game-Theoretic Approaches for Energy Cooperation in Energy Harvesting Small Cell Networks , 2017, IEEE Transactions on Vehicular Technology.

[9]  Nei Kato,et al.  Space-Air-Ground Integrated Network: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[10]  Victor C. M. Leung,et al.  Delay-Optimal Virtualized Radio Resource Scheduling in Software-Defined Vehicular Networks via Stochastic Learning , 2016, IEEE Transactions on Vehicular Technology.

[11]  Jia Yuan Yu,et al.  A Reinforcement Learning Technique for Optimizing Downlink Scheduling in an Energy-Limited Vehicular Network , 2017, IEEE Transactions on Vehicular Technology.

[12]  Feng Liu,et al.  A Neighborhood-Based Value Iteration Algorithm for POMDP Problems , 2018, 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI).

[13]  Nei Kato,et al.  Device-to-device communications for enhancing quality of experience in software defined multi-tier LTE-A networks , 2015, IEEE Network.

[14]  Bhaskar Krishnamachari,et al.  Optimal power allocation over multiple identical Gilbert-Elliott channels , 2013, 2013 IEEE Global Communications Conference (GLOBECOM).

[15]  Nirwan Ansari,et al.  Joint spectrum allocation and energy harvesting optimization in green powered heterogeneous cognitive radio networks , 2018, Comput. Commun..

[16]  Anthony R. Cassandra,et al.  Optimal Policies for Partially Observable Markov Decision Processes , 1994 .

[17]  Fuhui Zhou,et al.  Optimal Spectrum Sensing Interval in Energy-Harvesting Cognitive Radio Networks , 2017, IEEE Transactions on Cognitive Communications and Networking.

[18]  Nei Kato,et al.  New Perspectives on Future Smart FiWi Networks: Scalability, Reliability, and Energy Efficiency , 2016, IEEE Communications Surveys & Tutorials.

[19]  Hongsheng Xi,et al.  Observation-Based Optimization for POMDPs With Continuous State, Observation, and Action Spaces , 2019, IEEE Transactions on Automatic Control.

[20]  Nei Kato,et al.  A Markovian Analysis for Explicit Probabilistic Stopping-Based Information Propagation in Postdisaster Ad Hoc Mobile Networks , 2016, IEEE Transactions on Wireless Communications.

[21]  Aylin Yener,et al.  Energy Harvesting Networks With Energy Cooperation: Procrastinating Policies , 2015, IEEE Transactions on Communications.

[22]  Nei Kato,et al.  Device-to-Device Communication in LTE-Advanced Networks: A Survey , 2015, IEEE Communications Surveys & Tutorials.

[23]  Marco Miozzo,et al.  Energy sustainable paradigms and methods for future mobile networks: A survey , 2018, Comput. Commun..

[24]  Jing Yang,et al.  Energy Cooperation in Energy Harvesting Communications , 2013, IEEE Transactions on Communications.

[25]  Nei Kato,et al.  On the Outage Probability of Device-to-Device-Communication-Enabled Multichannel Cellular Networks: An RSS-Threshold-Based Perspective , 2016, IEEE Journal on Selected Areas in Communications.

[26]  Jiahao Dai,et al.  Analytical Modeling of Resource Allocation in D2D Overlaying Multihop Multichannel Uplink Cellular Networks , 2017, IEEE Transactions on Vehicular Technology.

[27]  Tain-Sao Chang,et al.  Novel Design on Multiple Channel Sensing for Partially Observable Cognitive Radio Networks , 2017, IEEE Transactions on Mobile Computing.

[28]  Ngo Anh Vien,et al.  A Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes , 2018, IEEE Access.

[29]  Wessam Ajib,et al.  Scheduling Energy Harvesting Roadside Units in Vehicular Ad Hoc Networks , 2016, 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall).

[30]  Michael L. Littman,et al.  A tutorial on partially observable Markov decision processes , 2009 .

[31]  Matthijs T. J. Spaan,et al.  Partially Observable Markov Decision Processes , 2010, Encyclopedia of Machine Learning.

[32]  Vincent W. S. Wong,et al.  Joint Energy Allocation for Sensing and Transmission in Rechargeable Wireless Sensor Networks , 2014, IEEE Transactions on Vehicular Technology.