Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning

In this paper, a stochastic model predictive control (MPC) method based on reinforcement learning is proposed for energy management of plug-in hybrid electric vehicles (PHEVs). Firstly, the power transfer of each component in a power-split PHEV is described in detail. Then an effective and convergent reinforcement learning controller is trained by the Q-learning algorithm according to the driving power distribution under multiple driving cycles. By constructing a multi-step Markov velocity prediction model, the reinforcement learning controller is embedded into the stochastic MPC controller to determine the optimal battery power in predicted time domain. Numerical simulation results verify that the proposed method achieves superior fuel economy that is close to that by stochastic dynamic programming method. In addition, the effective state of charge tracking in terms of different reference trajectories highlight that the proposed method is effective for online application requiring a fast calculation speed.

[1]  Hongwen He,et al.  Battery SOC constraint comparison for predictive energy management of plug-in hybrid electric bus , 2017 .

[2]  Minggao Ouyang,et al.  Design method of a power management strategy for variable battery capacities range-extended electric vehicles to improve energy efficiency and cost-effectiveness , 2017 .

[3]  Hongwen He,et al.  Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus , 2019, Applied Energy.

[4]  G Ripaccioli,et al.  A stochastic model predictive control approach for series hybrid electric vehicle power management , 2010, Proceedings of the 2010 American Control Conference.

[5]  Yujie Wang,et al.  Adaptive energy management strategy for fuel cell/battery hybrid vehicles using Pontryagin's Minimal Principle , 2019, Journal of Power Sources.

[6]  Abdul Aziz,et al.  Performance analysis of power split hybrid electric vehicles using autonomie , 2011, 2011 IEEE Student Conference on Research and Development.

[7]  Zheng Chen,et al.  A Hierarchical Energy Management Strategy for Power-Split Plug-in Hybrid Electric Vehicles Considering Velocity Prediction , 2018, IEEE Access.

[8]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[9]  Bo Geng,et al.  Energy Management Control of Microturbine-Powered Plug-In Hybrid Electric Vehicles Using the Telemetry Equivalent Consumption Minimization Strategy , 2011, IEEE Transactions on Vehicular Technology.

[10]  Lei Li,et al.  Convex programming energy management and components sizing of a plug-in fuel cell urban logistics vehicle , 2019, Journal of Power Sources.

[11]  Bin Shuai,et al.  Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle , 2019 .

[12]  Jing Sun,et al.  Adaptive model predictive control with propulsion load estimation and prediction for all-electric ship energy management , 2018 .

[13]  Yuan Zou,et al.  Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle , 2015 .

[14]  Zheng Chen,et al.  Energy Management for a Power-Split Plug-In Hybrid Electric Vehicle Based on Reinforcement Learning , 2018, Applied Sciences.

[15]  Din-Yuen Chan,et al.  A Reinforcement-Learning-Based Assisted Power Management With QoR Provisioning for Human–Electric Hybrid Bicycle , 2012, IEEE Transactions on Industrial Electronics.

[16]  Guoqing Xu,et al.  Stochastic Optimal Control of Parallel Hybrid Electric Vehicles , 2017 .

[17]  Chunting Chris Mi,et al.  A novel energy management method for series plug-in hybrid electric vehicles , 2015 .

[18]  Yue Zhao,et al.  Trip-oriented stochastic optimal energy management strategy for plug-in hybrid electric bus , 2016 .

[19]  Taisuke Kobayashi.  Student-t policy in reinforcement learning to acquire global optimum of robot control , 2019, Applied Intelligence.

[20]  Yujie Wang,et al.  Energy management strategy for battery/supercapacitor/fuel cell hybrid source vehicles based on finite state machine , 2019, Applied Energy.

[21]  Hongwen He,et al.  An energy management strategy based on stochastic model predictive control for plug-in hybrid electric buses , 2017 .

[22]  Yuan Zou,et al.  Reinforcement Learning of Adaptive Energy Management With Transition Probability for a Hybrid Electric Tracked Vehicle , 2015, IEEE Transactions on Industrial Electronics.

[23]  Maciej Wieczorek,et al.  A mathematical representation of an energy management strategy for hybrid energy storage system in electric vehicle and real time optimization using a genetic algorithm , 2017 .

[24]  Huei Peng,et al.  A stochastic control strategy for hybrid electric vehicles , 2004, Proceedings of the 2004 American Control Conference.

[25]  Jun Hou,et al.  A hierarchical energy management strategy for hybrid energy storage via vehicle-to-cloud connectivity , 2020, Applied Energy.

[26]  Yujie Wang,et al.  A comparative study of power allocation strategies used in fuel cell and ultracapacitor hybrid systems , 2019 .

[27]  Yujie Wang,et al.  A review of key issues for control and management in battery and ultra-capacitor hybrid energy storage systems , 2020 .

[28]  Jiayi Cao,et al.  Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle , 2018 .

[29]  Hongwen He,et al.  Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming , 2017 .

[30]  Chenghui Zhang,et al.  Analysis and Optimization of Star-Structured Switched-Capacitor Equalizers for Series-Connected Battery Strings , 2018, IEEE Transactions on Power Electronics.

[31]  Xiangyu Wang,et al.  Model predictive control strategy for energy optimization of series-parallel hybrid electric vehicle , 2018, Journal of Cleaner Production.

[32]  Sumedha Rajakaruna,et al.  High-Efficiency Control of Internal Combustion Engines in Blended Charge Depletion/Charge Sustenance Strategies for Plug-In Hybrid Electric Vehicles , 2015, IEEE Transactions on Vehicular Technology.

[33]  Dani Gamerman,et al.  Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference , 1997 .

[34]  Datong Qin,et al.  Research on a multi-objective hierarchical prediction energy management strategy for range extended fuel cell vehicles , 2019, Journal of Power Sources.

[35]  Yujie Wang,et al.  Multiple-grained velocity prediction and energy management strategy for hybrid propulsion systems , 2019 .

[36]  Zheng Chen,et al.  A Vehicle-Environment Cooperative Control Based Velocity Profile Prediction Method and Case Study in Energy Management of Plug-in Hybrid Electric Vehicles , 2019, IEEE Access.

[37]  Kumeresan A. Danapalasingam,et al.  A review on hybrid electric vehicles architecture and energy management strategies , 2016 .

[38]  M. Ouyang,et al.  Approximate Pontryagin’s minimum principle applied to the energy management of plug-in hybrid electric vehicles , 2014 .