Deep Deterministic Policy Gradient Based Energy Management Strategy for Hybrid Electric Tracked Vehicle With Online Updating Mechanism

In this paper, an online energy management strategy (EMS) for hybrid electric tracked vehicle (HETV) is developed based on deep deterministic policy gradient (DDPG) with time-varying weighting factor to further improve economic performance of HETV and reduce computational burden. The DDPG is applied to model the EMS problem for the target HETV. Especially, a time-varying weighting factor is introduced here to update old network parameters with experience learned from most recent cycle segment. Afterwards, simulation is conducted to verify the effectiveness and adaptability of the proposed method. Results show that DDPG-based EMS with online updating mechanism can achieve nearly 90% fuel economy performance as that of dynamic programming while computational time is greatly reduced. Finally, hardware-in-loop experiment is carried out to evaluate the real-world performance of the proposed method.

[1]  Jingyuan Zhan,et al.  Hybrid-Trip-Model-Based Energy Management of a PHEV With Computation-Optimized Dynamic Programming , 2018, IEEE Transactions on Vehicular Technology.

[2]  Ming Li,et al.  Adaptive Real-Time Energy Management Strategy for Plug-In Hybrid Electric Vehicle Based on Simplified-ECMS and a Novel Driving Pattern Recognition Method , 2018, Mathematical Problems in Engineering.

[3]  Benjamín Pla,et al.  Analytical Optimal Solution to the Energy Management Problem in Series Hybrid Electric Vehicles , 2018, IEEE Transactions on Vehicular Technology.

[4]  Junmin Wang,et al.  A Parallel Hybrid Electric Vehicle Energy Management Strategy Using Stochastic Model Predictive Control With Road Grade Preview , 2015, IEEE Transactions on Control Systems Technology.

[5]  Amir Khajepour,et al.  Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information , 2019 .

[6]  Giorgio Rizzoni,et al.  General supervisory control policy for the energy optimization of charge-sustaining hybrid electric vehicles , 2001 .

[7]  Junhui Shi,et al.  Optimization of integrated energy management for a dual-motor coaxial coupling propulsion electric city bus , 2019, Applied Energy.

[8]  Hongwen He,et al.  Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming , 2017 .

[9]  Yuan Zou,et al.  Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning , 2019, Applied Energy.

[10]  Feng Ding,et al.  MPC-based energy management with adaptive Markov-chain prediction for a dual-mode hybrid electric vehicle , 2017 .

[11]  Xiaosong Hu,et al.  Pontryagin’s Minimum Principle based model predictive control of energy management for a plug-in hybrid electric bus , 2019, Applied Energy.

[12]  Chen Lv,et al.  Optimal Energy Management and Sizing of a Dual Motor-Driven Electric Powertrain , 2019, IEEE Transactions on Power Electronics.

[13]  Zhang Chenghui,et al.  Particle Swarm Optimization for energy management fuzzy controller design in dual-source electric vehicle , 2007, 2007 IEEE Power Electronics Specialists Conference.

[14]  Yuan Zou,et al.  Reinforcement Learning of Adaptive Energy Management With Transition Probability for a Hybrid Electric Tracked Vehicle , 2015, IEEE Transactions on Industrial Electronics.

[15]  Hongmin Meng,et al.  Cooperative energy management optimization based on distributed MPC in grid-connected microgrids community , 2019, International Journal of Electrical Power & Energy Systems.

[16]  Jun Wang,et al.  Control rules extraction and parameters optimization of energy management for bus series-parallel AMT hybrid powertrain , 2018, J. Frankl. Inst..

[17]  Perry Y. Li,et al.  Energy management strategy for a power-split hydraulic hybrid vehicle based on Lagrange multiplier and its modifications , 2018, J. Syst. Control. Eng..

[18]  Nassim Rizoug,et al.  Optimal Energy Management for a Li-Ion Battery/Supercapacitor Hybrid Energy Storage System Based on a Particle Swarm Optimization Incorporating Nelder–Mead Simplex Approach , 2017, IEEE Transactions on Intelligent Vehicles.

[19]  Zheng Chen,et al.  Comparisons of Energy Management Methods for a Parallel Plug-In Hybrid Electric Vehicle between the Convex Optimization and Dynamic Programming , 2018 .

[20]  Yuan Zou,et al.  Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle , 2015 .

[21]  Jiankun Peng,et al.  Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space , 2019, Energy Conversion and Management.

[22]  F. Carlucci,et al.  Hybrid Electric Vehicles: Some Theoretical Considerations on Consumption Behaviour , 2018 .

[23]  R Bellman,et al.  On the Theory of Dynamic Programming. , 1952, Proceedings of the National Academy of Sciences of the United States of America.

[24]  Xiaolin Tang,et al.  Human-like Energy Management Based on Deep Reinforcement Learning and Historical Driving Experiences , 2020, ArXiv.

[25]  Yi-Hsuan Hung,et al.  Optimal strategies of energy management integrated with transmission control for a hybrid electric vehicle using dynamic particle swarm optimization , 2018, Energy.

[26]  Huei Peng,et al.  Optimal Control of Hybrid Electric Vehicles Based on Pontryagin's Minimum Principle , 2011, IEEE Transactions on Control Systems Technology.

[27]  Huei Peng,et al.  Comparative Study of Dynamic Programming and Pontryagin’s Minimum Principle on Energy Management for a Parallel Hybrid Electric Vehicle , 2013 .

[28]  Lino Guzzella,et al.  Engine On/Off Control for the Energy Management of a Serial Hybrid Electric Bus via Convex Optimization , 2014, IEEE Transactions on Vehicular Technology.

[29]  Jeffrey B. Burl,et al.  A New Real-Time Optimal Energy Management Strategy for Parallel Hybrid Electric Vehicles , 2019, IEEE Transactions on Control Systems Technology.

[30]  Hongwen He,et al.  Rule-based energy management for dual-source electric buses extracted by wavelet transform , 2018, Journal of Cleaner Production.

[31]  Yuan Zou,et al.  Reinforcement learning-based real-time energy management for a hybrid tracked vehicle , 2016 .

[32]  Yang Zhou,et al.  A survey on driving prediction techniques for predictive energy management of plug-in hybrid electric vehicles , 2019, Journal of Power Sources.

[33]  Alain Bouscayrol,et al.  Real-Time Energy Management of Battery/Supercapacitor Electric Vehicles Based on an Adaptation of Pontryagin's Minimum Principle , 2019, IEEE Transactions on Vehicular Technology.

[34]  Chunhua Zheng,et al.  An Energy Management Strategy of Hybrid Energy Storage Systems for Electric Vehicle Applications , 2018, IEEE Transactions on Sustainable Energy.

[35]  Yuankai Wu,et al.  Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle , 2020 .