Energy Management for a Hybrid Electric Vehicle Based on Blended Reinforcement Learning With Backward Focusing and Prioritized Sweeping

As a fundamental task of hybrid electric vehicles (HEVs), energy management strategies are critical for improving the performance. This paper proposes a new queue-Dyna reinforcement learning (RL) algorithm based energy management strategy (EMS), which can substantially reduce the online learning time while guarantees the control performance compared with widely used Q-learning. To solve the existing problems of direct and indirect RL based EMSs, a blended RL algorithm, Dyna, is introduced first. By reusing the actual experience to construct a model online, Dyna integrates direct and indirect RL, and thus has the advantages of both. Furthermore, two novel strategies of backward focusing and prioritized sweeping are incorporated in the Dyna framework, developing the queue-Dyna algorithm. To the best of our knowledge, it's the first attempt of adopting queue-Dyna in the EMS of a HEV. A comparative simulation of direct RL, indirect RL, Dyna and queue-Dyna is implemented, and the results demonstrate the proposed algorithm achieves a great improvement in fast learning and maintains satisfied fuel consumption. At last, a hardware-in-the-loop experiment verified the real-time performance of the proposed EMS.

[1]  Ali Emadi,et al.  Energy Management Systems for Electrified Powertrains: State-of-the-Art Review and Future Trends , 2019, IEEE Transactions on Vehicular Technology.

[2]  Hui Liu,et al.  Real-time optimal energy management strategy for a dual-mode power-split hybrid electric vehicle based on an explicit model predictive control algorithm , 2019, Energy.

[3]  Liang Li,et al.  Temporal-Difference Learning-Based Stochastic Energy Management for Plug-in Hybrid Electric Buses , 2019, IEEE Transactions on Intelligent Transportation Systems.

[4]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[5]  Jing Peng,et al.  Efficient Learning and Planning Within the Dyna Framework , 1993, Adapt. Behav..

[6]  Bin Shuai,et al.  Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle , 2019 .

[7]  Hongwen He,et al.  Deep Reinforcement Learning-Based Energy Management for a Series Hybrid Electric Vehicle Enabled by History Cumulative Trip Information , 2019, IEEE Transactions on Vehicular Technology.

[8]  Guoyuan Wu,et al.  Deep reinforcement learning enabled self-learning control for energy efficient driving , 2019, Transportation Research Part C: Emerging Technologies.

[9]  Yang Zhou,et al.  A survey on driving prediction techniques for predictive energy management of plug-in hybrid electric vehicles , 2019, Journal of Power Sources.

[10]  Hongwen He,et al.  Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus , 2019, Applied Energy.

[11]  Bernard Bäker,et al.  Optimal Energy Management and Velocity Control of Hybrid Electric Vehicles , 2018, IEEE Transactions on Vehicular Technology.

[12]  Yue Ma,et al.  Real-Time Energy Management Strategy Based on Driver-Action-Impact MPC for Series Hybrid Electric Vehicles , 2020, Complex..

[13]  Bo Wang,et al.  Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning , 2018, Energy.

[14]  Simona Onori,et al.  A Comparative Analysis of Energy Management Strategies for Hybrid Electric Vehicles , 2011 .

[15]  Zoran Filipi,et al.  Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle , 2020 .

[16]  Mehdi Karbalaye Zadeh,et al.  An Intelligent Power and Energy Management System for Fuel Cell/Battery Hybrid Electric Vehicle Using Reinforcement Learning , 2019, 2019 IEEE Transportation Electrification Conference and Expo (ITEC).

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Naehyuck Chang,et al.  Reinforcement learning based power management for hybrid electric vehicles , 2014, 2014 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[19]  Dongpu Cao,et al.  Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle , 2017, IEEE/ASME Transactions on Mechatronics.

[20]  Richard S. Sutton,et al.  Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming , 1990, ML.

[21]  Yuan Zou,et al.  Reinforcement Learning of Adaptive Energy Management With Transition Probability for a Hybrid Electric Tracked Vehicle , 2015, IEEE Transactions on Industrial Electronics.

[22]  Richard S. Sutton,et al.  Dyna, an integrated architecture for learning, planning, and reacting , 1990, SGAR.

[23]  Jiayi Cao,et al.  Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle , 2018 .

[24]  Manfred Mitschke,et al.  Dynamik der Kraftfahrzeuge , 1972 .