Route Planning and Power Management for PHEVs With Reinforcement Learning

In this article, we propose a novel control scheme for route planning with power management for plug-in hybrid electric vehicles (PHEVs). By considering the power management of PHEVs, we aim to find the route that leads to the minimum energy consumption. The scheme adopts a two-loop structure to achieve the control objective. Specifically, in the outer loop, the minimum energy consumption route is obtained by minimizing the difference between the value function of current round and the best value from all previous rounds. In the inner loop, the energy consumption index with respect to PHEV power management for each feasible route is trained with reinforcement learning (RL). Under the RL framework, a nonlinear approximator structure, which consists of an actor approximator and a critic approximator, is built to approximate control actions and energy consumption. In addition, the convergence of value function for PHEV power management in the inner loop and asymptotical stability of the closed-loop system are rigorously studied. Using Toyota Prius as the vehicle model and planning its route in a city with various navigation ranges, we demonstrate the effectiveness of the proposed control scheme with simulation studies.

[1]  Chao Yang,et al.  Cloud computing-based energy optimization control framework for plug-in hybrid electric bus , 2017 .

[2]  Simona Onori,et al.  Optimal energy management of hybrid electric vehicles including battery aging , 2011, Proceedings of the 2011 American Control Conference.

[3]  Alberto Bemporad,et al.  Stochastic MPC With Learning for Driver-Predictive Vehicle Control and its Application to HEV Energy Management , 2014, IEEE Transactions on Control Systems Technology.

[4]  Hai Yu,et al.  Trip-Oriented Energy Management Control Strategy for Plug-In Hybrid Electric Vehicles , 2014, IEEE Transactions on Control Systems Technology.

[5]  Huei Peng,et al.  Optimal Control of Hybrid Electric Vehicles Based on Pontryagin's Minimum Principle , 2011, IEEE Transactions on Control Systems Technology.

[6]  Hao Xu,et al.  Neural Network-Based Finite-Horizon Optimal Control of Uncertain Affine Nonlinear Discrete-Time Systems , 2015, IEEE Trans. Neural Networks Learn. Syst..

[7]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[8]  Florence Ossart,et al.  Comparative Study of Real-Time HEV Energy Management Strategies , 2017, IEEE Transactions on Vehicular Technology.

[9]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[10]  Naehyuck Chang,et al.  Reinforcement learning based power management for hybrid electric vehicles , 2014, 2014 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[11]  J.M. Miller,et al.  Hybrid electric vehicle propulsion system architectures of the e-CVT type , 2006, IEEE Transactions on Power Electronics.

[12]  Xiangyu Wang,et al.  Dual-Loop Self-Learning Fuzzy Control for AMT Gear Engagement: Design and Experiment , 2018, IEEE Transactions on Fuzzy Systems.

[13]  Stefan Fritsch,et al.  neuralnet: Training of Neural Networks , 2010, R J..

[14]  Neil Genzlinger A. and Q , 2006 .

[15]  W. Marsden I and J , 2012 .

[16]  Victor M. Becerra,et al.  Optimal control , 2008, Scholarpedia.

[17]  Jianqiu Li,et al.  Real time optimal energy management strategy targeting at minimizing daily operation cost for a plug-in fuel cell city bus , 2012 .

[18]  Chang Liu,et al.  Power management for Plug-in Hybrid Electric Vehicles using Reinforcement Learning with trip information , 2014, 2014 IEEE Transportation Electrification Conference and Expo (ITEC).

[19]  Quan An,et al.  Modelling and control for economy‐oriented car‐following problem of hybrid electric vehicle , 2019, IET Intelligent Transport Systems.

[20]  Qing Wang,et al.  Intelligent Hybrid Vehicle Power Control—Part II: Online Intelligent Energy Management , 2013, IEEE Transactions on Vehicular Technology.

[21]  Jun-ichi Imura,et al.  Road-Speed Profile for Enhanced Perception of Traffic Conditions in a Partially Connected Vehicle Environment , 2018, IEEE Transactions on Vehicular Technology.

[22]  Huei Peng,et al.  Power management strategy for a parallel hybrid electric truck , 2003, IEEE Trans. Control. Syst. Technol..

[23]  Reza Langari,et al.  Intelligent energy management agent for a parallel hybrid vehicle-part II: torque distribution, charge sustenance strategies, and performance results , 2005, IEEE Transactions on Vehicular Technology.

[24]  Xiangyu Wang,et al.  Hierarchical Control of Dry Clutch for Engine-Start Process in a Parallel Hybrid Electric Vehicle , 2016, IEEE Transactions on Transportation Electrification.

[25]  Fengjun Yan,et al.  Hybrid Electric Vehicle Model Predictive Control Torque-Split Strategy Incorporating Engine Transient Characteristics , 2012, IEEE Transactions on Vehicular Technology.

[26]  Lino Guzzella,et al.  Optimal control of parallel hybrid electric vehicles , 2004, IEEE Transactions on Control Systems Technology.

[27]  M. Ouyang,et al.  Approximate Pontryagin’s minimum principle applied to the energy management of plug-in hybrid electric vehicles , 2014 .

[28]  Maciej Wieczorek,et al.  A mathematical representation of an energy management strategy for hybrid energy storage system in electric vehicle and real time optimization using a genetic algorithm , 2017 .

[29]  Marco Sorrentino,et al.  Analysis of a rule-based control strategy for on-board energy management of series hybrid vehicles , 2011 .

[30]  Xinyu Yang,et al.  A Real-Time En-Route Route Guidance Decision Scheme for Transportation-Based Cyberphysical Systems , 2017, IEEE Transactions on Vehicular Technology.