Deep reinforcement learning enabled self-learning control for energy efficient driving

Abstract To address the air pollution problems and reduce greenhouse gas emissions (GHG), plug-in hybrid electric vehicles (PHEV) have been developed to achieve higher fuel efficiency. The Energy Management System (EMS) is a very important component of a PHEV in achieving better fuel economy and it is a very active research area. So far, most of the existing EMS strategies are either just simply following predefined rules that are not adaptive to changing driving conditions; or heavily relying on accurate prediction of future traffic conditions. Deep learning algorithms have been successfully applied to many complex problems and proved to even outperform human beings in some tasks (e.g., play chess) in recent years, which shows the great potential of such methods in practical engineering problems. In this study, a deep reinforcement learning (Deep Q-network or DQN) based PHEV energy management system is designed to autonomously learn the optimal fuel/electricity splits from interactions between the vehicle and the traffic environment. It is a fully data-driven and self-learning model that does not rely on any prediction, predefined rules or even prior human knowledge. The experiment results show that the proposed model is capable of achieving 16.3% energy savings (with the designed PHEV simulation model) on a typical commute trip, compared to conventional binary control strategies. In addition, a dueling Deep Q-network with dueling structure (DDQN) is also implemented and compared with single DQN in particular with respect to the convergence rate in the training process.

[1]  Guoyuan Wu,et al.  Data-Driven Reinforcement Learning–Based Real-Time Energy Management System for Plug-In Hybrid Electric Vehicles , 2016 .

[2]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[3]  Alain Desrochers,et al.  Fuzzy-based blended control for the energy management of a parallel plug-in hybrid electric vehicle , 2015 .

[4]  Simona Onori,et al.  Analysis of energy management strategies in plug-in hybrid electric vehicles: Application to the GM Chevrolet Volt , 2013, 2013 American Control Conference.

[5]  Hongwen He,et al.  Comparative Study on Different Energy Management Strategies for Plug-In Hybrid Electric Vehicles , 2013 .

[6]  Dongpu Cao,et al.  Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle , 2017, IEEE/ASME Transactions on Mechatronics.

[7]  Yuan Zou,et al.  Implementation of real-time energy management strategy based on reinforcement learning for hybrid electric vehicles and simulation validation , 2017, PloS one.

[8]  Chao Mao,et al.  A reinforcement learning framework for the adaptive routing problem in stochastic time-dependent network , 2018 .

[9]  Guoyuan Wu,et al.  Development and Evaluation of an Evolutionary Algorithm-Based OnLine Energy Management System for Plug-In Hybrid Electric Vehicles , 2017, IEEE Transactions on Intelligent Transportation Systems.

[10]  Naehyuck Chang,et al.  Reinforcement learning based power management for hybrid electric vehicles , 2014, 2014 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[11]  Yuan Zou,et al.  Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle , 2015 .

[12]  Chunyue Song,et al.  An energy management strategy for hybrid electric bus based on reinforcement learning , 2015, The 27th Chinese Control and Decision Conference (2015 CCDC).

[13]  Satish V. Ukkusuri,et al.  A junction-tree based learning algorithm to optimize network wide traffic control: A coordinated multi-agent framework , 2015 .

[14]  Ozgur Baskan,et al.  A modified reinforcement learning algorithm for solving coordinated signalized networks , 2015 .

[15]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[16]  Otman Basir,et al.  Range anxiety reduction in battery-powered vehicles , 2016, 2016 IEEE Transportation Electrification Conference and Expo (ITEC).

[17]  Naehyuck Chang,et al.  Machine learning-based energy management in a hybrid electric vehicle to minimize total operating cost , 2015, 2015 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[18]  Satish V. Ukkusuri,et al.  Accounting for dynamic speed limit control in a stochastic traffic environment: a reinforcement learning approach , 2014 .

[19]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[20]  Zheng Chen,et al.  Energy management of a power-split plug-in hybrid electric vehicle based on genetic algorithm and quadratic programming , 2014 .