Reinforcement Learning for Hybrid and Plug-In Hybrid Electric Vehicle Energy Management: Recent Advances and Prospects

Energy management is a critical technology in plug-in hybrid-electric vehicles (PHEVs) for maximizing efficiency, fuel economy, and range, as well as reducing pollutant emissions. At the same time, deep reinforcement learning (DRL) has become an effective and important methodology to formulate model-free and realtime energy-management strategies for HEVs and PHEVs. In this article, we describe the energy-management issues of HEVs/PHEVs and summarize a variety of potential DRL applications for onboard energy management. We also discuss the prospects for various DRL approaches in the energy-management field.

[1]  Rajit Johri,et al.  Neuro-Dynamic Programming and Reinforcement Learning for Optimal Energy Management of a Series Hydraulic Hybrid Vehicle Considering Engine Transient Emissions , 2011 .

[2]  H. Vincent Poor,et al.  Cost Minimization of Charging Stations With Photovoltaics: An Approach With EV Classification , 2015, IEEE Transactions on Intelligent Transportation Systems.

[3]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[4]  Naehyuck Chang,et al.  Reinforcement learning based power management for hybrid electric vehicles , 2014, 2014 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[5]  Hai Yu,et al.  Trip-Oriented Energy Management Control Strategy for Plug-In Hybrid Electric Vehicles , 2014, IEEE Transactions on Control Systems Technology.

[6]  Dinh Thai Hoang,et al.  Charging and Discharging of Plug-In Electric Vehicles (PEVs) in Vehicle-to-Grid (V2G) Systems: A Cyber Insurance-Based Model , 2017, IEEE Access.

[7]  Naehyuck Chang,et al.  Machine learning-based energy management in a hybrid electric vehicle to minimize total operating cost , 2015, 2015 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[8]  Yuan Zou,et al.  Reinforcement Learning of Adaptive Energy Management With Transition Probability for a Hybrid Electric Tracked Vehicle , 2015, IEEE Transactions on Industrial Electronics.

[9]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[10]  Guoyuan Wu,et al.  Data-Driven Reinforcement Learning–Based Real-Time Energy Management System for Plug-In Hybrid Electric Vehicles , 2016 .

[11]  Yuan Zou,et al.  Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle , 2015 .

[12]  Richard S. Sutton,et al.  Dyna, an integrated architecture for learning, planning, and reacting , 1990, SGAR.

[13]  Bo Wang,et al.  Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning , 2018, Energy.

[14]  Naehyuck Chang,et al.  Model-free learning-based online management of hybrid electrical energy storage systems in electric vehicles , 2014, IECON 2014 - 40th Annual Conference of the IEEE Industrial Electronics Society.

[15]  Rui Xiong,et al.  Battery and ultracapacitor in-the-loop approach to validate a real-time power management method for an all-climate electric vehicle , 2018 .

[16]  Weihao Hu,et al.  A Heuristic Planning Reinforcement Learning-Based Energy Management for Power-Split Plug-in Hybrid Electric Vehicles , 2019, IEEE Transactions on Industrial Informatics.

[17]  Dongpu Cao,et al.  Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle , 2017, IEEE/ASME Transactions on Mechatronics.

[18]  T. Y. Ji,et al.  Multiple agents and reinforcement learning for modelling charging loads of electric taxis , 2018, Applied Energy.

[19]  Sohrab Asgarpoor,et al.  Reinforcement Learning Approach for Optimal Distributed Energy Management in a Microgrid , 2018, IEEE Transactions on Power Systems.

[20]  C. Gouveia,et al.  Electric Vehicles Charging Management and Control Strategies , 2014, 2014 IEEE Vehicle Power and Propulsion Conference (VPPC).

[21]  Din-Yuen Chan,et al.  A Reinforcement-Learning-Based Assisted Power Management With QoR Provisioning for Human–Electric Hybrid Bicycle , 2012, IEEE Transactions on Industrial Electronics.

[22]  Teng Liu,et al.  A Bi-Level Control for Energy Efficiency Improvement of a Hybrid Tracked Vehicle , 2018, IEEE Transactions on Industrial Informatics.

[23]  H. JoséAntonioMartín,et al.  Dyna-H: A heuristic planning reinforcement learning algorithm applied to role-playing game strategy decision systems , 2011, Knowl. Based Syst..

[24]  Chau Yuen,et al.  Electric Vehicle Charging Station Placement for Urban Public Bus Systems , 2017, IEEE Transactions on Intelligent Transportation Systems.

[25]  Yuan Zou,et al.  Reinforcement learning-based real-time energy management for a hybrid tracked vehicle , 2016 .

[26]  Nolan Wagener,et al.  Information theoretic MPC for model-based reinforcement learning , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Guoyuan Wu,et al.  Deep reinforcement learning-based vehicle energy efficiency autonomous learning system , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[28]  Daniel Hissel,et al.  Fail-Safe Power for Hybrid Electric Vehicles: Implementing a Self-Sustained Global Energy Management System , 2018, IEEE Vehicular Technology Magazine.

[29]  Chang Liu,et al.  Power management for Plug-in Hybrid Electric Vehicles using Reinforcement Learning with trip information , 2014, 2014 IEEE Transportation Electrification Conference and Expo (ITEC).

[30]  Rakesh Gupta,et al.  Improving Hybrid Vehicle Fuel Efficiency Using Inverse Reinforcement Learning , 2012, AAAI.

[31]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[32]  Christian Schroer,et al.  Deep Reinforcement Learning for Advanced Energy Management of Hybrid Electric Vehicles , 2018, ICAART.

[33]  Yechen Qin,et al.  Online Energy Management for Multimode Plug-In Hybrid Electric Vehicles , 2019, IEEE Transactions on Industrial Informatics.

[34]  Chenming Li,et al.  Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning , 2018 .

[35]  Stefano Di Cairano,et al.  Cloud-Based Velocity Profile Optimization for Everyday Driving: A Dynamic-Programming-Based Solution , 2014, IEEE Transactions on Intelligent Transportation Systems.

[36]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[37]  Andreas A. Malikopoulos.  Supervisory Power Management Control Algorithms for Hybrid Electric Vehicles: A Survey , 2014, IEEE Transactions on Intelligent Transportation Systems.

[38]  Chau Yuen,et al.  Electric Vehicle Charge Scheduling Mechanism to Maximize Cost Efficiency and User Convenience , 2018, IEEE Transactions on Smart Grid.

[39]  Simona Onori,et al.  A Comparative Analysis of Energy Management Strategies for Hybrid Electric Vehicles , 2011 .

[40]  H. JoséAntonioMartín,et al.  Robust high performance reinforcement learning through weighted k-nearest neighbors , 2011, Neurocomputing.

[41]  Sergey Levine,et al.  Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[42]  Jiayi Cao,et al.  Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle , 2018 .

[43]  Youxian Sun,et al.  Scheduling of EV Battery Swapping—Part I: Centralized Solution , 2016, IEEE Transactions on Control of Network Systems.

[44]  Chunyue Song,et al.  An energy management strategy for hybrid electric bus based on reinforcement learning , 2015, The 27th Chinese Control and Decision Conference (2015 CCDC).

[45]  Naehyuck Chang,et al.  A deep reinforcement learning framework for optimizing fuel economy of hybrid electric vehicles , 2018, 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC).

[46]  Fengjun Yan,et al.  Hybrid Electric Vehicle Model Predictive Control Torque-Split Strategy Incorporating Engine Transient Characteristics , 2012, IEEE Transactions on Vehicular Technology.

[47]  Chau Yuen,et al.  Balancing Power Demand Through EV Mobility in Vehicle-to-Grid Mobile Energy Networks , 2016, IEEE Transactions on Industrial Informatics.

[48]  Hongwen He,et al.  Power Management for a Plug-in Hybrid Electric Vehicle Based on Reinforcement Learning with Continuous State and Action Spaces , 2017 .

[49]  Alain Bouscayrol,et al.  Energy Management in Fuel-Cell\/Battery Vehicles: Key Issues Identified in the IEEE Vehicular Technology Society Motor Vehicle Challenge 2017 , 2018, IEEE Vehicular Technology Magazine.

[50]  J. Karl Hedrick,et al.  Dynamic Traffic Feedback Data Enabled Energy Management in Plug-in Hybrid Electric Vehicles , 2015, IEEE Transactions on Control Systems Technology.