Uncertainty-aware Energy Management of Extended Range Electric Delivery Vehicles with Bayesian Ensemble

In recent years, deep reinforcement learning (DRL) algorithms have been widely studied and utilized in the area of Intelligent Transportation Systems (ITS). DRL agents are mostly trained with transition pairs and interaction trajectories generated from simulation, and they can achieve satisfying or near optimal performances under familiar input states. However, for relative rare visited or even unvisited regions in the state space, there is no guarantee that the agent could perform well. Unfortunately, novel conditions are inevitable in real-world problems and there is always a gap between the real data and simulated data. Therefore, to implement DRL algorithms in real-world transportation systems, we should not only train the agent learn a policy that maps states to actions, but also the model uncertainty associated with each action. In this study, we adapt the method of Bayesian ensemble to train a group of agents with imposed diversity for an energy management system of a delivery vehicle. The agents in the ensemble agree well on familiar states but show diverse results on unfamiliar or novel states. This uncertainty estimation facilitates the implementation of interpretable postprocessing modules which can ensure robust and safe operations under high uncertainty conditions.

[1]  Marc G. Bellemare,et al.  A Distributional Perspective on Reinforcement Learning , 2017, ICML.

[2]  Teng Liu,et al.  Reinforcement Learning for Hybrid and Plug-In Hybrid Electric Vehicle Energy Management: Recent Advances and Prospects , 2019, IEEE Industrial Electronics Magazine.

[3]  Kunsoo Huh,et al.  Deep Distributional Reinforcement Learning Based High-Level Driving Policy Determination , 2019, IEEE Transactions on Intelligent Vehicles.

[4]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[5]  Hongwen He,et al.  Deep Reinforcement Learning-Based Energy Management for a Series Hybrid Electric Vehicle Enabled by History Cumulative Trip Information , 2019, IEEE Transactions on Vehicular Technology.

[6]  Shashi Shekhar,et al.  Actor-Critic based Deep Reinforcement Learning Framework for Energy Management of Extended Range Electric Delivery Vehicles , 2019, 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[7]  Jonathan P. How,et al.  Safe Reinforcement Learning With Model Uncertainty Estimates , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[8]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[9]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[10]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[11]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[12]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[13]  Benjamin Van Roy,et al.  Ensemble Sampling , 2017, NIPS.

[14]  Alois Knoll,et al.  Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[15]  Bo Gao,et al.  Energy Management in Plug-in Hybrid Electric Vehicles: Recent Progress and a Connected Vehicles Perspective , 2017, IEEE Transactions on Vehicular Technology.

[16]  Marc G. Bellemare,et al.  Distributional Reinforcement Learning with Quantile Regression , 2017, AAAI.

[17]  Shashi Shekhar,et al.  A Deep Reinforcement Learning Framework for Energy Management of Extended Range Electric Delivery Vehicles , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[18]  Shashi Shekhar,et al.  Uncertainty Estimation with Distributional Reinforcement Learning for Applications in Intelligent Transportation Systems: A Case Study , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[19]  Zhu Han,et al.  A Deep Reinforcement Learning Network for Traffic Light Cycle Control , 2018, IEEE Transactions on Vehicular Technology.

[20]  Shangguan Wei,et al.  RA-TSC: Learning Adaptive Traffic Signal Control Strategy via Deep Reinforcement Learning , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[21]  Mohamed Zaki,et al.  Uncertainty in Neural Networks: Bayesian Ensembling , 2018, ArXiv.

[22]  Ian Osband,et al.  Risk versus Uncertainty in Deep Learning: Bayes, Bootstrap and the Dangers of Dropout , 2016 .

[23]  Ali Emadi,et al.  Classification and Review of Control Strategies for Plug-In Hybrid Electric Vehicles , 2011, IEEE Transactions on Vehicular Technology.