Risk-aware Energy Management of Extended Range Electric Delivery Vehicles with Implicit Quantile Network

Model-free reinforcement learning (RL) algorithms are used to solve sequential decision-making problems under uncertainty. They are data-driven methods and do not require an explicit model of the studied system or environment. Because of this characteristic, they are widely utilized in Intelligent Transportation Systems (ITS), as real-world transportation systems are highly complex and extremely difficult to model. However, in most literature, decisions are made according to the expected long-term return estimated by the RL algorithm, ignoring the underlying risk. In this work, a distributional RL algorithm called implicit quantile network is adapted for the energy management problem of a delivery vehicle. Instead of only estimating the expected long-term return, the full return distribution is estimated implicitly. This is highly beneficial for applications in ITS, as uncertainty and randomness are intrinsic characteristics of transportation systems. In addition, risk-aware strategies are integrated into the algorithm with the risk measure of conditional value at risk. In this study, we demonstrate that by changing a hyperparameter, the trade-off between fuel efficiency and the risk of running out of battery power during a delivery trip can be controlled according to different application scenarios and personal preferences.

[1]  Marc G. Bellemare,et al.  Distributional Reinforcement Learning with Quantile Regression , 2017, AAAI.

[2]  Rémi Munos,et al.  Implicit Quantile Networks for Distributional Reinforcement Learning , 2018, ICML.

[3]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[4]  Shashi Shekhar,et al.  Uncertainty Estimation with Distributional Reinforcement Learning for Applications in Intelligent Transportation Systems: A Case Study , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[5]  Bo Gao,et al.  Energy Management in Plug-in Hybrid Electric Vehicles: Recent Progress and a Connected Vehicles Perspective , 2017, IEEE Transactions on Vehicular Technology.

[6]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[7]  Shashi Shekhar,et al.  Actor-Critic based Deep Reinforcement Learning Framework for Energy Management of Extended Range Electric Delivery Vehicles , 2019, 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[8]  Jonathan P. How,et al.  Safe Reinforcement Learning With Model Uncertainty Estimates , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[9]  Ian Osband,et al.  Risk versus Uncertainty in Deep Learning: Bayes, Bootstrap and the Dangers of Dropout , 2016 .

[10]  Guoyuan Wu,et al.  Deep reinforcement learning-based vehicle energy efficiency autonomous learning system , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[11]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[12]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[13]  Liang Li,et al.  Temporal-Difference Learning-Based Stochastic Energy Management for Plug-in Hybrid Electric Buses , 2019, IEEE Transactions on Intelligent Transportation Systems.

[14]  Jonathan P. How,et al.  Decision Making Under Uncertainty: Theory and Application , 2015 .

[15]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[16]  Shashi Shekhar,et al.  A Deep Reinforcement Learning Framework for Energy Management of Extended Range Electric Delivery Vehicles , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[17]  Kunsoo Huh,et al.  Deep Distributional Reinforcement Learning Based High-Level Driving Policy Determination , 2019, IEEE Transactions on Intelligent Vehicles.

[18]  Zhu Han,et al.  A Deep Reinforcement Learning Network for Traffic Light Cycle Control , 2018, IEEE Transactions on Vehicular Technology.

[19]  Shangguan Wei,et al.  RA-TSC: Learning Adaptive Traffic Signal Control Strategy via Deep Reinforcement Learning , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[20]  Marc G. Bellemare,et al.  A Distributional Perspective on Reinforcement Learning , 2017, ICML.

[21]  Marco Pavone,et al.  How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics , 2017, ISRR.

[22]  Mohamed Zaki,et al.  Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning , 2018, 1805.11324.