Uncertainty Estimation with Distributional Reinforcement Learning for Applications in Intelligent Transportation Systems: A Case Study

Reinforcement learning (RL) algorithms have been successfully used in the area of Intelligent Transportation Systems (ITS) for applications such as energy management strategies (EMS) of hybrid electric vehicles, autonomous driving, traffic light cycle control and bottleneck decongestion. In this work, we investigate a distributional RL algorithm on an EMS problem as a case study to show the benefits of estimating the uncertainty associated with different actions at different states. The uncertainty estimation is highly beneficial to ITS applications as randomness and uncertainty are intrinsic characteristics of real-world problems due to incomplete knowledge of the environment and the stochastic nature of some real-world systems and human behaviors. The modeled uncertainty has the form of a return distribution for taking an action at a certain state. We provide a case study to show that only considering the expected reward value would result in a loss of important information such as the spread. By modeling the return distribution, domain knowledge can be incorporated to make the results more interpretable. Also, risk-sensitive strategies can be developed to build more robust solutions using a chosen utility function.