Modeling Human Driving Behavior in Highway Scenario using Inverse Reinforcement Learning

Human driving behavior modeling is of great importance for designing safe, smart, smooth as well as personalized autonomous driving systems. In this paper, an internal reward function-based driving model that emulates the human's internal decision-making mechanism is proposed. Besides, a sampling-based inverse reinforcement learning (IRL) algorithm that learns the reward function from human naturalistic driving data is also developed. A polynomial trajectory sampler is adopted to generate feasible trajectories and approximate the partition function in the maximum entropy IRL framework, and a dynamic and interactive environment is built upon the static driving dataset to estimate the generated trajectories considering the mutual dependency of agents' actions. The proposed method is applied to learn personalized reward functions for individual human drivers from the NGSIM dataset. The qualitative results demonstrate that the learned reward function is able to interpret their decisions. The quantitative results also reveal that the personalized modeling method significantly outperforms the general modeling approach, reducing the errors in human likeness by 24%, and the proposed reward-function-based method delivers better results compared to other baseline methods. Moreover, it is found that estimating the response actions of surrounding vehicles plays an integral role in estimating the trajectory accurately and achieving a better generalization ability.

[1]  Ying Nian Wu,et al.  Energy-Based Continuous Inverse Optimal Control , 2019 .

[2]  Huijing Zhao,et al.  A Human-like Trajectory Planning Method by Learning from Naturalistic Driving Data , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[3]  Stefan Roth,et al.  Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Driving , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[4]  Anca D. Dragan,et al.  Courteous Autonomous Cars , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[5]  Malte Risto,et al.  The social behavior of autonomous vehicles , 2016, UbiComp Adjunct.

[6]  Chen Lv,et al.  Interaction-Aware Trajectory Prediction of Connected Vehicles using CNN-LSTM Networks , 2020, IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society.

[7]  Alois Knoll,et al.  Graph Neural Networks for Modelling Traffic Participant Interaction , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[8]  Masayoshi Tomizuka,et al.  Expressing Diverse Human Driving Behavior with Probabilistic Rewards and Online Inference , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Sergey Levine,et al.  Continuous Inverse Optimal Control with Locally Optimal Examples , 2012, ICML.

[10]  Christian Laugier,et al.  Modeling Driver Behavior from Demonstrations in Dynamic Environments Using Spatiotemporal Lattices , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Dirk Helbing,et al.  General Lane-Changing Model MOBIL for Car-Following Models , 2007 .

[12]  Christian Laugier,et al.  High-speed highway scene prediction based on driver models learned from demonstrations , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[13]  Wei Zhan,et al.  Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[14]  Wolfram Burgard,et al.  Learning driving styles for autonomous vehicles from demonstration , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[15]  Dongpu Cao,et al.  Personalized Vehicle Trajectory Prediction Based on Joint Time-Series Modeling for Connected Vehicles , 2020, IEEE Transactions on Vehicular Technology.

[16]  Anind K. Dey,et al.  Maximum Entropy Inverse Reinforcement Learning , 2008, AAAI.

[17]  Mykel J. Kochenderfer,et al.  Modeling and Prediction of Human Driver Behavior: A Survey , 2020, ArXiv.

[18]  Alexiadis,et al.  The next generation simulation program , 2004 .

[19]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[20]  Masayoshi Tomizuka,et al.  Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning With Application to Autonomous Driving , 2020, IEEE Robotics and Automation Letters.

[21]  Mukesh Singhal,et al.  Do You Want Your Autonomous Car to Drive Like You? , 2015, 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI.

[22]  Wei Zhan,et al.  Interpretable Modelling of Driving Behaviors in Interactive Driving Scenarios based on Cumulative Prospect Theory , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[23]  Mianxiong Dong,et al.  Humanlike Driving: Empirical Decision-Making System for Autonomous Vehicles , 2018, IEEE Transactions on Vehicular Technology.

[24]  Masayoshi Tomizuka,et al.  Generic Prediction Architecture Considering both Rational and Irrational Driving Behaviors , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).