Learning Driver-Specific Behavior for Overtaking: A Combined Learning Framework

Learning-based methods have gained increasing attention in the intelligent vehicle community for developing highly autonomous vehicles and advanced driving assistance systems (ADAS). However, traditional offline learning methods lack the ability to adapt to individual driving behavior. To overcome this limitation, a combined learning framework (CLF) based on the Natural Actor Critic (NAC) learning and general regression neural network (GRNN) is developed in this paper. GRNN can be trained offline based on the historical data, while NAC is carried out online. In this way, the general behavior learned by the offline module can be reused and adjusted by the online module to capture the driver-specific behavior. Driving data collected from human drivers through a driving simulator are used to test the proposed learning framework. The complex overtaking behavior is selected to formulate the learning problem and test scenarios. Experimental results show that the proposed system performs well on learning driver-specific behavior for overtaking, and compared with the Gaussian mixture model-maximum-a-posterior method, CLF shows a more flexible performance when newly-involved drivers are considered.

[1]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[2]  Yeung Yam,et al.  Performance evaluation and optimization of human control strategy , 2002, Robotics Auton. Syst..

[3]  José Eugenio Naranjo,et al.  Lane-Change Fuzzy Control in Autonomous Vehicles for the Overtaking Maneuver , 2008, IEEE Transactions on Intelligent Transportation Systems.

[4]  Junmin Wang,et al.  A Driver Steering Model With Personalized Desired Path Generation , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[5]  Junmin Wang,et al.  A Personalizable Driver Steering Model Capable of Predicting Driver Behaviors in Vehicle Collision Avoidance Maneuvers , 2017, IEEE Transactions on Human-Machine Systems.

[6]  Christopher M. Clark,et al.  Reinforcement learning of dynamic collaborative driving Part II: lateral adaptive control , 2008 .

[7]  Yangsheng Xu,et al.  Human control strategy: abstraction, verification, and replication , 1997 .

[8]  Yuan Zou,et al.  A Real-Time Markov Chain Driver Model for Tracked Vehicles and Its Validation: Its Adaptability via Stochastic Dynamic Programming , 2017, IEEE Transactions on Vehicular Technology.

[9]  Charles Desjardins,et al.  Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach , 2011, IEEE Transactions on Intelligent Transportation Systems.

[10]  Yoshio Kano,et al.  A study on adaptation of SBW parameters to individual driver’s steer characteristics for improved driver–vehicle system performance , 2006 .

[11]  N. H. C. Yung,et al.  A Multiple-Goal Reinforcement Learning Method for Complex Vehicle Overtaking Maneuvers , 2011, IEEE Transactions on Intelligent Transportation Systems.

[12]  Gaetano Fusco,et al.  Artificial Neural Network Models for Car Following: Experimental Analysis and Calibration Issues , 2014, J. Intell. Transp. Syst..

[13]  Ali Ghaffari,et al.  A Modified Car-Following Model Based on a Neural Network Model of the Human Driver Effects , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[14]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Petros A. Ioannou,et al.  Personalized Driver/Vehicle Lane Change Models for ADAS , 2015, IEEE Transactions on Vehicular Technology.

[16]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[19]  Michel Verhaegen,et al.  Development of advanced driver assistance systems with vehicle hardware-in-the-loop simulations , 2006 .

[20]  Vicente Milanés Montero,et al.  Intelligent automatic overtaking system using vision for vehicle detection , 2012, Expert Syst. Appl..

[21]  Robert Babuska,et al.  A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[22]  Kyongsu Yi,et al.  Integrated Speed and Steering Control Driver Model for Vehicle–Driver Closed-Loop Simulation , 2016, IEEE Transactions on Vehicular Technology.

[23]  陈虹,et al.  Switching-Based Stochastic Model Predictive Control Approach for Modeling Driver Steering Skill , 2015 .

[24]  Tatsuya Suzuki,et al.  Modeling and Analysis of Driving Behavior Based on a Probability-Weighted ARX Model , 2013, IEEE Transactions on Intelligent Transportation Systems.

[25]  Luke Ng,et al.  Reinforcement Learning of Dynamic Collaborative Driving , 2008 .

[26]  Francesco Borrelli,et al.  A Learning-Based Framework for Velocity Control in Autonomous Driving , 2016, IEEE Transactions on Automation Science and Engineering.

[27]  Huei Peng,et al.  An adaptive lateral preview driver model , 2005 .

[28]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[29]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[30]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[31]  Aude Billard,et al.  Reinforcement learning for imitating constrained reaching movements , 2007, Adv. Robotics.

[32]  Motohiro Fujita,et al.  Car-following behavior with instantaneous driver–vehicle reaction delay: A neural-network-based methodology , 2013 .

[33]  Xin Li,et al.  Reinforcement learning based overtaking decision-making for highway autonomous driving , 2015, 2015 Sixth International Conference on Intelligent Control and Information Processing (ICICIP).

[34]  Tatsuya Suzuki,et al.  Modeling and Recognition of Driving Behavior Based on Stochastic Switched ARX Model , 2007, IEEE Transactions on Intelligent Transportation Systems.

[35]  Michael L. Littman,et al.  Reinforcement learning improves behaviour from evaluative feedback , 2015, Nature.

[36]  Kazuya Takeda,et al.  Modeling and adaptation of stochastic driver-behavior model with application to car following , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[37]  Mykel J. Kochenderfer,et al.  Analysis of Recurrent Neural Networks for Probabilistic Modeling of Driver Behavior , 2017, IEEE Transactions on Intelligent Transportation Systems.

[38]  Francesco Borrelli,et al.  Driver models for personalised driving assistance , 2015 .

[39]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.