Learning Accurate and Human-Like Driving using Semantic Maps and Attention

This paper investigates how end-to-end driving models can be improved to drive more accurately and human-like. To tackle the first issue we exploit semantic and visual maps from HERE Technologies and augment the existing Drive360 dataset with such. The maps are used in an attention mechanism that promotes segmentation confidence masks, thus focusing the network on semantic classes in the image that are important for the current driving situation. Human-like driving is achieved using adversarial learning, by not only minimizing the imitation loss with respect to the human driver but by further defining a discriminator, that forces the driving model to produce action sequences that are human-like. Our models are trained and evaluated on the Drive360 + HERE dataset, which features 60 hours and 3000 km of real-world driving data. Extensive experiments show that our driving models are more accurate and behave more human-like than previous methods.

[1]  Luc Van Gool,et al.  Failure Prediction for Autonomous Driving , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[2]  Ronald R. Mourant,et al.  A framework for modeling human-like driving behaviors for autonomous vehicles in driving simulators , 2001, AGENTS '01.

[3]  Philip Koopman,et al.  Challenges in Autonomous Vehicle Testing and Validation , 2016 .

[4]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[6]  Florian Michahelles,et al.  Driving behavior analysis with smartphones: insights from a controlled field study , 2012, MUM.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Bernard Ghanem,et al.  Driving Policy Transfer via Modularity and Abstraction , 2018, CoRL.

[9]  Cewu Lu,et al.  LiDAR-Video Driving Dataset: Learning Driving Policies Effectively , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  William Whittaker,et al.  Autonomous driving in urban environments: Boss and the Urban Challenge , 2008, J. Field Robotics.

[11]  Alexey Dosovitskiy,et al.  End-to-End Driving Via Conditional Imitation Learning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[12]  John H. L. Hansen,et al.  International Large-Scale Vehicle Corpora for Research on Driver Behavior on the Road , 2011, IEEE Transactions on Intelligent Transportation Systems.

[13]  Guangzhong Sun,et al.  Driving with knowledge from the physical world , 2011, KDD.

[14]  Andreas Geiger,et al.  Conditional Affordance Learning for Driving in Urban Environments , 2018, CoRL.

[15]  Jay A. Farrell,et al.  High-precision lane-level road map building for vehicle navigation , 2010, IEEE/ION Position, Location and Navigation Symposium.

[16]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[17]  Vladlen Koltun,et al.  On Offline Evaluation of Vision-based Driving Models , 2018, ECCV.

[18]  Luc Van Gool,et al.  Learning Accurate, Comfortable and Human-like Driving , 2019, ArXiv.

[19]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[20]  E D Dickmanns,et al.  AUTONOMOUS HIGH SPEED ROAD VEHICLE GUIDANCE BY COMPUTER VISION , 1987 .

[21]  Yann LeCun,et al.  Off-Road Obstacle Avoidance through End-to-End Learning , 2005, NIPS.

[22]  Mayank Bansal,et al.  ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst , 2018, Robotics: Science and Systems.

[23]  Li-Ta Hsu,et al.  Human-like motion planning model for driving in signalized intersections , 2017 .

[24]  Wolfram Burgard,et al.  Learning driving styles for autonomous vehicles from demonstration , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[25]  Eshed Ohn-Bar,et al.  Learning Situational Driving , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Andrew V. Goldberg,et al.  Route Planning in Transportation Networks , 2015, Algorithm Engineering.

[27]  Guy Rosman,et al.  Variational End-to-End Navigation and Localization , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[28]  Sanjiv Singh,et al.  The DARPA Urban Challenge: Autonomous Vehicles in City Traffic, George Air Force Base, Victorville, California, USA , 2009, The DARPA Urban Challenge.

[29]  Sergio Casas,et al.  End-To-End Interpretable Neural Motion Planner , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ramesh C. Jain,et al.  GPSView: A scenic driving route planner , 2013, TOMCCAP.

[31]  Alex Bewley,et al.  Learning to Drive from Simulation without Real World Labels , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[32]  Eder Santana,et al.  Exploring the Limitations of Behavior Cloning for Autonomous Driving , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[33]  Georg Maier,et al.  Generation of high precision digital maps using circular arc splines , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[34]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Stefano Ermon,et al.  Generative Adversarial Imitation Learning , 2016, NIPS.

[36]  Etienne Perot,et al.  Deep Reinforcement Learning framework for Autonomous Driving , 2017, Autonomous Vehicles and Machines.

[37]  John F. Canny,et al.  Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  David Janz,et al.  Learning to Drive in a Day , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[39]  Ronald R. Mourant,et al.  Toward More Realistic Driving Behavior Models for Autonomous Vehicles in Driving Simulators , 2003 .

[40]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[41]  Narciso García,et al.  Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[42]  Sanjiv Singh,et al.  The 2005 DARPA Grand Challenge: The Great Robot Race , 2007 .

[43]  Luc Van Gool,et al.  End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners , 2018, ECCV.

[44]  John F. Canny,et al.  Grounding Human-To-Vehicle Advice for Self-Driving Vehicles , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Jiming Chen,et al.  Join driving: A smart phone-based driving behavior evaluation system , 2013, 2013 IEEE Global Communications Conference (GLOBECOM).

[46]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[47]  Jr. G. Forney,et al.  Viterbi Algorithm , 1973, Encyclopedia of Machine Learning.

[48]  Gys Albertus Marthinus Meiring,et al.  A Review of Intelligent Driving Style Analysis Systems and Related Artificial Intelligence Algorithms , 2015, Sensors.

[49]  Shawn D. Newsam,et al.  Improving Semantic Segmentation via Video Propagation and Label Relaxation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Matthias Althoff,et al.  Formalising Traffic Rules for Accountability of Autonomous Vehicles , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[51]  Trevor Darrell,et al.  Deep Object-Centric Policies for Autonomous Driving , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[52]  Chunxiao Liu,et al.  Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks , 2018, AAAI.

[53]  Luc Van Gool,et al.  Learning a Curve Guardian for Motorcycles , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[54]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.