Safety-aware Motion Prediction with Unseen Vehicles for Autonomous Driving

Motion prediction of vehicles is critical but challenging due to the uncertainties in complex environments and the limited visibility caused by occlusions and limited sensor ranges. In this paper, we study a new task, safety-aware motion prediction with unseen vehicles for autonomous driving. Unlike the existing trajectory prediction task for seen vehicles, we aim at predicting an occupancy map that indicates the earliest time when each location can be occupied by either seen and unseen vehicles. The ability to predict unseen vehicles is critical for safety in autonomous driving. To tackle this challenging task, we propose a safetyaware deep learning model with three new loss functions to predict the earliest occupancy map. Experiments on the large-scale autonomous driving nuScenes dataset show that our proposed model significantly outperforms the stateof-the-art baselines on the safety-aware motion prediction task. To the best of our knowledge, our approach is the first one that can predict the existence of unseen vehicles in most cases. Project page at https://github.com/ xrenaa/Safety-Aware-Motion-Prediction.

[1]  Alexander Hauptmann,et al.  The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Philip H. S. Torr,et al.  DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Henggang Cui,et al.  Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[4]  Akansel Cosgun,et al.  Towards full automated drive in urban environments: A demonstration in GoMentum Station, California , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[5]  Tao Yang,et al.  Traffic Agent Trajectory Prediction Using Social Convolution and Attention Mechanism , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[6]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Qifeng Chen,et al.  TPCN: Temporal Point Cloud Networks for Motion Forecasting , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Dragomir Anguelov,et al.  VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Haoran Song,et al.  PiP: Planning-informed Trajectory Prediction for Autonomous Driving , 2020, ECCV.

[10]  J. Christian Gerdes,et al.  Safe Driving Envelopes for Shared Control of Ground Vehicles , 2013 .

[11]  Ruslan Salakhutdinov,et al.  Multiple Futures Prediction , 2019, NeurIPS.

[12]  Klaus H. Maier-Hein,et al.  A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities , 2019, ArXiv.

[13]  Benjamin Sapp,et al.  MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction , 2019, CoRL.

[14]  Denis Wolf,et al.  Scene Compliant Trajectory Forecast With Agent-Centric Spatio-Temporal Grids , 2019, IEEE Robotics and Automation Letters.

[15]  R. Urtasun,et al.  Learning Lane Graph Representations for Motion Forecasting , 2020, ECCV.

[16]  Andreas K. Maier,et al.  A 2D dilated residual U-Net for multi-organ segmentation in thoracic CT , 2019, SegTHOR@ISBI.

[17]  Wolfgang Branz,et al.  Uncertainty propagation in criticality measures for driver assistance , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[18]  Sanja Fidler,et al.  Neural Turtle Graphics for Modeling City Road Layouts , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Alberto Elfes,et al.  Using occupancy grids for mobile robot perception and navigation , 1989, Computer.

[20]  Marco Pavone,et al.  Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data , 2020, ECCV.

[21]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[22]  Shaojie Shen,et al.  Safe Trajectory Generation for Complex Urban Environments Using Spatio-Temporal Semantic Corridor , 2019, IEEE Robotics and Automation Letters.

[23]  Masayoshi Tomizuka,et al.  The Convex Feasible Set Algorithm for Real Time Optimization in Motion Planning , 2017, SIAM J. Control. Optim..

[24]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Renjie Liao,et al.  Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction , 2019, CoRL.

[27]  Kris M. Kitani,et al.  Predicting wide receiver trajectories in American football , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[28]  Alexandre Alahi,et al.  Injecting Knowledge in Data-driven Vehicle Trajectory Predictors , 2021, Transportation Research Part C: Emerging Technologies.

[29]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[30]  Thao Dang,et al.  A novel approach for the probabilistic computation of Time-To-Collision , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[31]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[32]  Mykel J. Kochenderfer,et al.  Analysis of Recurrent Neural Networks for Probabilistic Modeling of Driver Behavior , 2017, IEEE Transactions on Intelligent Transportation Systems.

[33]  Helbing,et al.  Social force model for pedestrian dynamics. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[34]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[35]  Klaus Dietmayer,et al.  Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labeling , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[36]  M WangJack,et al.  Gaussian Process Dynamical Models for Human Motion , 2008 .

[37]  Benjamin Sapp,et al.  Rules of the Road: Predicting Driving Behavior With a Convolutional Model of Semantic Interactions , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Joel W. Burdick,et al.  Artificial potential functions for highway driving with collision avoidance , 2008, 2008 IEEE International Conference on Robotics and Automation.

[39]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Marco Pavone,et al.  The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Raquel Urtasun,et al.  DAGMapper: Learning to Map by Discovering Lane Topology , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[42]  Mayank Bansal,et al.  ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst , 2018, Robotics: Science and Systems.

[43]  Wei Zhan,et al.  Probabilistic Prediction from Planning Perspective: Problem Formulation, Representation Simplification and Evaluation Metric , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[44]  Marco Pavone,et al.  A convex optimization approach to smooth trajectories for motion planning with car-like robots , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[45]  Silvio Savarese,et al.  Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks , 2019, NeurIPS.

[46]  Sergio Casas,et al.  Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations , 2020, ECCV.

[47]  Sammy Omari,et al.  One Thousand and One Hours: Self-driving Motion Prediction Dataset , 2020, CoRL.

[48]  Ömer Sahin Tas,et al.  Limited Visibility and Uncertainty Aware Motion Planning for Automated Driving , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[49]  Karl Iagnemma,et al.  A margin-based approach to threat assessment for autonomous highway navigation , 2014, 2014 IEEE Intelligent Vehicles Symposium Proceedings.

[50]  Santokh Singh,et al.  Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey , 2015 .

[51]  Eduard Zaloshnja,et al.  The Economic and Societal Impact of Motor Vehicle Crashes, 2010 (Revised) , 2015 .

[52]  Ying Nian Wu,et al.  Multi-Agent Tensor Fusion for Contextual Trajectory Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[54]  Julius Ziegler,et al.  Making Bertha Drive—An Autonomous Journey on a Historic Route , 2014, IEEE Intelligent Transportation Systems Magazine.

[55]  Martin Lauer,et al.  Safe but not Overcautious Motion Planning under Occlusions and Limited Sensor Range , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[56]  Henggang Cui,et al.  Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving , 2018, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[57]  Alexandre Alahi,et al.  Interpretable Social Anchors for Human Trajectory Forecasting in Crowds , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).