Vehicle Trajectory Prediction Using Generative Adversarial Network With Temporal Logic Syntax Tree Features

In this work, we propose a novel approach for integrating rules into traffic agent trajectory prediction. Consideration of rules is important for understanding how people behave—yet, it cannot be assumed that rules are always followed. To address this challenge, we evaluate different approaches of integrating rules as inductive biases into deep learning-based prediction models. We propose a framework based on generative adversarial networks that uses tools from formal methods, namely signal temporal logic and syntax trees. This allows us to leverage information on rule obedience as features in neural networks and improves prediction accuracy without biasing towards lawful behavior. We evaluate our method on a real-world driving dataset and show improvement in performance over off-the-shelf predictors.

[1]  Ufuk Topcu,et al.  Transfer of Temporal Logic Formulas in Reinforcement Learning , 2019, IJCAI.

[2]  Henggang Cui,et al.  Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization , 2020, KDD.

[3]  Masayoshi Tomizuka,et al.  Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network , 2020, ArXiv.

[4]  Mohan M. Trivedi,et al.  Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[5]  Emilio Frazzoli,et al.  Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[6]  Uttaran Bhattacharya,et al.  Forecasting Trajectory and Behavior of Road-Agents Using Spectral Clustering in Graph-LSTMs , 2020, IEEE Robotics and Automation Letters.

[8]  Mayank Bansal,et al.  ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst , 2018, Robotics: Science and Systems.

[9]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Luke Fletcher,et al.  Uncertainty-Aware Driver Trajectory Prediction at Urban Intersections , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[11]  Mohan M. Trivedi,et al.  Convolutional Social Pooling for Vehicle Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Alberto L. Sangiovanni-Vincentelli,et al.  Model predictive control with signal temporal logic specifications , 2014, 53rd IEEE Conference on Decision and Control.

[14]  Marco Pavone,et al.  Backpropagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods , 2020, WAFR.

[15]  Daniel Kroening,et al.  Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[16]  Jyotirmoy V. Deshmukh,et al.  Model-based Reinforcement Learning from Signal Temporal Logic Specifications , 2020, ArXiv.

[17]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[18]  Calin Belta,et al.  Minimum-violation scLTL motion planning for mobility-on-demand , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Marco Pavone,et al.  Trajectron++: Multi-Agent Generative Trajectory Forecasting With Heterogeneous Data for Control , 2020, ArXiv.

[20]  Marco Pavone,et al.  Backpropagation for Parametric STL , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[21]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  M. Chuah,et al.  GRIP++: Enhanced Graph-based Interaction-aware Trajectory Prediction for Autonomous Driving , 2019 .

[23]  Nicholas Roy,et al.  Inferring Task Goals and Constraints using Bayesian Nonparametric Inverse Reinforcement Learning , 2019, CoRL.

[24]  Shaojie Shen,et al.  Online Vehicle Trajectory Prediction using Policy Anticipation Network and optimization-based Context Reasoning , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[25]  Amnon Shashua,et al.  On a Formal Model of Safe and Scalable Self-driving Cars , 2017, ArXiv.

[26]  Philip H. S. Torr,et al.  DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Henggang Cui,et al.  Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[28]  Omar Y. Al-Jarrah,et al.  Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review , 2019, ArXiv.

[29]  Thomas Brox,et al.  Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Nikos Aréchiga,et al.  Specifying Safety of Autonomous Vehicles in Signal Temporal Logic , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[31]  Oded Maler,et al.  Robust Satisfaction of Temporal Logic over Real-Valued Signals , 2010, FORMATS.

[32]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.