Multimodal Trajectory Prediction: A Survey

Trajectory prediction is an important task to support safe and intelligent behaviours in autonomous systems. Many advanced approaches have been proposed over the years with improved spatial and temporal feature extraction. However, human behaviour is naturally multimodal and uncertain: given the past trajectory and surrounding environment information, an agent can have multiple plausible trajectories in the future. To tackle this problem, an essential task named multimodal trajectory prediction (MTP) has recently been studied, which aims to generate a diverse, acceptable and explainable distribution of future predictions for each agent. In this paper, we present the first survey for MTP with our unique taxonomies and comprehensive analysis of frameworks, datasets and evaluation metrics. In addition, we discuss multiple future directions that can help researchers develop novel multimodal trajectory prediction systems.

[1]  Alexandre Alahi,et al.  Safety-Compliant Generative Adversarial Networks for Human Trajectory Forecasting , 2022, IEEE Transactions on Intelligent Transportation Systems.

[2]  Yuanman Li,et al.  STGlow: A Flow-based Generative Framework with Dual Graphormer for Pedestrian Trajectory Prediction , 2022, ArXiv.

[3]  Hao Xue,et al.  PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting , 2022, 2210.08964.

[4]  Pia Bideau,et al.  Action-based Contrastive Learning for Trajectory Prediction , 2022, ECCV.

[5]  Fabio Cuzzolin,et al.  Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey , 2022, IJCAI.

[6]  M. Pagnucco,et al.  Graph-based Spatial Transformer with Memory Replay for Multi-future Pedestrian Trajectory Prediction , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  G. Hua,et al.  Social Interpretable Tree for Pedestrian Trajectory Prediction , 2022, AAAI.

[8]  Jia-Yu Pan,et al.  End-to-End Trajectory Distribution Prediction Based on Occupancy Grid Maps , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Jiwen Lu,et al.  Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Hae-Gon Jeon,et al.  Non-Probability Sampling Network for Stochastic Human Trajectory Prediction , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  J. Hayet,et al.  SocialVAE: Human Trajectory Prediction using Timewise Latents , 2022, ECCV.

[12]  Abduallah A. Mohamed,et al.  Social-Implicit: Rethinking Trajectory Prediction Evaluation and The Effectiveness of Implicit Maximum Likelihood Estimation , 2022, ECCV.

[13]  David J. Crandall,et al.  Stepwise Goal-Driven Networks for Trajectory Prediction , 2021, IEEE Robotics and Automation Letters.

[14]  Richard P. Wildes,et al.  Where are you heading? Dynamic Trajectory Prediction with Expert Goal Examples , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  Yongli Ren,et al.  MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction , 2021, NeurIPS.

[16]  Hang Zhao,et al.  DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Laura Leal-Taixe,et al.  MG-GAN: A Multi-Generator Model Preventing Out-of-Distribution Samples in Pedestrian Trajectory Prediction , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[18]  Jiwen Lu,et al.  Personalized Trajectory Prediction via Distribution Discrimination , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Alexandre Alahi,et al.  Interpretable Social Anchors for Human Trajectory Forecasting in Crowds , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jiquan Ngiam,et al.  Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  G. Hua,et al.  SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Kris Kitani,et al.  AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[23]  Christoph Schöller,et al.  FloMo: Tractable Motion Prediction with Normalizing Flows , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Yang An,et al.  From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Yecheng Jason Ma,et al.  Likelihood-Based Diverse Sampling for Trajectory Forecasting , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[26]  M. Pavone,et al.  Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approach , 2020, IEEE Robotics and Automation Letters.

[27]  Yang Song,et al.  An Improved Discriminator for GAN-Based Trajectory Prediction Models , 2020, 2020 Digital Image Computing: Techniques and Applications (DICTA).

[28]  Laura Leal-Taixé,et al.  Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimation , 2020, ACCV.

[29]  Bernt Schiele,et al.  Haar Wavelet Based Block Autoregressive Flows for Trajectories , 2020, GCPR.

[30]  Yi Shen,et al.  TNT: Target-driveN Trajectory Prediction , 2020, CoRL.

[31]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[32]  Dragomir Anguelov,et al.  VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Henggang Cui,et al.  Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization , 2020, KDD.

[34]  J. Malik,et al.  It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction , 2020, ECCV.

[35]  Abduallah A. Mohamed,et al.  Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  K. Murphy,et al.  The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Freddy A. Boulton,et al.  CoverNet: Multimodal Behavior Prediction Using Trajectory Sets , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Dariu M. Gavrila,et al.  Human motion trajectory prediction: a survey , 2019, Int. J. Robotics Res..

[39]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Vincent Aravantinos,et al.  What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction , 2019, IEEE Robotics and Automation Letters.

[41]  Efstratios Gavves,et al.  SafeCritic: Collision-Aware Trajectory Prediction , 2019, ArXiv.

[42]  Benjamin Sapp,et al.  MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction , 2019, CoRL.

[43]  Zhaoxin Li,et al.  STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Luca Anthony Thiede,et al.  Analyzing the Variety Loss in the Context of Probabilistic Trajectory Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[45]  Silvio Savarese,et al.  Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks , 2019, NeurIPS.

[46]  Simon Lucey,et al.  Argoverse: 3D Tracking and Forecasting With Rich Maps , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Julien Pettré,et al.  Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[48]  Marco Pavone,et al.  The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  Mark Reynolds,et al.  SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[52]  Philip H. S. Torr,et al.  DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Silvio Savarese,et al.  Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes , 2016, ECCV.

[54]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Eike Rehder,et al.  Goal-Directed Pedestrian Prediction , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[56]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[57]  Dani Lischinski,et al.  Crowds by Example , 2007, Comput. Graph. Forum.

[58]  Helbing,et al.  Social force model for pedestrian dynamics. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[59]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[60]  Alexandre Alahi,et al.  Adversarial Loss for Human Trajectory Prediction , 2022 .