Intention-aware Residual Bidirectional LSTM for Long-term Pedestrian Trajectory Prediction

Trajectory prediction is one of the key capabilities for robots to safely navigate and interact with pedestrians. Critical insights from human intention and behavioral patterns need to be effectively integrated into long-term pedestrian behavior forecasting. We present a novel intention-aware motion prediction framework, which consists of a Residual Bidirectional LSTM (ReBiL) and a mutable intention filter. Instead of learning step-wise displacement, we propose learning offset to warp a nominal intention-aware linear prediction, giving residual learning a physical intuition. Our intention filter is inspired by genetic algorithms and particle filtering, where particles mutate intention hypotheses throughout the pedestrian motion with ReBiL as the motion model. Through experiments on a publicly available dataset, we show that our method outperforms baseline approaches and the robust performance of our method is demonstrated under abnormal intention-changing scenarios.

[1]  Oladimeji Farri,et al.  Neural Paraphrase Generation with Stacked Residual LSTM Networks , 2016, COLING.

[2]  Fei Tian,et al.  Recurrent Residual Learning for Sequence Classification , 2016, EMNLP.

[3]  Roland Siegwart,et al.  A data-driven approach for pedestrian intention estimation , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[4]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Ke Xu,et al.  Online Monitoring for Safe Pedestrian-Vehicle Interactions , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[6]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jean Oh,et al.  Social Attention: Modeling Attention in Human Crowds , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[8]  Emilio Frazzoli,et al.  Intention-Aware Motion Planning , 2013, WAFR.

[9]  Nanning Zheng,et al.  SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  R. Blake,et al.  Perception of human motion. , 2007, Annual review of psychology.

[11]  Xuejie Zhang,et al.  Using a stacked residual LSTM model for sentiment intensity prediction , 2018, Neurocomputing.

[12]  Dariu Gavrila,et al.  UvA-DARE ( Digital Academic Repository ) Pedestrian Path Prediction with Recursive Bayesian Filters : A Comparative Study , 2013 .

[13]  Siddhartha S. Srinivasa,et al.  Planning-based prediction for pedestrians , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Dinesh Manocha,et al.  Reciprocal n-Body Collision Avoidance , 2011, ISRR.

[16]  Alberto Del Bimbo,et al.  Context-Aware Trajectory Prediction , 2017, 2018 24th International Conference on Pattern Recognition (ICPR).

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Dariu M. Gavrila,et al.  Human motion trajectory prediction: a survey , 2019, Int. J. Robotics Res..

[19]  Siddhartha S. Srinivasa,et al.  Legibility and predictability of robot motion , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[20]  Hanli Wang,et al.  Image captioning with deep LSTM based on sequential residual , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[21]  Antonio Jimeno-Yepes,et al.  Named Entity Recognition with Stack Residual LSTM and Trainable Bias Decoding , 2017, IJCNLP.

[22]  Marco Pavone,et al.  The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[23]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[24]  Barbara Majecka,et al.  Statistical models of pedestrian behaviour in the Forum , 2009 .

[25]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Rong Chen,et al.  A Theoretical Framework for Sequential Importance Sampling with Resampling , 2001, Sequential Monte Carlo Methods in Practice.

[27]  Markus Hiller,et al.  Improvements in pedestrian movement prediction by considering multiple intentions in a Multi-Hypotheses filter , 2018, 2018 IEEE/ION Position, Location and Navigation Symposium (PLANS).

[28]  Markus Hiller,et al.  Entropy-Based Intention Change Detection with a Multi-Hypotheses Filter , 2018, 2018 21st International Conference on Information Fusion (FUSION).

[29]  Jungwon Lee,et al.  Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition , 2017, INTERSPEECH.

[30]  Eike Rehder,et al.  Goal-Directed Pedestrian Prediction , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[31]  Björn Ommer,et al.  Learning to Forecast Pedestrian Intention from Pose Dynamics , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[32]  Gregory D. Hager,et al.  Intent-Aware Pedestrian Prediction for Adaptive Crowd Navigation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Mark Reynolds,et al.  Bi-Prediction: Pedestrian Trajectory Prediction Based on Bidirectional LSTM Classification , 2017, 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[34]  Marco Pavone,et al.  Trajectron++: Multi-Agent Generative Trajectory Forecasting With Heterogeneous Data for Control , 2020, ArXiv.

[35]  Stefan Becker,et al.  RED: A Simple but Effective Baseline Predictor for the TrajNet Benchmark , 2018, ECCV Workshops.

[36]  Jodie A. Baird,et al.  Discerning intentions in dynamic human action , 2001, Trends in Cognitive Sciences.

[37]  Wolfram Burgard,et al.  Feature-Based Prediction of Trajectories for Socially Compliant Navigation , 2012, Robotics: Science and Systems.

[38]  Rae-Hong Park,et al.  Residual LSTM Attention Network for Object Tracking , 2018, IEEE Signal Processing Letters.

[39]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[40]  Sridha Sridharan,et al.  Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection , 2017, Neural Networks.

[41]  Yu Zhao,et al.  Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors , 2017, Mathematical Problems in Engineering.

[42]  John K. Tsotsos,et al.  PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[43]  Marco Pavone,et al.  Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data , 2020, ECCV.

[44]  Shenghua Gao,et al.  Encoding Crowd Interaction with Deep Neural Network for Pedestrian Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Helbing,et al.  Social force model for pedestrian dynamics. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[46]  Jonathan P. How,et al.  Real-Time Predictive Modeling and Robust Avoidance of Pedestrians with Uncertain, Changing Intentions , 2014, WAFR.

[47]  Rainer Stiefelhagen,et al.  A Controlled Interactive Multiple Model Filter for Combined Pedestrian Intention Recognition and Path Prediction , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[48]  Danica Kragic,et al.  Long-term Prediction of Motion Trajectories Using Path Homology Clusters , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[49]  Yi Yang,et al.  An improved residual LSTM architecture for acoustic modeling , 2017, 2017 2nd International Conference on Computer and Communication Systems (ICCCS).

[50]  Luc Van Gool,et al.  You'll never walk alone: Modeling social behavior for multi-target tracking , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[51]  Stefano Soatto,et al.  Intent-aware long-term prediction of pedestrian motion , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[52]  Luis E. Ortiz,et al.  Who are you with and where are you going? , 2011, CVPR 2011.

[53]  Martial Hebert,et al.  Activity Forecasting , 2012, ECCV.