A Review of Deep Learning-Based Methods for Pedestrian Trajectory Prediction

Pedestrian trajectory prediction is one of the main concerns of computer vision problems in the automotive industry, especially in the field of advanced driver assistance systems. The ability to anticipate the next movements of pedestrians on the street is a key task in many areas, e.g., self-driving auto vehicles, mobile robots or advanced surveillance systems, and they still represent a technological challenge. The performance of state-of-the-art pedestrian trajectory prediction methods currently benefits from the advancements in sensors and associated signal processing technologies. The current paper reviews the most recent deep learning-based solutions for the problem of pedestrian trajectory prediction along with employed sensors and afferent processing methodologies, and it performs an overview of the available datasets, performance metrics used in the evaluation process, and practical applications. Finally, the current work exposes the research gaps from the literature and outlines potential new research directions.

[1]  Wang Jun,et al.  LIDAR and vision based pedestrian detection and tracking system , 2015, 2015 IEEE International Conference on Progress in Informatics and Computing (PIC).

[2]  Chunshi Guo,et al.  Cooperation between driver and automated driving system: Implementation and evaluation , 2017, Transportation Research Part F: Traffic Psychology and Behaviour.

[3]  Daniel Cremers,et al.  MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking , 2020, International Journal of Computer Vision.

[4]  Dariu Gavrila,et al.  UvA-DARE ( Digital Academic Repository ) Pedestrian Path Prediction with Recursive Bayesian Filters : A Comparative Study , 2013 .

[5]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[6]  Jianwei Zhang,et al.  Learning motion field of LiDAR point cloud with convolutional networks , 2019, Pattern Recognit. Lett..

[7]  Mark Reynolds,et al.  SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[8]  A. Bimbo,et al.  MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Marcin Detyniecki,et al.  Vehicle Telematics Via Exteroceptive Sensors: A Survey , 2020, ArXiv.

[10]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[11]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Yingnan Sun,et al.  Indoor Future Person Localization from an Egocentric Wearable Camera , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Mark Elshaw,et al.  Pedestrian and Cyclist Detection and Intent Estimation for Autonomous Vehicles: A Survey , 2019, Applied Sciences.

[14]  Thia Kirubarajan,et al.  Estimation with Applications to Tracking and Navigation: Theory, Algorithms and Software , 2001 .

[15]  Paulo Cortez,et al.  Automatic human trajectory destination prediction from video , 2018, Expert Syst. Appl..

[16]  H. Wechsler,et al.  Micro-Doppler effect in radar: phenomenon, model, and simulation study , 2006, IEEE Transactions on Aerospace and Electronic Systems.

[17]  Razvan Pascanu,et al.  Theano: new features and speed improvements , 2012, ArXiv.

[18]  Ming Liu,et al.  See the Future: A Semantic Segmentation Network Predicting Ego-Vehicle Trajectory With a Single Monocular Camera , 2020, IEEE Robotics and Automation Letters.

[19]  Tian Xia,et al.  Vehicle Detection from 3D Lidar Using Fully Convolutional Network , 2016, Robotics: Science and Systems.

[20]  Shenghua Gao,et al.  Encoding Crowd Interaction with Deep Neural Network for Pedestrian Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  M. J. Gilmartin INTRODUCTION TO AUTONOMOUS MOBILE ROBOTS, by Roland Siegwart and Illah R. Nourbakhsh, MIT Press, 2004, xiii+321 pp., ISBN 0-262-19502-X. (Hardback, £27.95) , 2005 .

[22]  Wolfgang Hübner,et al.  Particle-based Pedestrian Path Prediction using LSTM-MDL Models , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[23]  Roland Siegwart,et al.  A data-driven approach for pedestrian intention estimation , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[24]  Martin Lauer,et al.  A Literature Review on the Prediction of Pedestrian Behavior in Urban Scenarios , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[25]  Ruigang Yang,et al.  The ApolloScape Open Dataset for Autonomous Driving and Its Application , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Karl Granström,et al.  Pedestrian tracking using Velodyne data — Stochastic optimization for extended object tracking , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[27]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  K. Kulpa,et al.  The CLEAN type algorithms for radar signal processing , 2008, 2008 Microwaves, Radar and Remote Sensing Symposium.

[29]  Amir Khajepour,et al.  Real-time Pedestrian Localization and State Estimation Using Moving Horizon Estimation , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[30]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Silvio Savarese,et al.  Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes , 2016, ECCV.

[32]  Luc Van Gool,et al.  You'll never walk alone: Modeling social behavior for multi-target tracking , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[33]  Michael J. Black,et al.  On Human Motion Prediction Using Recurrent Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Julius Ziegler,et al.  Making Bertha Drive—An Autonomous Journey on a Historic Route , 2014, IEEE Intelligent Transportation Systems Magazine.

[35]  Martin Lauer,et al.  Pedestrian Prediction by Planning Using Deep Neural Networks , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[36]  Matthew Johnson-Roberson,et al.  Pedestrian Planar LiDAR Pose (PPLP) Network for Oriented Pedestrian Detection Based on Planar LiDAR and Monocular Images , 2020, IEEE Robotics and Automation Letters.

[37]  Jingtian Tang,et al.  Micro-Doppler Trajectory Estimation of Pedestrians Using a Continuous-Wave Radar , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Julien Pettré,et al.  Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Thomas Winkle,et al.  Safety Benefits of Automated Vehicles: Extended Findings from Accident Research for Development, Validation and Testing , 2016 .

[40]  Zhi Yan,et al.  Online learning for human classification in 3D LiDAR-based tracking , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[41]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[42]  Pieter Abbeel,et al.  InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , 2016, NIPS.

[43]  Dani Lischinski,et al.  Crowds by Example , 2007, Comput. Graph. Forum.

[44]  Chao Lu,et al.  Prediction of Pedestrian Risky Level for Intelligent Vehicles , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[45]  Jerry L. Eaves,et al.  Principles of Modern Radar , 1987 .

[46]  Yves Grandvalet,et al.  Driving among Flatmobiles: Bird-Eye-View occupancy grids from a monocular camera for holistic trajectory planning , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[47]  Yisong Yue,et al.  Generative Multi-Agent Behavioral Cloning , 2018, ArXiv.

[48]  Tanaya Guha,et al.  Multi-Camera Trajectory Forecasting: Pedestrian Trajectory Prediction in a Network of Cameras , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[49]  Thiago Oliveira-Santos,et al.  Handling Pedestrians in Crosswalks Using Deep Neural Networks in the IARA Autonomous Car , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[50]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Jaehoon Jung,et al.  Efficient and robust lane marking extraction from mobile lidar point clouds , 2019 .

[52]  Zhi Yan,et al.  3DOF Pedestrian Trajectory Prediction Learned from Long-Term Autonomous Mobile Robot Deployment Data , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[53]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Hermann Rohling,et al.  Radar CFAR Thresholding in Clutter and Multiple Target Situations , 1983, IEEE Transactions on Aerospace and Electronic Systems.

[55]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[56]  Holger H. Meinel,et al.  Evolving automotive radar — From the very beginnings into the future , 2014, The 8th European Conference on Antennas and Propagation (EuCAP 2014).

[57]  Zhiheng Li,et al.  Modeling Vehicle Interactions via Modified LSTM Models for Trajectory Prediction , 2019, IEEE Access.

[58]  Jonathan G. Fiscus,et al.  TRECVID 2018: Benchmarking Video Activity Detection, Video Captioning and Matching, Video Storytelling Linking and Video Search , 2018, TRECVID.

[59]  Bernt Schiele,et al.  Long-Term On-board Prediction of People in Traffic Scenes Under Uncertainty , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Yi Lu Murphey,et al.  Camera performance considerations for automotive applications , 2004, SPIE Optics East.

[61]  Klaus C. J. Dietmayer,et al.  The Ko-PER intersection laserscanner and video dataset , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[62]  John K. Tsotsos,et al.  Autonomous Vehicles That Interact With Pedestrians: A Survey of Theory and Practice , 2018, IEEE Transactions on Intelligent Transportation Systems.

[63]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Michael Meyer,et al.  Deep Learning Based 3D Object Detection for Automotive Radar and Camera , 2019, 2019 16th European Radar Conference (EuRAD).

[65]  Matthias Althoff,et al.  Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior , 2020, IEEE Transactions on Intelligent Transportation Systems.

[66]  Zhihai He,et al.  Pedestrian Motion Trajectory Prediction With Stereo-Based 3D Deep Pose Estimation and Trajectory Learning , 2020, IEEE Access.

[67]  Silvio Savarese,et al.  Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks , 2019, NeurIPS.

[68]  Cristiano Premebida,et al.  CNN-LIDAR pedestrian classification: combining range and reflectance data , 2018, 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES).

[69]  Yingfeng Cai,et al.  A 64-Line Lidar-Based Road Obstacle Sensing Algorithm for Intelligent Vehicles , 2018, Sci. Program..

[70]  Marco Pavone,et al.  Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data , 2020, ECCV.

[71]  Yoichi Sato,et al.  Future Person Localization in First-Person Videos , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[72]  Tarak Gandhi,et al.  Pedestrian Protection Systems: Issues, Survey, and Challenges , 2007, IEEE Transactions on Intelligent Transportation Systems.

[73]  Wolfram Burgard,et al.  Predicting Occupancy Distributions of Walking Humans With Convolutional Neural Networks , 2018, IEEE Robotics and Automation Letters.

[74]  Heng Wang,et al.  Robotics and Autonomous Systems , 2022 .

[75]  Xiao Song,et al.  Multi-information-based convolutional neural network with attention mechanism for pedestrian trajectory prediction , 2021, Image Vis. Comput..

[76]  Avik Santra,et al.  A Bayesian Framework for Integrated Deep Metric Learning and Tracking of Vulnerable Road Users Using Automotive Radars , 2021, IEEE Access.

[77]  Lutz Eckstein,et al.  The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections , 2019, 2020 IEEE Intelligent Vehicles Symposium (IV).

[78]  In So Kweon,et al.  KAIST Multi-Spectral Day/Night Data Set for Autonomous and Assisted Driving , 2018, IEEE Transactions on Intelligent Transportation Systems.

[79]  Matthias Althoff,et al.  Pedestrian Models for Autonomous Driving Part I: Low-Level Models, From Sensing to Tracking , 2020, IEEE Transactions on Intelligent Transportation Systems.

[80]  Dragomir Anguelov,et al.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  David Johnson,et al.  Radar Sensing for Intelligent Vehicles in Urban Environments , 2015, Sensors.

[82]  Günther Palm,et al.  Surround view pedestrian detection using heterogeneous classifier cascades , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[83]  Larry S. Davis,et al.  AVSS 2011 demo session: A large-scale benchmark dataset for event recognition in surveillance video , 2011, AVSS.

[84]  Abduallah A. Mohamed,et al.  Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[85]  Sami Terho,et al.  Radar based detection and tracking of a walking human , 2010 .

[86]  Alexandre Alahi,et al.  Human Trajectory Prediction using Adversarial Loss , 2019 .

[87]  Jochen Seitz,et al.  Vehicle-to-Pedestrian Communication for Vulnerable Road Users: Survey, Design Considerations, and Challenges , 2019, Sensors.

[88]  Ying Nian Wu,et al.  Multi-Agent Tensor Fusion for Contextual Trajectory Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[89]  M.C. Wicks,et al.  Space-time adaptive processing: a knowledge-based perspective for airborne radar , 2006, IEEE Signal Processing Magazine.

[90]  Klaus Dietmayer,et al.  Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labeling , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[91]  Jurgen Hasch,et al.  Driving towards 2020: Automotive radar technology trends , 2015, 2015 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM).

[92]  C. G. Keller,et al.  Will the Pedestrian Cross? A Study on Pedestrian Path Prediction , 2014, IEEE Transactions on Intelligent Transportation Systems.

[93]  Qiang Xu,et al.  nuScenes: A Multimodal Dataset for Autonomous Driving , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[94]  Simon Lucey,et al.  Argoverse: 3D Tracking and Forecasting With Rich Maps , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[95]  Jitendra Malik,et al.  Recurrent Network Models for Human Dynamics , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[96]  Sridha Sridharan,et al.  Tracking by Prediction: A Deep Generative Model for Mutli-person Localisation and Tracking , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[97]  Colin Raffel,et al.  Lasagne: First release. , 2015 .

[98]  John K. Tsotsos,et al.  PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[99]  Philip H. S. Torr,et al.  DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[100]  Byonghyo Shim,et al.  Moving Target Classification in Automotive Radar Systems Using Convolutional Recurrent Neural Networks , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[101]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling , 2018, ArXiv.

[102]  Takayuki Kanda,et al.  Person Tracking in Large Public Spaces Using 3-D Range Sensors , 2013, IEEE Transactions on Human-Machine Systems.

[103]  J. Ferryman,et al.  PETS2009: Dataset and challenge , 2009, 2009 Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance.

[104]  Yisong Yue,et al.  Generating Long-term Trajectories Using Deep Hierarchical Networks , 2016, NIPS.

[105]  Wenguang Wang,et al.  Pedestrian Detection with Lidar Point Clouds Based on Single Template Matching , 2019, Electronics.

[106]  Igal Bilik,et al.  Pedestrian motion direction estimation using simulated automotive MIMO radar , 2016, IEEE Transactions on Aerospace and Electronic Systems.

[107]  Dariu M. Gavrila,et al.  Human motion trajectory prediction: a survey , 2019, Int. J. Robotics Res..

[108]  Guang-Zhong Yang,et al.  ACNN: a Full Resolution DCNN for Medical Image Segmentation , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[109]  Naveed Muhammad,et al.  A Survey on Motion Prediction of Pedestrians and Vehicles for Autonomous Driving , 2021, IEEE Access.

[110]  Jianyu Chen,et al.  Intention-aware Long Horizon Trajectory Prediction of Surrounding Vehicles using Dual LSTM Networks , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[111]  Dinesh Manocha,et al.  TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[112]  Junxuan Zhao,et al.  Probabilistic Prediction of Pedestrian Crossing Intention Using Roadside LiDAR Data , 2019, IEEE Access.

[113]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[114]  Ian D. Reid,et al.  Guiding Visual Surveillance by Tracking Human Attention , 2009, BMVC.

[115]  Xiaogang Wang,et al.  Pedestrian Behavior Understanding and Prediction with Deep Neural Networks , 2016, ECCV.

[116]  Yuan Sun,et al.  Automatic Background Filtering Method for Roadside LiDAR Data , 2018, Transportation Research Record: Journal of the Transportation Research Board.

[117]  Yi-Ting Chen,et al.  The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[118]  Yingfeng Cai,et al.  Pedestrian Motion Trajectory Prediction in Intelligent Driving from Far Shot First-Person Perspective Video , 2021, IEEE Transactions on Intelligent Transportation Systems.

[119]  Patrick Held,et al.  Radar-Based Analysis of Pedestrian Micro-Doppler Signatures Using Motion Capture Sensors , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[120]  Hanky Sjafrie,et al.  Introduction to Self-Driving Vehicle Technology , 2019 .