SRA-LSTM: Social Relationship Attention LSTM for Human Trajectory Prediction

Pedestrian trajectory prediction for surveillance video is one of the important research topics in the field of computer vision and a key technology of intelligent surveillance systems. Social relationship among pedestrians is a key factor influencing pedestrian walking patterns but was mostly ignored in the literature. Pedestrians with different social relationships play different roles in the motion decision of target pedestrian. Motivated by this idea, we propose a Social Relationship Attention LSTM (SRA-LSTM) model to predict future trajectories. We design a social relationship encoder to obtain the representation of their social relationship through the relative position between each pair of pedestrians. Afterwards, the social relationship feature and latent movements are adopted to acquire the social relationship attention of this pair of pedestrians. Social interaction modeling is achieved by utilizing social relationship attention to aggregate movement information from neighbor pedestrians. Experimental results on two public walking pedestrian video datasets (ETH and UCY), our model achieves superior performance compared with state-of-theart methods. Contrast experiments with other attention methods also demonstrate the effectiveness of social relationship attention.

[1]  Haibo Luo,et al.  Adaptive feature fusion with attention mechanism for multi-scale target detection , 2020, Neural Computing and Applications.

[2]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Juan Carlos Niebles,et al.  Peeking Into the Future: Predicting Future Person Activities and Locations in Videos , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Hugo Proença,et al.  An Attention-Based Deep Learning Model for Multiple Pedestrian Attributes Recognition , 2020, Image Vis. Comput..

[5]  Ioannis E. Livieris,et al.  A CNN–LSTM model for gold price time-series forecasting , 2020, Neural Computing and Applications.

[6]  Julien Pettré,et al.  Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Anand Singh Jalal,et al.  Integration of textual cues for fine-grained image captioning using deep CNN and LSTM , 2019, Neural Computing and Applications.

[8]  Brendan Tran Morris,et al.  SSeg-LSTM: Semantic Scene Segmentation for Trajectory Prediction , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[9]  Sridha Sridharan,et al.  Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection , 2017, Neural Networks.

[10]  Mark Reynolds,et al.  Location-Velocity Attention for Pedestrian Trajectory Prediction , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[11]  Kai Huang,et al.  Collision-Free LSTM for Human Trajectory Prediction , 2018, MMM.

[12]  Abduallah A. Mohamed,et al.  Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Nanning Zheng,et al.  SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Masayoshi Tomizuka,et al.  Conditional Generative Neural System for Probabilistic Trajectory Prediction , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[15]  Ying Liu,et al.  Context-aware attention network for image recognition , 2019, Neural Computing and Applications.

[16]  Lamberto Ballan,et al.  Social and Scene-Aware Trajectory Prediction in Crowded Spaces , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[17]  Dani Lischinski,et al.  Crowds by Example , 2007, Comput. Graph. Forum.

[18]  Du Q. Huynh,et al.  A Location-Velocity-Temporal Attention LSTM Model for Pedestrian Trajectory Prediction , 2020, IEEE Access.

[19]  Zhaoxin Li,et al.  STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Qing-dao-er-ji Ren,et al.  Research on the LSTM Mongolian and Chinese machine translation based on morpheme encoding , 2018, Neural Computing and Applications.

[21]  Jean Oh,et al.  Social Attention: Modeling Attention in Human Crowds , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Ryosuke Shibasaki,et al.  Pedestrian Trajectory Prediction in Extremely Crowded Scenarios , 2019, Sensors.

[23]  Silvio Savarese,et al.  Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks , 2019, NeurIPS.

[24]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Xiaogang Wang,et al.  Pedestrian Behavior Modeling From Stationary Crowds With Applications to Intelligent Surveillance , 2016, IEEE Transactions on Image Processing.

[27]  Yue Hu,et al.  Collaborative Motion Prediction via Neural Motion Message Passing , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Mark Reynolds,et al.  SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[29]  Zhineng Chen,et al.  Adaptive multi-branch correlation filters for robust visual tracking , 2020, Neural Computing and Applications.

[30]  Helbing,et al.  Social force model for pedestrian dynamics. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[31]  Jin Guo,et al.  A spatial-temporal attention model for human trajectory prediction , 2020, IEEE/CAA Journal of Automatica Sinica.

[32]  Cewu Lu,et al.  Recursive Social Behavior Graph for Trajectory Prediction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Chunni Dai.  Online surveillance object classification with training data updating , 2016, 2016 International Conference on Audio, Language and Image Processing (ICALIP).

[34]  Xiaogang Wang,et al.  Understanding pedestrian behaviors from stationary crowd groups , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[36]  Jaime S. Cardoso,et al.  Cross-layer classification framework for automatic social behavioural analysis in surveillance scenario , 2017, Neural Computing and Applications.

[37]  Kamaruzzaman Seman,et al.  A new spatio-temporal background–foreground bimodal for motion segmentation and detection in urban traffic scenes , 2019, Neural Computing and Applications.

[38]  Yang Chen,et al.  TPPO: A Novel Trajectory Predictor With Pseudo Oracle , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[39]  Shenghua Gao,et al.  Encoding Crowd Interaction with Deep Neural Network for Pedestrian Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Min Zhang,et al.  Two-branch encoding and iterative attention decoding network for semantic segmentation , 2020, Neural Computing and Applications.

[41]  Debi Prosad Dogra,et al.  Surveillance scene representation and trajectory abnormality detection using aggregation of multiple concepts , 2018, Expert Syst. Appl..