Tapestry of Time and Actions: Modeling Human Activity Sequences Using Temporal Point Process Flows

Human beings always engage in a vast range of activities and tasks that demonstrate their ability to adapt to different scenarios. Any human activity can be represented as a temporal sequence of actions performed to achieve a certain goal. Unlike the time series datasets extracted from electronics or machines, these action sequences are highly disparate in their nature -- the time to finish a sequence of actions can vary between different persons. Therefore, understanding the dynamics of these sequences is essential for many downstream tasks such as activity length prediction, goal prediction, next action recommendation, etc. Existing neural network-based approaches that learn a continuous-time activity sequence (or CTAS) are limited to the presence of only visual data or are designed specifically for a particular task, i.e., limited to next action or goal prediction. In this paper, we present ProActive, a neural marked temporal point process (MTPP) framework for modeling the continuous-time distribution of actions in an activity sequence while simultaneously addressing three high-impact problems -- next action prediction, sequence-goal prediction, and end-to-end sequence generation. Specifically, we utilize a self-attention module with temporal normalizing flows to model the influence and the inter-arrival times between actions in a sequence. In addition, we propose a novel addition over the ProActive model that can handle variations in the order of actions, i.e., different methods of achieving a given goal. We demonstrate that this variant can learn the order in which the person or actor prefers to do their actions. Extensive experiments on sequences derived from three activity recognition datasets show the significant accuracy boost of ProActive over the state-of-the-art in terms of action and goal prediction, and the first-ever application of end-to-end action sequence generation.

[1]  Vinayak Gupta.  Modeling Time-Series and Spatial Data for Recommendations and Other Applications , 2022, ArXiv.

[2]  Srikanta J. Bedathur,et al.  ProActive: Self-Attentive Temporal Point Process Flows for Activity Sequences , 2022, KDD.

[3]  Srikanta J. Bedathur,et al.  Learning Temporal Point Processes for Efficient Retrieval of Continuous Time Event Sequences , 2022, AAAI.

[4]  Srikanta J. Bedathur,et al.  Region Invariant Normalizing Flows for Mobility Transfer , 2021, CIKM.

[5]  Shaghayegh Sahebi,et al.  Stimuli-Sensitive Hawkes Processes for Personalized Student Procrastination Modeling , 2021, WWW.

[6]  Sunita Sarawagi,et al.  Long Horizon Forecasting with Temporal Point Processes , 2021, WSDM.

[7]  Srikanta J. Bedathur,et al.  Modeling Implicit Communities from Geo-Tagged Event Traces Using Spatio-Temporal Point Processes , 2020, WISE.

[8]  Li Jin,et al.  User Independent Estimations of Gait Events With Minimal Sensor Data , 2020, IEEE Journal of Biomedical and Health Informatics.

[9]  Karishma Sharma,et al.  Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours , 2020, KDD.

[10]  Jack T. Dennerlein,et al.  A database of human gait performance on irregular and uneven surfaces collected by wearable sensors , 2020, Scientific Data.

[11]  Ivan Kobyzev,et al.  Normalizing Flows: An Introduction and Review of Current Methods , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Hongyuan Zha,et al.  Transformer Hawkes Process , 2020, ICML.

[13]  Yujie Wang,et al.  Time Interval Aware Self-Attention for Sequential Recommendation , 2020, WSDM.

[14]  Stephan Günnemann,et al.  Intensity-Free Learning of Temporal Point Processes , 2019, ICLR.

[15]  Marcus A. Brubaker,et al.  Point Process Flows , 2019, ArXiv.

[16]  Emine Yilmaz,et al.  Self-Attentive Hawkes Processes , 2019, ArXiv.

[17]  K. Aihara,et al.  Fully Neural Network based Model for General Temporal Point Processes , 2019, NeurIPS.

[18]  Jiawei He,et al.  A Variational Auto-Encoder Model for Stochastic Point Processes , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Silviu Pitis,et al.  Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach , 2019, AAAI.

[20]  Yee Whye Teh,et al.  Set Transformer , 2018, ICML.

[21]  Patrick Lucey,et al.  Where Will They Go? Predicting Fine-Grained Adversarial Multi-agent Motion Using Conditional Variational Autoencoders , 2018, ECCV.

[22]  Julian J. McAuley,et al.  Self-Attentive Sequential Recommendation , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[23]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[24]  Seiichi Uchida,et al.  Biosignal Data Augmentation Based on Generative Adversarial Networks , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[25]  Yazan Abu Farha,et al.  When will you do what? - Anticipating Temporal Occurrences of Activities , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Swapnil Mishra,et al.  SIR-Hawkes: Linking Epidemic Models and Hawkes Processes to Model Diffusions in Finite Populations , 2017, WWW.

[27]  Jitendra Malik,et al.  What will Happen Next? Forecasting Player Moves in Sports Videos , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Amit K. Roy-Chowdhury,et al.  Joint Prediction of Activity Labels and Starting Times in Untrimmed Videos , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[30]  Le Song,et al.  Wasserstein Learning of Deep Generative Point Process Models , 2017, NIPS.

[31]  Luke Bornn,et al.  Learning Person Trajectory Representations for Team Activity Analysis , 2017, ArXiv.

[32]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[33]  Hongyuan Zha,et al.  Modeling the Intensity Function of Point Process Via Recurrent Neural Networks , 2017, AAAI.

[34]  Mani B. Srivastava,et al.  SenseGen: A deep learning architecture for synthetic sensor data generation , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[35]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[36]  Utkarsh Upadhyay,et al.  Recurrent Marked Temporal Point Processes: Embedding Event History to Vector , 2016, KDD.

[37]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[38]  Stan Sclaroff,et al.  Learning Activity Progression in LSTMs for Activity Detection and Early Detection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[40]  Scott Sanner,et al.  Expecting to be HIP: Hawkes Intensity Processes for Social Media Popularity , 2016, WWW.

[41]  Li Fei-Fei,et al.  Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos , 2015, International Journal of Computer Vision.

[42]  Jure Leskovec,et al.  SEISMIC: A Self-Exciting Point Process Model for Predicting Tweet Popularity , 2015, KDD.

[43]  Bernard Ghanem,et al.  ActivityNet: A large-scale video benchmark for human activity understanding , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[45]  E. Bacry,et al.  Hawkes Processes in Finance , 2015, 1502.04592.

[46]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[47]  Silvio Savarese,et al.  A Hierarchical Representation for Future Action Prediction , 2014, ECCV.

[48]  Thomas Serre,et al.  The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Fernando De la Torre,et al.  Max-Margin Early Event Detectors , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Michael S. Ryoo,et al.  Human activity prediction: Early recognition of ongoing activities from streaming videos , 2011, 2011 International Conference on Computer Vision.

[51]  A. Hawkes.  Spectra of some self-exciting and mutually exciting point processes , 1971 .

[52]  Srikanta J. Bedathur,et al.  Learning Temporal Point Processes with Intermittent Observations , 2021, AISTATS.

[53]  Panna Felsen,et al.  Learning to Predict Human Behavior from Video , 2019 .

[54]  Mihaela van der Schaar,et al.  Time-series Generative Adversarial Networks , 2019, NeurIPS.

[55]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.