Variational Policy for Guiding Point Processes

Temporal point processes have been widely applied to model event sequence data generated by online users. In this paper, we consider the problem of how to design the optimal control policy for point processes, such that the stochastic system driven by the point process is steered to a target state. In particular, we exploit the key insight to view the stochastic optimal control problem from the perspective of optimal measure and variational inference. We further propose a convex optimization framework and an efficient algorithm to update the policy adaptively to the current system state. Experiments on synthetic and real-world data show that our algorithm can steer the user activities much more accurately and efficiently than other stochastic control methods.

[1]  Éva Tardos,et al.  Maximizing the Spread of Influence through a Social Network , 2015, Theory Comput..

[2]  Yosihiko Ogata,et al.  On Lewis' simulation method for point processes , 1981, IEEE Trans. Inf. Theory.

[3]  Vinayak A. Rao,et al.  A Multitask Point Process Predictive Model , 2015, ICML.

[4]  Wolfgang J. Runggaldier,et al.  Connections between stochastic control and dynamic games , 1996, Math. Control. Signals Syst..

[5]  Evangelos Theodorou,et al.  Nonlinear Stochastic Control and Information Theoretic Dualities: Connections, Interdependencies and Thermodynamic Interpretations , 2015, Entropy.

[6]  Le Song,et al.  Coevolutionary Latent Feature Processes for Continuous-Time User-Item Interactions , 2016, NIPS.

[7]  Le Song,et al.  A Stochastic Differential Equation Framework for Guiding Online User Activities in Closed Loop , 2016, AISTATS.

[8]  Le Song,et al.  Fast and Simple Optimization for Poisson Likelihood Models , 2016, ArXiv.

[9]  Yuan Qi,et al.  Content-based Modeling of Reciprocal Relationships using Hawkes and Gaussian Processes , 2016, UAI.

[10]  Le Song,et al.  Linking Micro Event History to Macro Prediction in Point Process Models , 2017, AISTATS.

[11]  Michael D. Lemmon,et al.  Event-Triggered Feedback in Control, Estimation, and Optimization , 2010 .

[12]  Le Song,et al.  Predicting User Activity Level In Point Processes With Mass Transport Equation , 2017, NIPS.

[13]  P. Brémaud.  Point Processes and Queues , 1981 .

[14]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[15]  Le Song,et al.  Isotonic Hawkes Processes , 2016, ICML.

[16]  Xi Chen,et al.  Temporal Collaborative Filtering with Bayesian Probabilistic Tensor Factorization , 2010, SDM.

[17]  Vinayak A. Rao,et al.  Markov-modulated Marked Poisson Processes for Check-in Data , 2016, ICML.

[18]  James M. Rehg,et al.  Aggressive driving with model predictive path integral control , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Arnold Neumaier,et al.  Introduction to Numerical Analysis , 2001 .

[20]  Daryl J. Daley,et al.  Introduction to the General Theory of Point Processes , 1998 .

[21]  Peter E. Caines,et al.  Stochastic optimal control under Poisson-distributed observations , 2000, IEEE Trans. Autom. Control..

[22]  Le Song,et al.  Deep Coevolutionary Network: Embedding User and Item Features for Recommendation , 2016, 1609.03675.

[23]  Erhan Bayraktar,et al.  LIQUIDATION IN LIMIT ORDER BOOKS WITH CONTROLLED INTENSITY , 2011, ArXiv.

[24]  A. Hawkes.  Spectra of some self-exciting and mutually exciting point processes , 1971 .

[25]  Jure Leskovec,et al.  Steering user behavior with badges , 2013, WWW.

[26]  Ulrike Goldschmidt,et al.  An Introduction To The Theory Of Point Processes , 2016 .

[27]  Thomas P. Minka,et al.  Divergence measures and message passing , 2005 .

[28]  James R. Foulds,et al.  HawkesTopic: A Joint Model for Network Inference and Topic Modeling from Text-Based Cascades , 2015, ICML.

[29]  Subramanian Ramamoorthy,et al.  Applied Stochastic Control of Jump Diffusions , 2011 .

[30]  Eduardo Fernandez-Camacho,et al.  Introduction to Model Based Predictive Control , 1995 .

[31]  Jimeng Sun,et al.  Rubik: Knowledge Guided Tensor Factorization and Completion for Health Data Analytics , 2015, KDD.

[32]  Marko Bacic,et al.  Model predictive control , 2003 .

[33]  Thomas Josef Liniger,et al.  Multivariate Hawkes processes , 2009 .

[34]  B. Øksendal,et al.  Applied Stochastic Control of Jump Diffusions , 2004, Universitext.

[35]  Olivier Sigaud,et al.  Path Integral Policy Improvement with Covariance Matrix Adaptation , 2012, ICML.

[36]  Le Song,et al.  Shaping Social Activity by Incentivizing Users , 2014, NIPS.

[37]  O. Aalen,et al.  Survival and Event History Analysis: A Process Point of View , 2008 .

[38]  Stefan Schaal,et al.  Policy Gradient Methods for Robotics , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Le Song,et al.  Smart Broadcasting: Do You Want to be Seen? , 2016, KDD.

[40]  Mohamed Darouach,et al.  Sensing and actuation strategies for event triggered stochastic optimal control , 2013, 52nd IEEE Conference on Decision and Control.

[41]  Manuel Gomez Rodriguez,et al.  Learning Opinion Dynamics in Social Networks , 2015 .

[42]  Paulo Tabuada,et al.  An introduction to event-triggered and self-triggered control , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[43]  Floyd B. Hanson,et al.  Applied stochastic processes and control for jump-diffusions - modeling, analysis, and computation , 2007, Advances in design and control.

[44]  Vinayak A. Rao,et al.  Modeling Correlated Arrival Events with Latent Semi-Markov Processes , 2014, ICML.

[45]  Le Song,et al.  Steering Opinion Dynamics in Information Diffusion Networks , 2016, ArXiv.

[46]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[47]  Le Song,et al.  Time-Sensitive Recommendation From Recurrent User Activities , 2015, NIPS.

[48]  Huy En Pham.  Optimal Stopping of Controlled Jump Diiusion Processes: a Viscosity Solution Approach , 1998 .

[49]  Le Song,et al.  Learning Social Infectivity in Sparse Low-rank Networks Using Multi-dimensional Hawkes Processes , 2013, AISTATS.

[50]  Niloy Ganguly,et al.  Learning and Forecasting Opinion Dynamics in Social Networks , 2015, NIPS.

[51]  Shuang Li,et al.  COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolution , 2015, NIPS.

[52]  Hamid R. Rabiee,et al.  RedQueen: An Online Algorithm for Smart Broadcasting in Social Networks , 2016, WSDM.

[53]  P. Varaiya,et al.  Optimal Control of Jump Processes , 1977 .