Discovering Nonlinear Relations with Minimum Predictive Information Regularization

Identifying the underlying directional relations from observational time series with nonlinear interactions and complex relational structures is key to a wide range of applications, yet remains a hard problem. In this work, we introduce a novel minimum predictive information regularization method to infer directional relations from time series, allowing deep learning models to discover nonlinear relations. Our method substantially outperforms other methods for learning nonlinear relations in synthetic datasets, and discovers the directional relations in a video game environment and a heart-rate vs. breath-rate dataset.

[1]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[2]  Razvan Pascanu,et al.  Visual Interaction Networks: Learning a Physics Simulator from Video , 2017, NIPS.

[3]  H. White,et al.  Granger Causality and Dynamic Structural Systems , 2010 .

[4]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[5]  Joshua B. Tenenbaum,et al.  A Compositional Object-Based Approach to Learning Physical Dynamics , 2016, ICLR.

[6]  T. Bliss,et al.  Synaptic plasticity, memory and the hippocampus: a neural network approach to causality , 2012, Nature Reviews Neuroscience.

[7]  Peter Bühlmann,et al.  Characterization and Greedy Learning of Interventional Markov Equivalence Classes of Directed Acyclic Graphs (Abstract) , 2011, UAI.

[8]  Jürgen Schmidhuber,et al.  Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions , 2018, ICLR.

[9]  David Maxwell Chickering,et al.  Optimal Structure Identification With Greedy Search , 2002, J. Mach. Learn. Res..

[10]  Daniele Marinazzo,et al.  Radial basis function approach to nonlinear Granger causality of time series. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[11]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[12]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents , 2012, J. Artif. Intell. Res..

[13]  C. Granger.  Investigating causal relations by econometric models and cross-spectral methods , 1969 .

[14]  Schreiber,et al.  Measuring information transfer , 2000, Physical review letters.

[15]  Vikas Sindhwani,et al.  Scalable Matrix-valued Kernel Learning for High-dimensional Nonlinear Multivariate Regression and Granger Causality , 2012, UAI.

[16]  Daniele Marinazzo,et al.  Kernel method for nonlinear granger causality. , 2007, Physical review letters.

[17]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[18]  Danilo Comminiello,et al.  Group sparse regularization for deep neural networks , 2016, Neurocomputing.

[19]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[20]  Albert Y. Zomaya,et al.  Local information transfer as a spatiotemporal filter for complex systems. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[21]  Moritz Grosse-Wentrup,et al.  Quantifying causal influences , 2012, 1203.6502.

[22]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[23]  Naftali Harris,et al.  PC algorithm for nonparanormal graphical models , 2013, J. Mach. Learn. Res..

[24]  Bernhard Schölkopf,et al.  Elements of Causal Inference: Foundations and Learning Algorithms , 2017 .

[25]  R Quian Quiroga,et al.  Performance of different synchronization measures in real data: a case study on electroencephalographic signals. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[26]  Aaron C. Courville,et al.  MINE: Mutual Information Neural Estimation , 2018, ArXiv.

[27]  Yedid Hoshen,et al.  VAIN: Attentional Multi-agent Predictive Modeling , 2017, NIPS.

[28]  Craig Hiemstra,et al.  Testing for Linear and Nonlinear Granger Causality in the Stock Price-Volume Relation , 1994 .

[29]  C. Granger.  Testing for causality: a personal viewpoint , 1980 .

[30]  J. Pearl.  Causal inference in statistics: An overview , 2009 .

[31]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[32]  Daniele Marinazzo,et al.  Kernel-Granger causality and the analysis of dynamical networks. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[33]  E. Fox,et al.  Neural Granger Causality for Nonlinear Time Series , 2018, 1802.05842.

[34]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[35]  Clive W. J. Granger,et al.  A Bivariate Causality between Stock Prices and Exchange Rates: Evidence from Recent Asia Flu , 1998 .

[36]  David Rolnick,et al.  The power of deeper networks for expressing natural functions , 2017, ICLR.

[37]  Halbert White,et al.  Settable Systems: An Extension of Pearl's Causal Model with Optimization, Equilibrium, and Learning , 2009, J. Mach. Learn. Res..

[38]  Alain Hauser,et al.  High-dimensional consistency in score-based and hybrid structure learning , 2015, The Annals of Statistics.

[39]  Xun Lu,et al.  Linking Granger Causality and the Pearl Causal Model with Settable Systems , 2009, NIPS Mini-Symposium on Causality in Time Series.

[40]  Nathaniel Virgo,et al.  Permutation-equivariant neural networks applied to dynamics prediction , 2016, ArXiv.

[41]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[42]  P. Bühlmann,et al.  The group lasso for logistic regression , 2008 .

[43]  Andre Cohen,et al.  An object-oriented representation for efficient reinforcement learning , 2008, ICML '08.

[44]  Naoki Abe,et al.  Grouped graphical Granger modeling for gene expression regulatory networks discovery , 2009, Bioinform..

[45]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[46]  J. Stock,et al.  Interpreting Evidence on Money-Income Causality , 1987 .

[47]  A. Seth,et al.  Granger Causality Analysis in Neuroscience and Neuroimaging , 2015, The Journal of Neuroscience.