High-Dimensional Bernoulli Autoregressive Process with Long-Range Dependence

We consider the problem of estimating the parameters of a multivariate Bernoulli process with auto-regressive feedback in the high-dimensional setting where the number of samples available is much less than the number of parameters. This problem arises in learning interconnections of networks of dynamical systems with spiking or binary-valued data. We allow the process to depend on its past up to a lag $p$, for a general $p \ge 1$, allowing for more realistic modeling in many applications. We propose and analyze an $\ell_1$-regularized maximum likelihood estimator (MLE) under the assumption that the parameter tensor is approximately sparse. Rigorous analysis of such estimators is made challenging by the dependent and non-Gaussian nature of the process as well as the presence of the nonlinearities and multi-level feedback. We derive precise upper bounds on the mean-squared estimation error in terms of the number of samples, dimensions of the process, the lag $p$ and other key statistical properties of the model. The ideas presented can be used in the high-dimensional analysis of regularized $M$-estimators for other sparse nonlinear and non-Gaussian processes with long-range dependence.

[1]  Y. Guanche,et al.  Autoregressive logistic regression applied to atmospheric circulation patterns , 2013, Climate Dynamics.

[2]  Min Wu,et al.  Robust Estimation of Self-Exciting Generalized Linear Models With Application to Neuronal Modeling , 2015, IEEE Transactions on Signal Processing.

[3]  M. Rudelson,et al.  Hanson-Wright inequality and sub-gaussian concentration , 2013 .

[4]  秀俊 松井,et al.  Statistics for High-Dimensional Data: Methods, Theory and Applications , 2014 .

[5]  M. Krein,et al.  On Regularly Convex Sets in the Space Conjugate to a Banach Space , 1940 .

[6]  R. Srikant,et al.  Mixing Times and Structural Inference for Bernoulli Autoregressive Processes , 2016, IEEE Transactions on Network Science and Engineering.

[7]  A. Rhodius.  On the maximum of ergodicity coefficients, the Dobrushin ergodicity coefficient, and products of stochastic matrices , 1997 .

[8]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[9]  Paul-Marie Samson,et al.  Concentration of measure inequalities for Markov chains and $\Phi$-mixing processes , 2000 .

[10]  M. Wilson,et al.  Analyzing Functional Connectivity Using a Network Likelihood Model of Ensemble Neural Spiking Activity , 2005, Neural Computation.

[11]  Roummel F. Marcia,et al.  Sequential Anomaly Detection in the Presence of Noise and Limited Feedback , 2009, IEEE Transactions on Information Theory.

[12]  Martin J. Wainwright,et al.  Minimax Rates of Estimation for High-Dimensional Linear Regression Over $\ell_q$ -Balls , 2009, IEEE Transactions on Information Theory.

[13]  S. Geer.  On Hoeffding's Inequality for Dependent Random Variables , 2002 .

[14]  D. Politis,et al.  High-dimensional autocovariance matrices and optimal linear prediction , 2015 .

[15]  Garvesh Raskutti,et al.  Network Estimation From Point Process Data , 2018, IEEE Transactions on Information Theory.

[16]  K. Ramanan,et al.  Concentration Inequalities for Dependent Random Variables via the Martingale Method , 2006, math/0609835.

[17]  Garvesh Raskutti,et al.  Non-Parametric Sparse Additive Auto-Regressive Network Models , 2018, IEEE Transactions on Information Theory.

[18]  Mathew W. McLean,et al.  Forecasting emergency medical service call arrival rates , 2011, 1107.4919.

[19]  Rebecca Willett,et al.  Network estimation via poisson autoregressive models , 2017, 2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[20]  Garvesh Raskutti,et al.  Convex regularization for high-dimensional multiresponse tensor regression , 2015, The Annals of Statistics.

[21]  Rebecca Willett,et al.  Inference of High-dimensional Autoregressive Generalized Linear Models , 2016, ArXiv.

[22]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[23]  Roberto Casarin,et al.  Sparse Graphical Vector Autoregression: A Bayesian Approach , 2014 .

[24]  G. Michailidis,et al.  Regularized estimation in sparse high-dimensional time series models , 2013, 1311.4175.

[25]  R. Startz.  Binomial Autoregressive Moving Average Models With an Application to U.S. Recessions , 2006 .

[26]  Ambuj Tewari,et al.  Sequential complexities and uniform martingale laws of large numbers , 2015 .

[27]  Jonathan W. Pillow,et al.  Capturing the Dynamical Repertoire of Single Neurons with Generalized Linear Models , 2016, Neural Computation.

[28]  Harrison H. Zhou,et al.  Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation , 2016 .

[29]  Keming Yu,et al.  Using auto‐regressive logit models to forecast the exceedance probability for financial risk management , 2016 .

[30]  Robert M. Gray,et al.  Toeplitz and Circulant Matrices: A Review , 2005, Found. Trends Commun. Inf. Theory.

[31]  B. Keshtegar,et al.  An autoregressive logistic model to predict the reciprocal effects of oviductal fluid components on in vitro spermophagy by neutrophils in cattle , 2017, Scientific Reports.

[32]  José M. F. Moura,et al.  Signal Processing on Graphs: Causal Modeling of Unstructured Data , 2015, IEEE Transactions on Signal Processing.

[33]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[34]  K. Marton.  Bounding $\bar{d}$-distance by informational divergence: a method to prove measure concentration , 1996 .

[35]  L. Kontorovich.  Obtaining Measure Concentration from Markov Contraction , 2007, 0711.0987.

[36]  Emery N. Brown,et al.  Estimating a State-space Model from Point Process Observations Emery N. Brown , 2022 .

[37]  R. Kass,et al.  Multiple neural spike train data analysis: state-of-the-art and future challenges , 2004, Nature Neuroscience.

[38]  Cun-Hui Zhang,et al.  The sparsity and bias of the Lasso selection in high-dimensional linear regression , 2008, 0808.0967.

[39]  Abbas Kazemipour,et al.  Compressed Sensing Beyond the IID and Static Domains: Theory, Algorithms and Applications , 2018, 1806.11194.