OPTIMAL SUBSAMPLING ALGORITHMS FOR BIG DATA REGRESSIONS

To fast approximate maximum likelihood estimators with massive data, this paper studies the Optimal Subsampling Method under the A-optimality Criterion (OSMAC) for generalized linear models. The consistency and asymptotic normality of the estimator from a general subsampling algorithm are established, and optimal subsampling probabilities under the A- and L-optimality criteria are derived. Furthermore, using Frobenius norm matrix concentration inequalities, finite sample properties of the subsample estimator based on optimal subsampling probabilities are also derived. Since the optimal subsampling probabilities depend on the full data estimate, an adaptive two-step algorithm is developed. Asymptotic normality and optimality of the estimator from this adaptive algorithm are established. The proposed methods are illustrated and evaluated through numerical experiments on simulated and real datasets.

[1]  M. H. Hansen,et al.  On the Theory of Sampling from Finite Populations , 1943 .

[2]  M. Silvapulle.  On the Existence of Maximum Likelihood Estimators for the Binomial Response Models , 1981 .

[3]  L. Fahrmeir,et al.  Correction: Consistency and Asymptotic Normality of the Maximum Likelihood Estimator in Generalized Linear Models , 1985 .

[4]  L. Brown.  Fundamentals of statistical exponential families: with applications in statistical decision theory , 1986 .

[5]  Andy H. Lee.  Diagnostic Displays for Assessing Leverage and Influence in Generalized Linear Models , 1987 .

[6]  Michael Jackson,et al.  Optimal Design of Experiments , 1994 .

[7]  L. Fahrmeir,et al.  Multivariate statistical modelling based on generalized linear models , 1994 .

[8]  J. Sacks,et al.  Artic sea ice variability: Model sensitivities and a multidecadal simulation , 1994 .

[9]  T. Ferguson.  A Course in Large Sample Theory , 1996 .

[10]  Carl-Erik Särndal,et al.  Model Assisted Survey Sampling , 1997 .

[11]  Claudia Czado,et al.  Noncanonical links in generalized linear models – when is the effort justified? , 2000 .

[12]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[13]  S. Muthukrishnan,et al.  Sampling algorithms for l2 regression and applications , 2006, SODA '06.

[14]  Petros Drineas,et al.  Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication , 2006, SIAM J. Comput..

[15]  Anthony C. Atkinson,et al.  Optimum Experimental Designs, with SAS , 2007 .

[16]  Shifeng Xiong,et al.  Some results on the convergence of conditional distributions , 2008 .

[17]  Jerome Sacks,et al.  Choosing the Sample Size of a Computer Experiment: A Practical Guide , 2009, Technometrics.

[18]  Michael W. Mahoney.  Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[19]  Thierry Bertin-Mahieux,et al.  The Million Song Dataset , 2011, ISMIR.

[20]  S. Muthukrishnan,et al.  Faster least squares approximation , 2007, Numerische Mathematik.

[21]  David Z. Goodson.  Mathematical Methods for Physical and Analytical Chemistry , 2011 .

[22]  David P. Woodruff,et al.  Fast approximation of matrix coherence and statistical leverage , 2011, ICML.

[23]  Stéphan Clémençon,et al.  Scaling up M-estimation via sampling designs: The Horvitz-Thompson stochastic gradient descent , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[24]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[25]  Jean-Michel Loubes,et al.  Oracle Inequalities for a Group Lasso Procedure Applied to Generalized Linear Models in High Dimension , 2013, IEEE Transactions on Information Theory.

[26]  Xiaoxiao Sun,et al.  Leveraging for big data regression , 2015 .

[27]  Sadique Sheik,et al.  Reservoir computing compensates slow response of chemosensor arrays exposed to fast varying gas concentrations in continuous monitoring , 2015 .

[28]  Ping Ma,et al.  A statistical perspective on algorithmic leveraging , 2013, J. Mach. Learn. Res..

[29]  Ming-Hui Chen,et al.  Statistical methods and computing for big data. , 2015, Statistics and its interface.

[30]  Trevor Hastie,et al.  Computer Age Statistical Inference: Algorithms, Evidence, and Data Science , 2016 .

[31]  Jinzhu Jia,et al.  Elastic-net Regularized High-dimensional Negative Binomial Regression: Consistency and Weak Signals Detection , 2017, 1712.03412.

[32]  Min Yang,et al.  Information-Based Optimal Subdata Selection for Big Data Linear Regression , 2017, Journal of the American Statistical Association.

[33]  HaiYing Wang,et al.  Optimal subsampling for softmax regression , 2019, Statistical Papers.

[34]  Rong Zhu,et al.  Optimal Subsampling for Large Sample Logistic Regression , 2017, Journal of the American Statistical Association.

[35]  HaiYing Wang,et al.  Divide-and-Conquer Information-Based Optimal Subdata Selection Algorithm , 2019, Journal of Statistical Theory and Practice.