Debiasing and Distributed Estimation for High-Dimensional Quantile Regression

Distributed and parallel computing is becoming more important with the availability of extremely large data sets. In this article, we consider this problem for high-dimensional linear quantile regression. We work under the assumption that the coefficients in the regression model are sparse; therefore, a LASSO penalty is naturally used for estimation. We first extend the debiasing procedure, which is previously proposed for smooth parametric regression models to quantile regression. The technical challenges include dealing with the nondifferentiability of the loss function and the estimation of the unknown conditional density. In this article, the main objective is to derive a divide-and-conquer estimation approach using the debiased estimator which is useful under the big data setting. The effectiveness of distributed estimation is demonstrated using some numerical examples.

[1]  Alexander J. Smola,et al.  Parallelized Stochastic Gradient Descent , 2010, NIPS.

[2]  Victor Chernozhukov,et al.  Conditional Quantile Processes Based on Series or Many Regressors , 2011, Journal of Econometrics.

[3]  So K Kb.  EFFICIENT SEMIPARAMETRIC ESTIMATION OF A PARTIALLY LINEAR QUANTILE REGRESSION MODEL , 2003 .

[4]  Huaguang Zhang,et al.  Consensus-Based Distributed Cooperative Learning From Closed-Loop Neural Control Systems , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[5]  LianHeng,et al.  Divide-and-conquer for debiased l1-norm support vector machine in ultra-high dimensions , 2017 .

[6]  Qiang Liu,et al.  Communication-efficient Sparse Regression , 2017, J. Mach. Learn. Res..

[7]  Xiangyu Chang,et al.  Distributed Semi-supervised Learning with Kernel Ridge Regression , 2017, J. Mach. Learn. Res..

[8]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[9]  Laurence L. George,et al.  The Statistical Analysis of Failure Time Data , 2003, Technometrics.

[10]  Han Liu,et al.  A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models , 2014, 1412.8765.

[11]  S. Geer,et al.  On asymptotically optimal confidence regions and tests for high-dimensional models , 2013, 1303.0518.

[12]  Ding-Xuan Zhou,et al.  Distributed Learning with Regularized Least Squares , 2016, J. Mach. Learn. Res..

[13]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[14]  Cheng Wu,et al.  Two Efficient Twin ELM Methods With Prediction Interval , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Han Liu,et al.  A PARTIALLY LINEAR FRAMEWORK FOR MASSIVE HETEROGENEOUS DATA. , 2014, Annals of statistics.

[16]  Suleyman S. Kozat,et al.  Online Training of LSTM Networks in Distributed Systems for Variable Length Data Sequences , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Naomi S. Altman,et al.  Quantile regression , 2019, Nature Methods.

[18]  J. Kalbfleisch,et al.  The Statistical Analysis of Failure Time Data , 1980 .

[19]  R. Koenker,et al.  Hierarchical Spline Models for Conditional Quantiles and the Demand for Electricity , 1990 .

[20]  Gideon S. Mann,et al.  Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models , 2009, NIPS.

[21]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[22]  Hongbin Sun,et al.  Distribution-Free Probability Density Forecast Through Deep Neural Networks , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Martin J. Wainwright,et al.  Communication-efficient algorithms for statistical optimization , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[24]  A. Belloni,et al.  L1-Penalised quantile regression in high-dimensional sparse models , 2009 .

[25]  Gang Sun,et al.  Distributed Neural Network Control for Adaptive Synchronization of Uncertain Dynamical Multiagent Systems , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Yufeng Liu,et al.  VARIABLE SELECTION IN QUANTILE REGRESSION , 2009 .

[27]  Guang Cheng,et al.  Distributed inference for quantile regression processes , 2017, The Annals of Statistics.

[28]  Heng Lian,et al.  Divide-and-Conquer for Debiased $l_1$-norm Support Vector Machine in Ultra-high Dimensions , 2017, J. Mach. Learn. Res..

[29]  Moulinath Banerjee,et al.  Divide and conquer in nonstandard problems and the super-efficiency phenomenon , 2016, The Annals of Statistics.

[30]  Le Song,et al.  Communication Efficient Distributed Kernel Principal Component Analysis , 2015, KDD.

[31]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[32]  Brent A. Johnson,et al.  Penalized Estimating Functions and Variable Selection in Semiparametric Regression Models , 2008, Journal of the American Statistical Association.

[33]  T. Cai,et al.  A Constrained ℓ1 Minimization Approach to Sparse Precision Matrix Estimation , 2011, 1102.2233.

[34]  Martin J. Wainwright,et al.  Divide and conquer kernel ridge regression: a distributed algorithm with minimax optimal rates , 2013, J. Mach. Learn. Res..

[35]  Jianqing Fan,et al.  ADAPTIVE ROBUST VARIABLE SELECTION. , 2012, Annals of statistics.

[36]  Runze Li,et al.  Quantile Regression for Analyzing Heterogeneity in Ultra-High Dimension , 2012, Journal of the American Statistical Association.

[37]  Abbas Khosravi,et al.  Incorporating Wind Power Forecast Uncertainties Into Stochastic Unit Commitment Using Neural Network-Based Prediction Intervals , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[38]  A. Tsiatis.  Semiparametric Theory and Missing Data , 2006 .

[39]  C. L. Philip Chen,et al.  Uncertain Data Clustering in Distributed Peer-to-Peer Networks , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[40]  Jonathan D. Rosenblatt,et al.  On the Optimality of Averaging in Distributed Statistical Learning , 2014, 1407.2724.