A statistical perspective of sampling scores for linear regression

In this paper, we consider a statistical problem of learning a linear model from noisy samples. Existing work has focused on approximating the least squares solution by using leverage-based scores as an importance sampling distribution. However, no finite sample statistical guarantees and no computationally efficient optimal sampling strategies have been proposed. To evaluate the statistical properties of different sampling strategies, we propose a simple yet effective estimator, which is easy for theoretical analysis and is useful in multitask linear regression. We derive the exact mean square error of the proposed estimator for any given sampling scores. Based on minimizing the mean square error, we propose the optimal sampling scores for both estimator and predictor, and show that they are influenced by the noise-to-signal ratio. Numerical simulations match the theoretical analysis well.

[1]  Michael W. Mahoney,et al.  Optimal Subsampling Approaches for Large Sample Linear Regression , 2015, 1509.05111.

[2]  Rong Jin,et al.  An Explicit Sampling Dependent Spectral Error Bound for Column Subset Selection , 2015, ICML.

[3]  Michael W. Mahoney.  Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[4]  Petros Drineas,et al.  CUR matrix decompositions for improved data analysis , 2009, Proceedings of the National Academy of Sciences.

[5]  S. Muthukrishnan,et al.  Sampling algorithms for l2 regression and applications , 2006, SODA '06.

[6]  David P. Woodruff,et al.  Fast approximation of matrix coherence and statistical leverage , 2011, ICML.

[7]  F. Pukelsheim.  Optimal Design of Experiments (Classics in Applied Mathematics) (Classics in Applied Mathematics, 50) , 2006 .

[8]  Roy E. Welsch,et al.  Efficient Computing of Regression Diagnostics , 1981 .

[9]  AvronHaim,et al.  Blendenpik: Supercharging LAPACK's Least-Squares Solver , 2010 .

[10]  Christos Boutsidis,et al.  Random Projections for the Nonnegative Least-Squares Problem , 2008, ArXiv.

[11]  José M. F. Moura,et al.  Signal Recovery on Graphs: Variation Minimization , 2014, IEEE Transactions on Signal Processing.

[12]  Justin K. Romberg,et al.  Beyond Nyquist: Efficient Sampling of Sparse Bandlimited Signals , 2009, IEEE Transactions on Information Theory.

[13]  Ping Ma,et al.  A statistical perspective on algorithmic leveraging , 2013, J. Mach. Learn. Res..

[14]  S. Muthukrishnan,et al.  Faster least squares approximation , 2007, Numerische Mathematik.

[15]  Sivan Toledo,et al.  Blendenpik: Supercharging LAPACK's Least-Squares Solver , 2010, SIAM J. Sci. Comput..

[16]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[17]  Bernard Chazelle,et al.  Faster dimension reduction , 2010, Commun. ACM.

[18]  Michael Jackson,et al.  Optimal Design of Experiments , 1994 .

[19]  Aarti Singh,et al.  Column Subset Selection with Missing Data via Active Sampling , 2015, AISTATS.

[20]  Jelena Kovacevic,et al.  Signal recovery on graphs: Random versus experimentally designed sampling , 2015, 2015 International Conference on Sampling Theory and Applications (SampTA).

[21]  Robert D. Nowak,et al.  Distilled Sensing: Adaptive Sampling for Sparse Detection and Estimation , 2010, IEEE Transactions on Information Theory.

[22]  Michael A. Saunders,et al.  LSRN: A Parallel Iterative Solver for Strongly Over- or Underdetermined Systems , 2011, SIAM J. Sci. Comput..

[23]  R. Welsch,et al.  The Hat Matrix in Regression and ANOVA , 1978 .

[24]  Aarti Singh,et al.  An empirical comparison of sampling techniques for matrix column subset selection , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[25]  Deanna Needell,et al.  Constrained Adaptive Sensing , 2015, IEEE Transactions on Signal Processing.

[26]  Akshay Krishnamurthy,et al.  Low-Rank Matrix and Tensor Completion via Adaptive Sampling , 2013, NIPS.

[27]  David P. Woodruff,et al.  The Fast Cauchy Transform and Faster Robust Linear Regression , 2012, SIAM journal on computing (Print).

[28]  Yang Weng,et al.  Graphical model for state estimation in electric power systems , 2013, 2013 IEEE International Conference on Smart Grid Communications (SmartGridComm).

[29]  Richard G. Baraniuk,et al.  oASIS: Adaptive Column Sampling for Kernel Matrix Approximation , 2015, ArXiv.

[30]  E.J. Candes.  Compressive Sampling , 2022 .

[31]  Dean P. Foster,et al.  New Subsampling Algorithms for Fast Least Squares Regression , 2013, NIPS.