On Computationally Tractable Selection of Experiments in Measurement-Constrained Regression Models

We derive computationally tractable methods to select a small subset of experiment settings from a large pool of given design points. The primary focus is on linear regression models, while the technique extends to generalized linear models and Delta's method (estimating functions of linear regression models) as well. The algorithms are based on a continuous relaxation of an otherwise intractable combinatorial optimization problem, with sampling or greedy procedures as post-processing steps. Formal approximation guarantees are established for both algorithms, and numerical results on both synthetic and real-world data confirm the effectiveness of the proposed methods.

[1]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[2]  S. Muthukrishnan,et al.  Faster least squares approximation , 2007, Numerische Mathematik.

[3]  Rémi Munos,et al.  Active Regression by Stratification , 2014, NIPS.

[4]  S. Muthukrishnan,et al.  Relative-Error CUR Matrix Decompositions , 2007, SIAM J. Matrix Anal. Appl..

[5]  Jelena Kovacevic,et al.  Signal Representations on Graphs: Tools and Applications , 2015, ArXiv.

[6]  Ming Gu,et al.  Efficient Algorithms for Computing a Strong Rank-Revealing QR Factorization , 1996, SIAM J. Sci. Comput..

[7]  Michael W. Mahoney,et al.  Statistical and Algorithmic Perspectives on Randomized Sketching for Ordinary Least-Squares , 2015, ICML.

[8]  Arumugam Manthiram,et al.  Microwave-assisted Low-temperature Growth of Thin Films in Solution , 2012, Scientific reports.

[9]  Nikhil Srivastava,et al.  Graph sparsification by effective resistances , 2008, SIAM J. Comput..

[10]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[11]  Nikhil Srivastava,et al.  Interlacing Families I: Bipartite Ramanujan Graphs of All Degrees , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[12]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[13]  M. Ghosh,et al.  Design Issues for Generalized Linear Models: A Review , 2006, math/0701088.

[14]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[15]  S. Goreinov,et al.  A Theory of Pseudoskeleton Approximations , 1997 .

[16]  Sujay Sanghavi,et al.  Completing any low-rank matrix, provably , 2013, J. Mach. Learn. Res..

[17]  John A. Pyles,et al.  Exploration of complex visual feature spaces for object perception , 2014, Front. Comput. Neurosci..

[18]  Sham M. Kakade,et al.  A tail inequality for quadratic forms of subgaussian random vectors , 2011, ArXiv.

[19]  Ping Ma,et al.  A statistical perspective on algorithmic leveraging , 2013, J. Mach. Learn. Res..

[20]  F. Bunea,et al.  On the sample covariance matrix estimator of reduced effective rank population matrices, with applications to fPCA , 2012, 1212.5321.

[21]  Klaus-Robert Müller,et al.  Covariate Shift Adaptation by Importance Weighted Cross Validation , 2007, J. Mach. Learn. Res..

[22]  J. Kadane,et al.  Design for low‐temperature microwave‐assisted crystallization of ceramic thin films , 2017 .

[23]  Elad Hazan,et al.  Hard-Margin Active Linear Regression , 2014, ICML.

[24]  D. Spielman,et al.  Interlacing Families II: Mixed Characteristic Polynomials and the Kadison-Singer Problem , 2013, 1306.3969.

[25]  Nikhil Srivastava,et al.  Twice-ramanujan sparsifiers , 2008, STOC '09.

[26]  Maria-Florina Balcan,et al.  Active and passive learning of linear separators under log-concave distributions , 2012, COLT.

[27]  References , 1971 .

[28]  Yang Liu,et al.  Fast Relative-Error Approximation Algorithm for Ridge Regression , 2015, UAI.

[29]  F. Hoog,et al.  A note on subset selection for matrices , 2011 .

[30]  Alan J. Miller,et al.  A Fedorov Exchange Algorithm for D-optimal Design , 1994 .

[31]  D. Botstein,et al.  For Personal Use. Only Reproduce with Permission from the Lancet Publishing Group , 2022 .

[32]  Deanna Needell,et al.  Constrained Adaptive Sensing , 2015, IEEE Transactions on Signal Processing.

[33]  David P. Woodruff Sketching as a Tool for Numerical Linear Algebra , 2014, Found. Trends Theor. Comput. Sci..

[34]  Maria-Florina Balcan,et al.  Margin Based Active Learning , 2007, COLT.

[35]  Ming Gu,et al.  An Efficient Algorithm for Unweighted Spectral Graph Sparsification , 2014, ArXiv.

[36]  Mark Rudelson,et al.  Sampling from large matrices: An approach through geometric functional analysis , 2005, JACM.

[37]  David P. Woodru Sketching as a Tool for Numerical Linear Algebra , 2014 .

[38]  Andreas Krause,et al.  Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies , 2008, J. Mach. Learn. Res..

[39]  Phillip Ein-Dor,et al.  Attributes of the performance of central processing units: a relative performance prediction model , 1987, CACM.

[40]  Anton van den Hengel,et al.  Semidefinite Programming , 2014, Computer Vision, A Reference Guide.

[41]  Sham M. Kakade,et al.  Convergence Rates of Active Learning for Maximum Likelihood Estimation , 2015, NIPS.

[42]  Michael W. Mahoney,et al.  Optimal Subsampling Approaches for Large Sample Linear Regression , 2015, 1509.05111.

[43]  Hao Su,et al.  Efficient Euclidean Projections onto the Intersection of Norm Balls , 2012, ICML.

[44]  Michael W. Mahoney,et al.  Fast Randomized Kernel Ridge Regression with Statistical Guarantees , 2015, NIPS.

[45]  Michael Jackson,et al.  Optimal Design of Experiments , 1994 .

[46]  Malik Magdon-Ismail,et al.  On selecting a maximum volume sub-matrix of a matrix and related problems , 2009, Theor. Comput. Sci..

[47]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[48]  Stephen P. Boyd,et al.  Sensor Selection via Convex Optimization , 2009, IEEE Transactions on Signal Processing.

[49]  Russ B. Altman,et al.  Missing value estimation methods for DNA microarrays , 2001, Bioinform..

[50]  Christos Boutsidis,et al.  Faster Subset Selection for Matrices and Applications , 2011, SIAM J. Matrix Anal. Appl..

[51]  Michael I. Jordan,et al.  Matrix concentration inequalities via the method of exchangeable pairs , 2012, 1201.6002.

[52]  H. Chernoff Locally Optimal Designs for Estimating Parameters , 1953 .

[53]  Stratis Ioannidis,et al.  Budget Feasible Mechanisms for Experimental Design , 2013, LATIN.

[54]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[55]  W. Hoeffding Probability Inequalities for sums of Bounded Random Variables , 1963 .

[56]  Gábor Pataki,et al.  On the Rank of Extreme Matrices in Semidefinite Programs and the Multiplicity of Optimal Eigenvalues , 1998, Math. Oper. Res..

[57]  Maxim Sviridenko,et al.  Pipage Rounding: A New Method of Constructing Algorithms with Proven Performance Guarantee , 2004, J. Comb. Optim..

[58]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[59]  R. Tibshirani The Lasso Problem and Uniqueness , 2012, 1206.0313.