Simultaneous support recovery in high dimensions : Benefits and perils of block l 1 / l ∞-regularization

Given a collection of r ≥ 2 linear regression problems in p dimensions, suppose that the regression coefficients share partially common supports of size at most s. This set-up suggests the use of l1/l∞-regularized regression for joint estimation of the p × r matrix of regression coefficients. We analyze the high-dimensional scaling ofl1/l∞-regularized quadratic programming, considering both consistency rates in l∞-norm, and how the minimal sample size n required for consistent variable selection scales with model dimension, sparsity, and overlap between the supports. We first establish bounds on thel∞-error as well sufficient conditions for exact variable selection for fixed design matrices, as well as for designs drawn randomly from general Gaussian distributions. Specializing to the caser = 2 linear regression problems with standard Gaussian designs whose supports overlap in a fraction α ∈ [0, 1] of their entries, we prove that l1/l∞-regularized method undergoes a phase transition characterized by the rescaled sample size θ1,∞(n, p, s, α) = n/{(4 − 3α)s log(p − (2 − α) s)}. An implication is that the use ofl1/l∞-regularization yields improved statistical efficiency if the overlap parameter is large enough ( α > 2/3), but has worse statistical efficiency than a naive Lasso-based approach for moderate to small overlap (α < 2/3). Empirical simulations illustrate the close agreement between theory and actual behavior in practice. These results show that caution must be exercised in applyingl1/l∞ block regularization: if the data does not match its structure very closely, it can impair statistical performance relative to computationally less expensive schemes.