LASSO risk and phase transition under dependence

We consider the problem of recovering a k-sparse signal β0 ∈ Rp from noisy observations y = Xβ0+w ∈ Rn. One of the most popular approaches is the l1-regularized least squares, also known as LASSO. We analyze the mean square error of LASSO in the case of random designs in which each row of X is drawn from distribution N(0,Σ) with general Σ. We first derive the asymptotic risk of LASSO in the limit of n, p → ∞ with n/p → δ. We then examine conditions on n, p, and k for LASSO to exactly reconstruct β0 in the noiseless case w = 0. A phase boundary δc = δ(ǫ) is precisely established in the phase space defined by 0 ≤ δ, ǫ ≤ 1, where ǫ = k/p. Above this boundary, LASSO perfectly recovers β0 with high probability. Below this boundary, LASSO fails to recover β0 with high probability. While the values of the non-zero elements of β0 do not have any effect on the phase transition curve, our analysis shows that δc does depend on the signed pattern of the nonzero values of β0 for general Σ 6= Ip. This is in sharp contrast to the previous phase transition results derived in i.i.d. case with Σ = Ip where δc is completely determined by ǫ regardless of the distribution of β0. Underlying our formalism is a recently developed efficient algorithm called approximate message passing (AMP) algorithm. We generalize the state evolution of AMP from i.i.d. case to general case with Σ 6= Ip. Extensive computational experiments confirm that our theoretical predictions are consistent with simulation results on moderate size system.