Robustness of subspace-based algorithms with respect to the distribution of the noise: Application to DOA estimation

Abstract This paper addresses the theoretical analysis of the robustness of subspace-based algorithms with respect to non-Gaussian noise distributions using perturbation expansions. Its purpose is twofold. It aims, first, to derive the asymptotic distribution of the estimated projector matrix obtained from the sample covariance matrix (SCM) for arbitrary distributions of the useful signal and the noise. It proves that this distribution depends only of the second-order statistics of the useful signal, but also on the second and fourth-order statistics of the noise. Second, it derives the asymptotic distribution of the estimated projector matrix obtained from any M-estimate of the covariance matrix for both real (RES) and complex elliptical symmetric (CES) distributed observations. Applied to the MUSIC algorithm for direction-of-arrival (DOA) estimation, these theoretical results allow us to theoretically evaluate the performance loss of this algorithm for heavy-tailed noise distributions when it is based on the SCM, which is significant for weak signal-to-noise ratio (SNR) or closely spaced sources. These results also make it possible to prove that this performance loss can be alleviated by replacing the SCM by an M-estimate of the covariance for CES distributed observations, which has been observed until now only by numerical experiments.