Noise Correlation Effect on Detection: Signals in Equicorrelated or Autoregressive(1) Gaussian

In this letter, we consider the effect of noise correlation on the error performance of binary hypothesis signal detection, when one of two deterministic signals is received in correlated Gaussian noise. For the likelihood ratio detection scheme, analytical performance results are derived for equicorrelated and autoregressive order one models. Although it is known previously that the best signal lies in the direction of eigenvector corresponding to the minimum eigenvalue of the noise covariance matrix, our investigation of the variation of mean signal-to-noise power ratio as a function of correlation parameter (i) shows how correlation leads to increased probability of error up to a point, beyond which monotonic decrease in error probability with increasing correlation is possible and (ii) provides a max–min signal design solution for the unknown correlation parameter case. Numerical results are also included for some specific signals.