Spectral density estimation with amplitude modulation and outlier detection