The art of fitting p-mode spectra. II. Leakage and noise covariance matrices