Method for determining number of coherent signals in the presence of colored noise