Joint optimization of signal and filter in the problems of extraction of signals from interfering reflections