An extremal problem in Fourier analysis with applications to operator theory