Classical black hole scattering from a worldline quantum field theory