Conformal correlators in the critical $O(N)$ vector model