Minimally non-local nucleon-nucleon potentials in $\chi$EFT at order $Q^4$