The VANDELS Survey: new constraints on the high-mass X-ray binary populations in normal star-forming galaxies at 3 < z < 5.5