Impact of microstructure on the effective diffusivity in random packings of hard spheres