Constraining the nature of the most distant gamma-ray burst host galaxies