Limits on Early Matter Domination from the Isotropic Gamma-Ray Background