Robust Limits on Lorentz Violation from Gamma-Ray Bursts