Matter-gravity couplings and Lorentz violation