Modulation spectroscopy of Rydberg atoms in an optical lattice