We report on an extensive study of the influence of spin anisotropy on spin glass aging dynamics. New temperature cycle experiments allow us to compare quantitatively the memory effect in four Heisenberg spin glasses with various degrees of random anisotropy and one Ising spin glass. The sharpness of the memory effect appears to decrease continuously with the spin anisotropy. Besides, the spin glass coherence length is determined by magnetic field change experiments for the first time in the Ising sample. For three representative samples, from Heisenberg to Ising spin glasses, we can consistently account for both sets of experiments (temperature cycle and magnetic field change) using a single expression for the growth of the coherence length with time.