Off-forward anomalous dimensions in the leading-$n_f$ limit