Symmetries of abelian Chern-Simons theories and arithmetic