Symmetries and spectral statistics in chaotic conformal field theories