Conformal holonomy, symmetric spaces, and skew symmetric torsion