Beyond Moments: Robustly Learning Affine Transformations with Asymptotically Optimal Error