Higher loop nonplanar anomalous dimensions from symmetry