U(2) flavor physics without U(2) symmetry