Numerical optimization on the rotation group