Finite groups in which elements of the same order outside the center are conjugate