Unconventional magnetic phase transition in antiferromagnetic heavy-fermion YbIrGe