Successive magnetic transitions in a frustrated compound YbAgGe