Are LLMs Robust for Spoken Dialogues?