Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding