Improving Language Models via Plug-and-Play Retrieval Feedback