On Prompt-Driven Safeguarding for Large Language Models