Are Prompt-based Models Clueless?