Trust, but Verify: Using Self-supervised Probing to Improve Trustworthiness