You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM