You can't pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM