Higher spin quasinormal modes and one-loop determinants in the BTZ black hole