Twisted $L^2$-torsion on the character variety