On special representations of $p$-adic reductive groups