On permutation characters and Sylow $p$-subgroups of $\mathfrak{S}_n$