Integrality of the higher Rademacher symbols