Scarcity of congruences for the partition function