Permutations and the combinatorics of gauge invariants for general N