Iterative demodulation and decoding of coded non-square QAM