SEP of the M-ary θ-QAM signals under η -μ fading and AWGN noise in a communication system using spatial diversity

In this paper, an upper bound on the symbol error probability (SEP) of the M-ary $$\theta $$-quadrature amplitude modulation scheme, in a channel subject to additive white Gaussian noise (AWGN) and $$\eta -\mu $$ fading, is calculated by means of the union bound. In addition, an exact and closed-form expression for the pairwise error probability (PEP) of the system is derived. The PEP expression obtained is written in terms of the Beta function and the Lauricella hypergeometric function. Another expression for the PEP, written in terms of the Beta function and the Appell hypergeometric function, and two bounds for the PEP, one lower and one upper, are also presented. An exact expression for the optimum rotation angle for quadrature phase shift keying constellation, considering this channel model, is also determined. In addition, an analysis, in terms of PEP, of the modulation diversity system combined with a maximum ratio combining receiver in channels subject to $$\eta -\mu $$ fading and AWGN noise, is performed. PEP and SEP curves, as a function of signal-to-noise ratio, are plotted and corroborated by simulations performed with the Monte Carlo method under different parameters that characterize the channel mathematically. All the expressions determined in this article are closed and new.