Performance Analysis of $M$-PSK/ $M$-QAM Modulated Orthogonal Space–Time Block Codes in Keyhole Channels

It is known that the keyhole or pinhole effect can severely degrade the performance of multiple-input-multiple-output (MIMO) communication systems. In this paper, the outage probability and error probability of orthogonal space-time block codes with <i>m</i> transmit and <i>n</i> receive antennas over keyhole fading channels are analyzed. Closed-form outage probability and symbol/bit error rate expressions are derived with <i>M</i>-ray phase-shift keying (<i>M</i>-PSK) and <i>M</i>-ray quadrature amplitude modulation (<i>M</i>-QAM) signaling. It is found that the exact error probability expressions have two different forms for <i>m</i> = <i>n</i> and <i>m</i> ne <i>n</i>, respectively. Furthermore, when <i>m</i> ne <i>n</i>, approximate error rate expressions are derived at high signal-to-noise ratios (SNRs), whereas there are no asymptotic expressions for <i>m</i> = <i>n</i>. Simulation results are provided to verify our analytical results.