One of the critical and costly components in digital cellular communication systems is the RF power amplifier. Theoretically, one of the main concerns in an RF power amplifier design is the nonlinear effect of the amplifier. Quantitatively, so far, no explicit relationship or expression currently exists between the out-of-band emission level and the nonlinearity description related to the third-order intercept point (IP3). Further, in experiments and analysis, it was discovered that, in some situations, using IP3 only is not accurate enough to describe the spectrum regrowth, especially when the fifth-order intercept point (IP5) is relatively significant compared to the third-order intermodulation. In this article, we analyze the nonlinear effect of an RF power amplifier in the MIRS M-16-QAM system, give an expression of the estimation of the out-of-band emission levels for an MIRS signal power spectrum in terms of the IP3 and the IP5, as well as the power level of the signal. This result will be useful in the design of RF power amplifier for the MIRS M-16-QAM system and other wireless communication systems.