Detection of High Baud-Rate Signals With Pattern Dependent Distortion Using Hidden Markov Modeling

In high baud-rate systems, the bandwidth limitations and nonlinearities of drive amplifiers and optical modulators can introduce pattern dependent distortion (PDD) that limits system performance. One solution entails detecting the transmitted symbols with the aid of a look-up table (LUT) containing prototypes of the PDD degraded signal. To improve the performance-complexity trade-offs of this approach, we model the PDD degraded signal as drawn from a hidden Markov model (HMM). Detection of the transmitted symbols given the received signal is performed by finding the HMM state sequence that emits the received signal with maximum a posteriori probability (MAP). Computational complexity is kept manageable by using the Viterbi algorithm to find the MAP state sequence, and by simplifying the HMM emission probability functions to produce variants of the algorithm. The resultant set of algorithm variants subsume a few recent LUT-based nonsequential detection schemes. In a back-to-back experiment, the proposed solutions demonstrate 6-fold lower computational complexity, compared to their nonsequential counterparts for the same target bit error ratio. In a 3 $\times$ 402 Gb/s dual-polarization 16-QAM superchannel transmission experiment, the sequential approach offers a 37% reach extension over a nonsequential LUT-based benchmark algorithm. Overall, HMM-based sequential detection offers superior performance-complexity trade-offs over the LUT-based nonsequential detection algorithms.