Ultralow Complexity Long Short-Term Memory Network for Fiber Nonlinearity Mitigation in Coherent Optical Communication Systems

Fiber Kerr nonlinearity is a fundamental limitation to the achievable capacity of long-distance optical fiber communication. Digital backpropagation (DBP) is a primary methodology to mitigate both linear and nonlinear impairments by solving the inverse-propagating nonlinear Schrödinger equation (NLSE), which requires detailed link information. Recently, the paradigms based on neural network (NN) were proposed to mitigate nonlinear transmission impairments in optical communication systems. However, almost all neural network-based equalization schemes yield high computation complexity, which prevents the practical implementation in commercial transmission systems. In this paper, we propose a center-oriented long short-term memory network (Co-LSTM) incorporating a simplified mode with a recycling mechanism in the equalization operation, which can mitigate fiber nonlinearity in coherent optical communication systems with ultralow complexity. To validate the proposed methodology, we carry out an experiment of ten-channel wavelength division multiplexing (WDM) transmission with 64 Gbaud polarization-division-multiplexed 16-ary quadrature amplitude modulation (16-QAM) signals. Co-LSTM and DBP achieve a comparable performance of nonlinear mitigation. However, the complexity of Co-LSTM with a simplified mode is almost independent of the transmission distance, which is much lower than that of the DBP. The proposed Co-LSTM methodology presents an attractive approach for low complexity nonlinearity mitigation with neural networks. Fiber Kerr nonlinearity imposes a fundamental limitation to the achievable transmission distance and information capacity of optical fiber communication1. This limitation is mainly attributed to deterministic Kerr fiber nonlinearities and the interaction of fiber nonlinearity with amplified spontaneous emission noise from cascaded optical amplifiers2-3, which is identified as a nonlinear Shannon capacity limit. Overcoming fiber nonlinearity is one of the most challenging tasks to extend the capacity and the transmission distance of optical fiber communication systems. To mitigate nonlinear effects in optical fibers, several nonlinear compensation (NLC) techniques have been developed, such as digital backpropagation (DBP)4, Volterra series filtering5, perturbation-based compensation6, optical phase conjugation (OPC)7-8, and phase-conjugated twin waves (PCTW)9. DBP is a primary methodology to mitigate both linear and nonlinear impairments by solving the inverse-propagating nonlinear Schrödinger equation (NLSE). However, DBP is not feasible for commercial implementation due to its high complexity, especially for long-haul and/or multi-channel systems. The Volterra filtering and the perturbation method also demand high computation resources due to the operation of the summarization of nonlinear triplets. The requirement of additional optical apparatus for phase conjugation at the mid-span of the fiber link increases both the cost and the flexibility of the OPC technique. For PCTW, it is also impractical since the data rate is reduced by half due to the conjugated twin waves. Recently, machine learning (ML) techniques have been proposed to mitigate nonlinear transmission impairments in optical communication systems. The learned DBP technique was proposed by treating multi split-step Fourier iterations as neural network (NN) layers for parameter optimization10, which shows better performance than the conventional DBP method. NN with the designed nonlinear perturbation triplets was proposed to pre-distort symbols at the transmitter11. This approach has better performance than the filtered-DBP while achieving a slight complexity advantage. A complex NN approach designed with reference to the averaged Manakov equations was proposed to mitigate fiber nonlinearity12. The simulation results show that its performance is better than that of the DBP with 2 steps per span (StPs), but its complexity is much higher than that of DBP-2-StPs12. In addition, a bi-directional long short-term memory network (Bi-LSTM) was numerically studied for the mitigation of fiber nonlinearity in the coherent system13 and the nonlinear Fourier transform (NFT) based system14. The main advantage of Bi-LSTM is that they can efficiently handle inter-symbol-interference (ISI) among preceding and succeeding symbols caused by chromatic dispersion. However, Bi-LSTM demands high computation resources due to its iterative calculation based on time steps. Thousands or even more multiplications are required in long-distance optical transmission systems with Bi-LSTM equalization. 1State Key Laboratory of Advanced Optical Communication Systems and Networks, Frontiers Science Center for Nano-optoelectronics, Department of Electronics, Peking University, Beijing 100871, China 2Peng Cheng Laboratory, Shenzhen 518055, China *Corresponding author: fzhang@pku.edu.cn †These three authors contributed equally to this work. In this paper, we propose a novel center-oriented long short-term memory network (Co-LSTM) scheme incorporating a simplified mode based on a recycling mechanism for ultralow complexity operation. We verify our method in an experiment of 10×512Gb/s wavelength division multiplexing (WDM) transmission over 1600km standard single-mode fiber (SSMF) with polarization division multiplexing (PDM) 16-ary quadrature amplitude modulation (16-QAM). The experiments show that the Co-LSTM can effectively mitigate fiber nonlinearity with a 0.51dB Q2 factor gain and reduce the complexity to 5.2% of that of the Bi-LSTM and 28.4% of that of the DBP with 1 step per span. Our results suggest emerging low-complexity neural network-based nonlinear equalization in coherent optical transmission systems. The principle of LSTM. Recurrent neural networks (RNNs) are mainly used to process sequence data, which predict the output of the current state by encoding the current and the previous data. However, standard RNNs face a major practical problem: the gradient of the total output error with respect to previous inputs quickly vanishes as the time delays between relevant inputs and errors increase15. LSTM improves the internal structure of the standard RNNs and solves the problem of vanishing gradients16. Moreover, LSTM can lean long-term dependencies by enforcing constant error flow through constant error carousels (CECs) within special units16, 17. LSTML LSTML LSTML FCL