Low computationally complex recurrent neural network for high speed optical fiber transmission

Abstract The demand for high speed data transmission has increased rapidly over the past few years, leading to the development of the data center concept. Considering that vertical cavity surface emitting lasers (VCSELs) based optical interconnect is evolving to 100 Gb/s, relative intensity noise and mode partition noise are becoming significant, which cannot be equalized by conventional equalizers efficiently. On the other hand, optical fiber suffers from inter-symbol interference (ISI) and nonlinear channel response, making the equalization process even more challenging. Recently, several machine learning techniques have already been applied to recover signal from nonlinear distortions. However, previous experiments mainly focused on achieving low BER and have neglected the computational complexity. In this paper, we have designed a recurrent neural network (RNN) based equalizer. The equalizer is tested over a PAM-4 modulated VCSEL-MMF optical interconnect link, and shows BER performance improvement over equalizers based on ANN. A variant, Half-RNN, is also proposed and tested, whose computational complexity is 70% lower than ANN with similar BER performance. Our experiments provide guidance for designing neural network structure when using deep learning for equalization, justifying the significance of our work.