Semiconductor optical amplifier (SOA) is a promising solution for future O-band optical amplification. However, the SOA-induced nonlinearity will affect the quality of the signals and cause bit errors. Nowadays, the neural network is a hot topic in communication networks and has been considered a practical algorithm for signals equalization. In this paper, we investigate the performance of the three types of neural networks: deep neural networks (DNN), convolutional neural network (CNN), and long short term memory (LSTM) neural networks for the compensation of the SOA-induced nonlinearity. We first compare the fitting ability of the three neural networks on the saturated response of SOA. Based on the simulation result, LSTM has the best nonlinear expressiveness for the SOA-induced nonlinearity. Subsequently, we do some experimental validations in an intensity modulation and direct-detection (IM/DD) system, where the 100 Gbit/s pulse-amplitude-modulation-4 (PAM4) signals are transmitted over 80 km at O-band using an 18 GHz electro-absorption modulated laser (EML). The bit-error-ratio (BER) results show that the cell states in LSTM play a crucial role in nonlinearity compensation. Furthermore, LSTM offers the best equalization performance and can achieve a receiver sensitivity of −18.0 dBm.