In the 5G era, user equipment connected to 5G base stations can obtain better communication services. However, due to the limited coverage of base stations, the movement of users may cause frequent handover of base stations. With the widespread deployment of 5G base stations, how to reduce unnecessary handover times when users connect to the base station becomes particularly important. In recent years, user trajectory data has been mined and applied to many scenarios. In the 5G network, by judging the user’s movement trajectory, the number of handovers required for the user to connect to the 5G base station can be effectively reduced. In this paper, we propose a 5G base station handover method based on trajectory prediction. A CNN-LSTM neural network, which combines a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) has been proposed to predict the user’s trajectory. The evaluation results show that our mechanism can effectively reduce the number of base station handovers and improve the efficiency of users using the network. In addition, the stability of 5G networks can be improved by reducing inefficient base station handover.