With the continuous development of rail transit fully automatic operation, the urgent need to improve train operation safety makes obstacle detection become the research focus. In this work, a flexible and efficient multiscale one-stage object detector FE-YOLO was proposed for image obstacle detection. The feature extraction network is composed of attention module, downsampling module, residual block, spatial pyramid pooling (SPP) module, and so on. A repeatable bidirectional cross-scale path aggregation module was designed as the core of the feature fusion network. The dataset RT2021 of rail transit obstacles was constructed based on the real scene. The mean of average precision (mAP), detection time, iteration time, parameters, and anti-interference ability were used to compare FE-YOLO with other classic object detectors. The results showed that FE-YOLO has the best comprehensive performance. The mAP can reach 92.57%, and the single-frame detection time on the onboard embedded device is up to 0.0989 s. The ablation experiment verified the effectiveness of each module of FE-YOLO and has the best generalization performance on the PASCAL VOC dataset. Finally, a rail transit obstacle detection system was developed, and multiple sensors were used to improve the detection accuracy. Experiments showed that the detection system can work normally in different environments.