During the operation of the railway system, it is inevitable that there will be track defects that affect train operation safety, especially the defects of the rail and fastener. Therefore, it is necessary to detect these defects in time to ensure the safety of train operations. In recent years, with the development of deep learning and computer vision technology, intelligent detection of track defects has made great progress. However, the existing methods not only suffer from the scarcity of defective samples but also their fine-grained recognition is low due to subtle differences between similar classes; furthermore, their real-time performance also needs to be improved. To solve these problems, this article presents a real-time defect detection method for track components based on instance segmentation. First, an improved lightweight instance segmentation network is proposed for the segmentation and location of fastener and rail. Second, a method that detects fastener defects by analyzing the geometric features of the fastener masks is proposed, which can overcome the scarcity of defective fastener samples. A cascade defect detection network is proposed to realize multiresolution and high-resolution detection of rail, which improves the accuracy of rail defect detection. Finally, the TensorRT inference framework is used to accelerate the defect detection network and realize edge end deployment. The experimental results show that the proposed method can achieve 95.1% recall rate of fastener defects, 98% detection rate of rail defects, and 93.5% classification accuracy of rail defects; 33.4 FPS defect detection speed is implemented in the Jetson AGX Xavier embedded device, which verifies the accuracy and real-time performance of the proposed method.