This paper has proposed a real-time detection method for the handling mode of surge arrester. Surge arrester is one of the most important elements in power system, which is always used to protect electrical equipment from high transient overvoltage and to limit continuation time. However, because of its special structure, the lead line of the surge arrester is easy to be damaged in the process of construction. Under this circumstance, it is necessary to design a strict automatic detection method in the power system. Io order to satisfy these requirements, this paper combines the YOLOv3 object detection algorithm and the open-pose structure to monitor the surge arrester handling mode in the process of power system construction. It has been proved that the method designed in this paper has a good performance and is able to present an accurate judgement result.