Aiming at the current call number extraction technology that only deals with the useful area, which is greatly affected by complex color background factors, a method for locating and extracting call numbers from randomly collected images of books on the shelf under a complex color background is designed. According to the color characteristics at the edge of the call number label, the horizontal projection analysis is used to analyze the image features, filter out other color background interference, and locate the call number area. Combine mathematical morphology and iterative global threshold segmentation algorithm to segment and extract the call number area. The experimental results show that the algorithm's Intersection-over-Union (IOU) index reaches 89.91% and the Precision index reaches 96.11%. Compared with the commonly used algorithms, they are respectively increased by 29.94% and 31.08%, which can effectively extract and segment the call number area without being affected by complex color background, with high efficiency and strong anti-interference.