This letter presents an innovative approach to sense surface contamination severity of in-service metal oxide surge arrester through combining handcrafted and deep learning-based features. For this purpose, different color regions in the captured infrared thermal (IRT) images are isolated to generate superpixel regions through grouping of informative and meaningful pixels. Thereafter, handcrafted color features are extracted from each superpixel region. Additionally, the IRT images are analyzed through pretrained “ResNet50” deep architecture, and deep features are extracted. Thereafter, handcrafted color features and deep features are combined and fed to a classifier for classification of different surface contamination severity. The findings of this letter indicate that the proposed fusion strategy strongly boosts the classification performance compared to each stand-alone approach. Therefore, proposed framework with feature fusion strategy can be implemented for surface condition assessment of metal oxide surge arrester is service.