Hierarchical deep semantic representation for visual categorization

Abstract Visual features are unsatisfactory to effectively describe the visual semantics. However, single layer based semantic modeling may be not able to cope with complicated semantic contents. In this paper, we propose Hierarchical Deep Semantic Representation (H-DSR), a hierarchical framework which combines semantic context modeling with visual features. First, the input image is sampled with spatially fixed grids. Deep features are then extracted for each sample in particular location. Second, using pre-learned classifiers, a detection response map is constructed for each patch. Semantic representation is then extracted from the map, which have a sense of latent semantic context. We combine the semantic and visual representations for joint representation. Third, a hierarchical deep semantic representation is built with recurrent reconstructions using three layers. The concatenated visual and semantic representations are used as the inputs of subsequent layers for semantic representation extraction. Finally, we verify the effectiveness of H-DSR for visual categorization on two publicly available datasets: Oxford Flowers 17 and UIUC-Sports. Improved performances are obtained over many baseline methods.