Show, Observe and Tell: Attribute-driven Attention Model for Image Captioning

Attribute-based approaches and attention-based approaches have been proven to be effective in image captioning. However, most attribute-based approaches simply predict attributes independently without taking the co-occurrence dependencies among attributes into account. Most attention-based captioning models directly leverage the feature map extracted from CNN, in which many features may be redundant in relation to the image content. In this paper, we propose an attribute-driven attention model for image captioning. We focus on training a good attribute-inference model via the recurrent neural network (RNN) for image captioning, where the co-occurrence dependencies among attributes can be maintained. The uniqueness of our inference model lies in the usage of a RNN with the visual attention mechanism to observe the image before generating captions. Additionally, it is noticed that compact and attribute-driven features will be more useful for the attention-based captioning model. Therefore, we extract the context feature for each attribute, and enable the captioning model to adaptively attend to these context features. We verify the effectiveness and superiority of the proposed approach over other captioning approaches by conducting massive experiments and comparisons on the MS COCO image captioning dataset.