Multimodal Attention for Fusion of Audio and Spatiotemporal Features for Video Description

Video description approaches are based on encoderdecoder sentence generation using recurrent neural networks (RNNs) by integrating temporal attention mechanisms into each model, in which the decoder network predicts each word in the description by selectively giving more weight to encoded features from specific time frames. We incorporate audio features, in addition to image and motion features, for video description based on encoder-decoder recurrent neural networks (RNNs). To fuse these modalities, we introduce a multimodal attention model that can selectively utilize features from different modalities for each word in the output description. We apply our new framework for video description using MFCC and state-of-the-art audio features such as SoundNet. Results confirm that our attention-based multimodal fusion of audio features with visual features outperforms conventional video description approaches.