Translating video into language by enhancing visual and language representations

Abstract It is a fundamental task of translating videos into natural language automatically by computer. At present, the models for video description based on deep learning have made a great breakthrough. However, the static information loss is serious during encoding stage for motion feature of videos, and the linguistic feature from LSTM network lack personalized expression, leading to inappropriate words and poor semantics in generation sentences. In this work, a model with enhanced features of visual and language is proposed to address the challenges. First, static features of video frames from the first LSTM layer are incorporated, then fed into another LSTM layer according by frame sequence. Second, the feature of word is combined with the output of LSTM network for predicted probability of candidate word on each time step. The experimental results demonstrate effectiveness of the proposed approach with competitive performance compared with other state-of-the-art methods on various metrics.