Audio Visual Scene-Aware Dialog Track in DSTC8

Dialog systems need to understand scenes in order to have conversations with users about the objects and events in them. We introduced a new challenge task and dataset for Audio Visual Scene-Aware Dialog (AVSD) in DSTC7, which was the first attempt to combine conversation and multimodal video description into a single end-to-end differentiable network to build scene-aware dialog systems. The winning system of the challenge applied hierarchical attention mechanisms to combine text and visual information, yielding a relative improvement of 22% in the human rating of the output of the winning system vs. that of the baseline system. The language models trained from QA contributed most to this improvement, while the benefits from visual models (such as object or event recognition from videos) were limited. This means that there is still more opportunity to boost performance on the AVSD challenge using video features. To encourage such progress, we propose a second challenge for DSTC8 as a follow-up to the video-based sceneaware dialog track from DSTC7. The task is to generate or select a system response to a query that occurs during a dialog about a video. Participants will use the video, audio, and dialog text data to train end-to-end models. We will again use the AVSD data sets that we collected and used at DSTC7. We may also include additional data sets, such as the How2 and Dense-Captioning datasets, to provide example dialogs that have more long-term history dependence.