论文信息 - Ensembling Visual Explanations for VQA

Ensembling Visual Explanations for VQA

Explanations make AI systems more transparent and also justify their predictions. The top-ranked Visual Question Answering (VQA) systems are ensembles of multiple systems; however, there has been no work on generating explanations for such ensembles. In this paper, we propose different methods for ensembling visual explanations for VQA using the localization maps of the component systems. Our crowd-sourced human evaluation indicates that our ensemble visual explanation is superior to each of the individual system’s visual explanation, although the results vary depending on the individual system that the ensemble is compared against as well as the number of individual systems that agree with the ensemble model’s answer. Overall, our ensemble explanation is better 63% of the time when compared to any individual system’s explanation. Our algorithm is also efficient and scales linearly in the number of component systems in the ensemble.

Nazneen Fatema Rajani and Raymond J. Mooney

[1] Trevor Darrell,et al. Generating Visual Explanations , 2016, ECCV.

[2] Kate Saenko,et al. Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering , 2015, ECCV.

[3] Abhishek Das,et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[4] Trevor Darrell,et al. Attentive Explanations: Justifying Decisions and Pointing to the Evidence , 2016, ArXiv.

[5] Dan Klein,et al. Learning to Compose Neural Networks for Question Answering , 2016, NAACL.

[6] Margaret Mitchell,et al. VQA: Visual Question Answering , 2015, International Journal of Computer Vision.

[7] Jiasen Lu,et al. Hierarchical Question-Image Co-Attention for Visual Question Answering , 2016, NIPS.

[8] Wei Xu,et al. ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering , 2015, ArXiv.

[9] Trevor Darrell,et al. Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding , 2016, EMNLP.

[10] Yash Goyal,et al. Towards Transparent AI Systems: Interpreting Visual Question Answering Models , 2016, 1608.08974.

[11] Raymond J. Mooney,et al. Stacking With Auxiliary Features , 2016, IJCAI.