SCANViz: Interpreting the Symbol-Concept Association Captured by Deep Neural Networks through Visual Analytics

Two fundamental problems in machine learning are recognition and generation. Apart from the tremendous amount of research efforts devoted to these two problems individually, finding the association between them has attracted increasingly more attention recently. Symbol-Concept Association Network (SCAN) is one of the most popular models for this problem proposed by Google DeepMind lately, which integrates an unsupervised concept abstraction process and a supervised symbol-concept association process. Despite the outstanding performance of this deep neural network, interpreting and evaluating it remain challenging. Guided by the practical needs from deep learning experts, this paper proposes a visual analytics attempt, i.e., SCANViz, to address this challenge in the visual domain. Specifically, SCANViz evaluates the performance of SCAN through its power of recognition and generation, facilitates the exploration of the latent space derived from both the unsupervised extraction and supervised association processes, empowers interactive training of SCAN to interpret the model’s understanding on a particular visual concept. Through concrete case studies with multiple deep learning experts, we validate the effectiveness of SCANViz.