WheaCha: A Method for Explaining the Predictions of Code Summarization Models

The last decade has witnessed a rapid advance in learning-based program analysis tools. While the black-box nature of these systems may produce powerful predictions, it cannot be directly explained, posing a threat to the continuing application of machine learning technology in programming language research. Recently, attribution methods have emerged as a popular approach to interpreting model predictions based on the relevance of input features. Although their feature importance ranking can provide insights of how models arrive at a prediction from a raw input, they do not give a clear-cut definition of the key features models use for the prediction. In this paper, we present a new method, called WheaCha, for explaining the predictions of code summarization models, a broad class of code models that predict the name of a method given its body. Although WheaCha employs the same mechanism of tracing model predictions back to the input features, it differs from all existing attribution methods in crucial ways. Specifically, WheaCha divides an input method into “wheat” (i.e., the defining features that are the reason for which models predict the label that they predict) and the rest “chaff ” (i.e., the auxiliary features that may help models predict the label that they predict) for any prediction of a learned code summarization model. We realizeWheaCha in a tool, HuoYan, and use it to explain four prominent code summarization models: code2vec, code2seq, sequence GNN, and extreme summarizer. Results show (1) HuoYan is efficient — taking on average under fifteen seconds to compute the wheat for an input method in an end-to-end fashion (i.e., including the model prediction time); (2) the wheat that all models use to predict input methods is made of simple syntactic or even lexical properties (i.e., identifier names), nevertheless, they are often intuitive and well-aligned to the features humans would have used for inferring the name of the same methods; (3) some of the most noteworthy attribution methods do not precisely capture the wheat that models use for prediction. Finally, we show wheat naturally lends itself to a novel, powerful attack mechanism on code summarization models. An extensive evaluation shows our attack mechanism significantly outperforms the state-of-the-art in both non-targeted and targeted attacks, and more importantly, in many cases renders a popular defense technique — adversarial training — almost non-existent. Our work opens up this exciting, new direction of studying what models have learned from source code.