A New Word Embedding Approach to Evaluate Potential Fixes for Automated Program Repair

Debugging is frequently a manual and costly task. Some work recently presented automated program repair methods aiming to reduce debugging time. Despite different approaches, the process of evaluating source code patches (potential fixes) is crucial for most of them, e.g., generate-and-validate systems. The evaluation is a complex task given that patches with different syntaxes might share the same semantics, behaving equally for typically limited specifications. Hence, many approaches fail to better explore the search space of patches, leading them to not reach the patch that fixes the bug. Some research points that a buggy code is more entropic, i.e., less natural than its fixed version. So, this work proposes applying Word2vec, a word embedding model, to improve the repair evaluation process based on the naturalness obtained from a corpus of known fixes. Word2vec captures co-occurrence relationships between words in a given context and then predicts the contextual words of a given word. This technique has been applied to deal with richer semantic relationships in a text. Word2vec evaluates patches according to distances of document vectors and a softmax output layer. We analyze the performance of our proposal with mutated patches created from correct source codes and we simulate potential fixes generated by automated program repair approaches. Thus, the main contribution of this paper is a new method to evaluate patches used in automated program repair methods. The results show that Word2vec-based metrics are capable of analyzing source code naturalness and be used to evaluate source code patches.