Code Completion with Neural Attention and Pointer Networks

Intelligent code completion has become an essential tool to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and investigate the effectiveness of attention mechanism on the code completion task. However, standard neural language models even with attention mechanism cannot correctly predict out-of-vocabulary (OoV) words thus restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer networks which can reproduce words from local context, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or copy an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.

[1]  Martin T. Vechev,et al.  Probabilistic model for code with decision trees , 2016, OOPSLA.

[2]  Martin White,et al.  Toward Deep Learning Software Repositories , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[3]  Earl T. Barr,et al.  Learning Python Code Suggestion with a Sparse Pointer Network , 2016, ArXiv.

[4]  Tao Wang,et al.  Convolutional Neural Networks over Tree Structures for Programming Language Processing , 2014, AAAI.

[5]  Richard Socher,et al.  Pointer Sentinel Mixture Models , 2016, ICLR.

[6]  Wang Ling,et al.  Latent Predictor Networks for Code Generation , 2016, ACL.

[7]  Rob Miller,et al.  Code Completion from Abbreviated Input , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[8]  Bowen Zhou,et al.  Pointing the Unknown Words , 2016, ACL.

[9]  Eran Yahav,et al.  Code completion with statistical language models , 2014, PLDI.

[10]  K. Steiner.  Volume 49 , 2001 .

[11]  Zhendong Su,et al.  On the naturalness of software , 2012, ICSE 2012.

[12]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[13]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[14]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[15]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[16]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[17]  Dawn Song,et al.  Neural Code Completion , 2017 .

[18]  Christof Monz,et al.  Recurrent Memory Networks for Language Modeling , 2016, NAACL.

[19]  Premkumar T. Devanbu,et al.  On the naturalness of software , 2016, Commun. ACM.

[20]  Premkumar T. Devanbu,et al.  On the localness of software , 2014, SIGSOFT FSE.

[21]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[22]  Charles A. Sutton,et al.  A Convolutional Attention Network for Extreme Summarization of Source Code , 2016, ICML.

[23]  Hang Li,et al.  “ Tony ” DNN Embedding for “ Tony ” Selective Read for “ Tony ” ( a ) Attention-based Encoder-Decoder ( RNNSearch ) ( c ) State Update s 4 SourceVocabulary Softmax Prob , 2016 .

[24]  Jian Li,et al.  Software Defect Prediction via Convolutional Neural Network , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[25]  Navdeep Jaitly,et al.  Pointer Networks , 2015, NIPS.

[26]  Martin T. Vechev,et al.  PHOG: Probabilistic Model for Code , 2016, ICML.

[27]  Mirella Lapata,et al.  Long Short-Term Memory-Networks for Machine Reading , 2016, EMNLP.

[28]  Quoc V. Le,et al.  Addressing the Rare Word Problem in Neural Machine Translation , 2014, ACL.

[29]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[30]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[31]  Charles A. Sutton,et al.  Mining idioms from source code , 2014, SIGSOFT FSE.

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.