Improved Retrieval of Programming Solutions With Code Examples Using a Multi-featured Score

Developers often depend on code search engines to obtain solutions for their programming tasks. However, finding an expected solution containing code examples along with their explanations is challenging due to several issues. There is a vocabulary mismatch between the search keywords (the query) and the appropriate solutions. Semantic gap may increase for similar bag of words due to antonyms and negation. Moreover, documents retrieved by search engines might not contain solutions containing both code examples and their explanations. So, we propose CRAR (Crowd Answer Recommender) to circumvent those issues aiming at improving retrieval of relevant answers from Stack Overflow containing not only the expected code examples for the given task but also their explanations. Given a programming task, we investigate the effectiveness of combining information retrieval techniques along with a set of features to enhance the ranking of important threads (i.e., the units containing questions along with their answers) for the given task and then selects relevant answers contained in those threads, including semantic features, like word embeddings and sentence embeddings, for instance, a Convolutional Neural Network (CNN). CRAR also leverages social aspects of Stack Overflow discussions like popularity to select relevant answers for the tasks. Our experimental evaluation shows that the combination of the different features performs better than each one individually. We also compare the retrieval performance with the state-of-art CROKAGE (Crowd Knowledge Answer Generator), which is also a system aimed at retrieving relevant answers from Stack Overflow. We show that CRAR outperforms CROKAGE in Mean Reciprocal Rank and Mean Recall with small and medium effect sizes, respectively. Email addresses: rodrigofernandes@ufu.br (Rodrigo F. Silva), masud.rahman@dal.ca (Mohammad Masudur Rahman), carlos.dantas@ufu.br (Carlos Eduardo Dantas), chanchal.roy@usask.ca (Chanchal Roy), foutse.khomh@polymtl.ca (Foutse Khomh), marcelo.maia@ufu.br (Marcelo A. Maia) Preprint submitted to Journal of Systems and Software October 24, 2021 ar X iv :2 10 8. 02 70 2v 1 [ cs .S E ] 5 A ug 2 02 1

[1]  Sushil Krishna Bajracharya,et al.  Analyzing and mining a code search engine usage log , 2010, Empirical Software Engineering.

[2]  Chanchal Kumar Roy,et al.  RACK: Code Search in the IDE Using Crowdsourced Knowledge , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[3]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[4]  Junfeng Zhao,et al.  Toward accurate link between code and software documentation , 2018, Science China Information Sciences.

[5]  Koushik Sen,et al.  Retrieval on source code: a neural code search , 2018, MAPL@PLDI.

[6]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[7]  P. Alam ‘S’ , 2021, Composites Engineering: An A–Z Guide.

[8]  Carlo Strapparava,et al.  Corpus-based and Knowledge-based Measures of Text Semantic Similarity , 2006, AAAI.

[9]  Amiram Yehudai,et al.  Example Overflow: Using social media for code recommendation , 2012, 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE).

[10]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[11]  Christoph Treude,et al.  SOTorrent: Reconstructing and Analyzing the Evolution of Stack Overflow Posts , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[12]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[13]  Chanchal Kumar Roy,et al.  STRICT: Information retrieval based search term identification for concept location , 2017, 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[14]  Martin P. Robillard,et al.  Selection and presentation practices for code example summarization , 2014, SIGSOFT FSE.

[15]  Thomas de Quincey [C] , 2000, The Works of Thomas De Quincey, Vol. 1: Writings, 1799–1820.

[16]  Gabriele Bavota,et al.  Mining StackOverflow to turn the IDE into a self-confident programming prompter , 2014, MSR 2014.

[17]  It Informatics Bing (Search Engine) , 2011 .

[18]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[19]  Cristina V. Lopes,et al.  Software reuse through methodical component reuse and amethodical snippet remixing , 2012, CSCW.

[20]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[21]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[22]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.

[23]  Xiaochen Li,et al.  Query Expansion Based on Crowd Knowledge for Code Search , 2016, IEEE Transactions on Services Computing.

[24]  Marcelo de Almeida Maia,et al.  CROKAGE: effective solution recommendation for programming tasks by leveraging crowd knowledge , 2020, Empir. Softw. Eng..

[25]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[26]  Mohammad Masudur Rahman Supporting Source Code Search with Context-Aware and Semantics-Driven Query Reformulation , 2019 .

[27]  Matteo Pagliardini,et al.  Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features , 2017, NAACL.

[28]  Jennifer J. Richler,et al.  Effect size estimates: current use, calculations, and interpretation. , 2012, Journal of experimental psychology. General.

[29]  Zhenchang Xing,et al.  API Method Recommendation without Worrying about the Task-API Knowledge Gap , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[30]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[31]  Xiaodong Gu,et al.  Deep Code Search , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[32]  Christoph Treude,et al.  Augmenting API Documentation with Insights from Stack Overflow , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[33]  Chanchal Kumar Roy,et al.  Recommending insightful comments for source code using crowdsourced knowledge , 2015, 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[34]  Chanchal Kumar Roy,et al.  Effective Reformulation of Query for Code Search Using Crowdsourced Knowledge and Extra-Large Data Analytics , 2018, 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[35]  Diomidis Spinellis,et al.  Semantic Source Code Models Using Identifier Embeddings , 2019, 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR).

[36]  Marcelo de Almeida Maia,et al.  Recommending Comprehensive Solutions for Programming Tasks by Mining Crowd Knowledge , 2019, 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC).

[37]  Xiao Ma,et al.  From Word Embeddings to Document Similarities for Improved Information Retrieval in Software Engineering , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[38]  Stephen E. Robertson,et al.  Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval , 1994, SIGIR '94.

[39]  Ying Zou,et al.  Learning to rank code examples for code search engines , 2017, Empirical Software Engineering.

[40]  Chanchal Kumar Roy,et al.  Improved query reformulation for concept location using CodeRank and document structures , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[41]  Parosh Aziz Abdulla,et al.  Replication Package for the paper: Optimal Stateless Model Checking under the Release-Acquire Semantics , 2018 .

[42]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[43]  Zhenchang Xing,et al.  AnswerBot: Automated generation of answer summary to developers' technical questions , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[44]  Chanchal Kumar Roy,et al.  RACK: Automatic API Recommendation Using Crowdsourced Knowledge , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[45]  Manaal Faruqui,et al.  Non-distributional Word Vector Representations , 2015, ACL.

[46]  Christoph Treude,et al.  NLP2Code: Code Snippet Content Assist via Natural Language Tasks , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[47]  Yves Le Traon,et al.  Learning to Spot and Refactor Inconsistent Method Names , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[48]  Gabriele Bavota,et al.  Automatic query reformulations for text retrieval in software engineering , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[49]  Koushik Sen,et al.  When deep learning met code search , 2019, ESEC/SIGSOFT FSE.

[50]  Marcelo de Almeida Maia,et al.  A Replication Package For The Paper "Improved Retrieval of Programming Solutions with Code Examples Using a Multi-featured Score" , 2021 .

[51]  Klaus Krippendorff,et al.  Computing Krippendorff's Alpha-Reliability , 2011 .

[52]  Themistoklis G. Diamantopoulos,et al.  Employing Source Code Information to Improve Question-Answering in Stack Overflow , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.