Learning to Rank for Question-Oriented Software Text Retrieval (T)

Question-oriented text retrieval, aka natural language-based text retrieval, has been widely used in software engineering. Earlier work has concluded that questions with the same keywords but different interrogatives (such as how, what) should result in different answers. But what is the difference? How to identify the right answers to a question? In this paper, we propose to investigate the "answer style" of software questions with different interrogatives. Towards this end, we build classifiers in a software text repository and propose a re-ranking approach to refine search results. The classifiers are trained by over 16,000 answers from the StackOverflow forum. Each answer is labeled accurately by its question's explicit or implicit interrogatives. We have evaluated the performance of our classifiers and the refinement of our re-ranking approach in software text retrieval. Our approach results in 13.1% and 12.6% respectively improvement with respect to text retrieval criteria nDCG@1 and nDCG@10 compared to the baseline. We also apply our approach to FAQs of 7 open source projects and show 13.2% improvement with respect to nDCG@1. The results of our experiments suggest that our approach could find answers to FAQs more precisely.

[1]  Nitesh V. Chawla,et al.  Information Gain, Correlation and Support Vector Machines , 2006, Feature Extraction.

[2]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[3]  Jian Pei,et al.  MAPO: Mining and Recommending API Usage Patterns , 2009, ECOOP.

[4]  Bing Xie,et al.  Interrogative-guided re-ranking for question-oriented software text retrieval , 2014, ASE.

[5]  Tetsuya Takiguchi,et al.  Towards Domain Independent Why Text Segment Classification Based on Bag of Function Words , 2012, Australasian Conference on Artificial Intelligence.

[6]  Gerald Reif,et al.  Supporting developers with natural language queries , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[7]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[8]  A. Leelaphattarakij,et al.  Ranking application software retrieval results using application attributes and context , 2012, 2012 Ninth International Conference on Computer Science and Software Engineering (JCSSE).

[9]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR Forum.

[10]  Tie-Yan Liu,et al.  Learning to Rank for Information Retrieval , 2011 .

[11]  Feng Xu,et al.  Want a Good Answer? Ask a Good Question First! , 2013, ArXiv.

[12]  Hans van Halteren,et al.  Learning to rank for why-question answering , 2011, Information Retrieval.

[13]  David Lo,et al.  Finding relevant answers in software forums , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[14]  Stanley Letovsky,et al.  Cognitive processes in program comprehension , 1986, J. Syst. Softw..

[15]  Changsheng Liu,et al.  Finding the merits and drawbacks of software resources from comments , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[16]  Lijie Wang,et al.  A Semi-supervised Approach for Component Recommendation Based on Citations , 2011, ICSR.

[17]  Ryuichiro Higashinaka,et al.  Corpus-based Question Answering for why-Questions , 2008, IJCNLP.

[18]  David Lo,et al.  Identifying Linux bug fixing patches , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[19]  Ling Yin.  A Two-Stage Approach to Retrieving Answers for How-To Questions , 2006, EACL.

[20]  Silvio Romero de Lemos Meira,et al.  Combining rule-based and information retrieval techniques to assign software change requests , 2014, ASE.

[21]  Ling Yin,et al.  Adapting the Naive Bayes Classifier to Rank Procedural Texts , 2006, ECIR.

[22]  David Madigan,et al.  Large-Scale Bayesian Logistic Regression for Text Categorization , 2007, Technometrics.

[23]  Christoph Treude,et al.  How do programmers ask and answer questions on the web?: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[24]  Tie-Yan Liu,et al.  A Theoretical Analysis of NDCG Type Ranking Measures , 2013, COLT.

[25]  David Lo,et al.  Active code search: incorporating user feedback to improve code search relevance , 2014, ASE.

[26]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[27]  Zengchang Qin,et al.  Question Classification using Head Words and their Hypernyms , 2008, EMNLP.

[28]  Kathryn T. Stolee,et al.  Solving the Search for Source Code , 2014, ACM Trans. Softw. Eng. Methodol..

[29]  Y. Li,et al.  Result Refinement in Web Services Retrieval Based on Multiple Instances Learning , 2008, Int. J. Web Serv. Res..

[30]  Martin Porter,et al.  Snowball: A language for stemming algorithms , 2001 .

[31]  Frank Maurer,et al.  What makes a good code example?: A study of programming Q&A in StackOverflow , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[32]  Michele Lanza,et al.  Extracting structured data from natural language documents with island parsing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[33]  Harris Drucker,et al.  Support vector machines for spam categorization , 1999, IEEE Trans. Neural Networks.

[34]  Gabriele Bavota,et al.  Automatic query reformulations for text retrieval in software engineering , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[35]  Yuanzhi Li,et al.  A Theoretical Analysis of NDCG Ranking Measures , 2013 .

[36]  Suzan Verberne,et al.  What Is Not in the Bag of Words for Why-QA? , 2010, CL.

[37]  F. Maxwell Harper,et al.  Facts or friends?: distinguishing informational and conversational questions in social Q&A sites , 2009, CHI.

[38]  Mira Mezini,et al.  Querying source code with natural language , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[39]  Emily Hill,et al.  Improving source code search with natural language phrasal representations of method signatures , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[40]  Pável Calado,et al.  Exploiting user feedback to learn to rank answers in q&a forums: a case study with stack overflow , 2013, SIGIR.