Automatic Classifying Self-Admitted Technical Debt Using N-Gram IDF

Technical Debt (TD) introduces a quality problem and increases maintenance cost since it may require improvements in the future. Several studies show that it is possible to automatically detect TD from source code comments that developers intentionally created, so-called self-admitted technical debt (SATD). Those studies proposed to use binary classification technique to predict whether a comment shows SATD. However, SATD has different types (e.g. design SATD and requirement SATD). In this paper, we therefore propose an approach using N-gram Inverse Document Frequency (IDF) and employ a multi-class classification technique to build a model that can identify different types of SATD. From the empirical evaluation on 10 open-source projects, our approach outperforms alternative methods (e.g. using BOW and TF-IDF). Our approach also improves the prediction performance over the baseline benchmark by 33%.

[1]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[2]  Jǐŕı Kléma.  Automatic Categorization of Fanatic Texts , 2007 .

[3]  Alexander R. Statnikov,et al.  Text classification for automatic detection of alcohol use-related tweets: A feasibility study , 2014, Proceedings of the 2014 IEEE 15th International Conference on Information Reuse and Integration (IEEE IRI 2014).

[4]  Tony R. Martinez,et al.  An instance level analysis of data complexity , 2014, Machine Learning.

[5]  Jorge I. Galván-Tejada,et al.  An Analysis of Audio Features to Develop a Human Activity Recognition Model Using Genetic Algorithms, Random Forests, and Neural Networks , 2016, Mob. Inf. Syst..

[6]  Yiannis Kompatsiaris,et al.  News Articles Classification Using Random Forests and Weighted Multimodal Features , 2014, IRFC.

[7]  Emad Shihab,et al.  Examining the Impact of Self-Admitted Technical Debt on Software Quality , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[8]  Hideaki Hata,et al.  Identifying Design and Requirement Self-Admitted Technical Debt Using N-gram IDF , 2018, 2018 9th International Workshop on Empirical Software Engineering in Practice (IWESEP).

[9]  Vili Podgorelec,et al.  Enhanced Feature Selection Using Word Embeddings for Self-Admitted Technical Debt Identification , 2018, 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[10]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[11]  David Lo,et al.  Identifying self-admitted technical debt in open source projects using text mining , 2017, Empirical Software Engineering.

[12]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[13]  K. R. Chowdhary,et al.  Comparison of SVM and Naive Bayes Text Classification Algorithms using WEKA , 2017 .

[14]  Emad Shihab,et al.  Detecting and quantifying different types of self-admitted technical Debt , 2015, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD).

[15]  Mário André de Freitas Farias,et al.  A Contextualized Vocabulary Model for identifying technical debt on code comments , 2015, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD).

[16]  Michael A. Shepherd,et al.  Support vector machines for text categorization , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[17]  Nikolaos Tsantalis,et al.  Using Natural Language Processing to Automatically Detect Self-Admitted Technical Debt , 2017, IEEE Transactions on Software Engineering.

[18]  Taghi M. Khoshgoftaar,et al.  An Empirical Study of Learning from Imbalanced Data Using Random Forest , 2007 .

[19]  Andrew Zisserman,et al.  Image Classification using Random Forests and Ferns , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[20]  Hideaki Hata,et al.  Bug or Not? Bug Report Classification Using N-Gram IDF , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[21]  Aditya K. Ghose,et al.  Characterization and Prediction of Issue-Related Risks in Software Projects , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[22]  José Augusto Baranauskas,et al.  How Many Trees in a Random Forest? , 2012, MLDM.

[23]  David Lo,et al.  Automating Change-Level Self-Admitted Technical Debt Determination , 2019, IEEE Transactions on Software Engineering.

[24]  Emad Shihab,et al.  An Exploratory Study on Self-Admitted Technical Debt , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[25]  Jan Bosch,et al.  The Introduction of Technical Debt Tracking in Large Companies , 2016, 2016 23rd Asia-Pacific Software Engineering Conference (APSEC).

[26]  Shojiro Nishio,et al.  N-gram IDF: A Global Term Weighting Scheme Based on Information Distance , 2015, WWW.

[27]  Gabriele Bavota,et al.  A Large-Scale Empirical Study on Self-Admitted Technical Debt , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[28]  Yanjun Qi,et al.  Sentiment classification based on supervised latent n-gram analysis , 2011, CIKM '11.

[29]  Ward Cunningham,et al.  The WyCash portfolio management system , 1992, OOPSLA '92.

[30]  Hideaki Hata,et al.  Sentiment Classification Using N-Gram Inverse Document Frequency and Automated Machine Learning , 2019, IEEE Software.

[31]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[32]  Arash Joorabchi,et al.  A new text representation scheme combining Bag-of-Words and Bag-of-Concepts approaches for automatic text classification , 2013, 2013 7th IEEE GCC Conference and Exhibition (GCC).

[33]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[34]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[35]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[36]  S. H. Gawande,et al.  A Comparative Study on Different Types of Approaches to Text Categorization , 2012 .

[37]  Adhistya Erna Permanasari,et al.  Study of Undersampling Method: Instance Hardness Threshold with Various Estimators for Hate Speech Classification , 2018, IJITEE (International Journal of Information Technology and Electrical Engineering).

[38]  Zhenchang Xing,et al.  Neural Network-based Detection of Self-Admitted Technical Debt: From Performance to Explainability , 2019, ACM Trans. Softw. Eng. Methodol..