Which log level should developers choose for a new logging statement?

This is an extended abstract of a paper published in the Empirical Software Engineering journal. The original paper is communicated by Mark Grechanik. The paper empirically studied how developers assign log levels to their logging statements and proposed an automated approach to help developers determine the most appropriate log level when they add a new logging statement. We analyzed the development history of four open source projects (Hadoop, Directory Server, Hama, and Qpid). We found that our automated approach can accurately suggest the levels of logging statements with an AUC of 0.75 to 0.81. We also found that the characteristics of the containing block of a newly-added logging statement, the existing logging statements in the containing source code file, and the content of the newly-added logging statement play important roles in determining the appropriate log level for that logging statement.

[1]  P. McCullagh.  Regression Models for Ordinal Data , 1980 .

[2]  Heng Li,et al.  Which log level should developers choose for a new logging statement? , 2017, Empirical Software Engineering.

[3]  Michele Lanza,et al.  Evaluating defect prediction approaches: a benchmark and an extensive comparison , 2011, Empirical Software Engineering.

[4]  Richard M. Huggins,et al.  Variables Selection using the Wald Test and a Robust Cp , 1996 .

[5]  P Barton,et al.  Systematic review and individual patient data meta-analysis of diagnosis of heart failure, with modelling of implications of different diagnostic strategies in primary care. , 2009, Health technology assessment.

[6]  Cor-Paul Bezemer,et al.  Examining the Stability of Logging Statements , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[7]  R. McKelvey,et al.  A statistical model for the analysis of ordinal level dependent variables , 1975 .

[8]  Andreas Zeller,et al.  Mining Version Histories to Guide Software Changes , 2004 .

[9]  Max Kuhn,et al.  Applied Predictive Modeling , 2013 .

[10]  Michael I. Jordan,et al.  Detecting large-scale system problems by mining console logs , 2009, SOSP '09.

[11]  J. Lawless,et al.  Efficient Screening of Nonnormal Regression Models , 1978 .

[12]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[13]  Leonardo Mariani,et al.  A toolset for automated failure analysis , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[14]  Shane McIntosh,et al.  An empirical study of the impact of modern code review practices on software quality , 2015, Empirical Software Engineering.

[15]  Wei Xu,et al.  Advances and challenges in log analysis , 2011, Commun. ACM.

[16]  Ding Yuan,et al.  Characterizing logging practices in open-source software , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[17]  Qiang Fu,et al.  Learning to Log: Helping Developers Make Informed Logging Decisions , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[18]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[19]  Yu Luo,et al.  Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems , 2014, OSDI.

[20]  Cor-Paul Bezemer,et al.  Logging Library Migrations: A Case Study for the Apache Software Foundation Projects , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[21]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[22]  Michael W. Godfrey,et al.  An Exploratory Study of the Evolution of Communicated Information about the Execution of Large Software Systems , 2011, 2011 18th Working Conference on Reverse Engineering.

[23]  Freda Kemp.  Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences , 2003 .

[24]  Leonardo Mariani,et al.  Automated Identification of Failure Causes in System Logs , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[25]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[26]  Shane McIntosh,et al.  The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects , 2014, MSR 2014.

[27]  Domenico Cotroneo,et al.  Industry Practices and Event Logging: Assessment of a Critical Software Development Process , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[28]  Kv Petrides,et al.  Regression analysis for categorical moderators , 2006 .

[29]  Ahmed E. Hassan,et al.  Understanding the impact of code and process metrics on post-release defects: a case study on the Eclipse project , 2010, ESEM '10.

[30]  Ding Yuan,et al.  Improving Software Diagnosability via Log Enhancement , 2012, TOCS.

[31]  N. Mantel.  Why Stepdown Procedures in Variable Selection , 1970 .

[32]  Ding Yuan,et al.  SherLog: error diagnosis by connecting clues from run-time logs , 2010, ASPLOS XV.

[33]  Ahmed E. Hassan,et al.  Predicting faults using the complexity of code changes , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[34]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[35]  Yang Liu,et al.  Be conservative: enhancing failure diagnosis with proactive logging , 2012, OSDI 2012.

[36]  James E. Helmreich.  Regression Modeling Strategies with Applications to Linear Models, Logistic and Ordinal Regression and Survival Analysis (2nd Edition) , 2016 .

[37]  B. Efron.  How Biased is the Apparent Error Rate of a Prediction Rule , 1986 .

[38]  Qiang Fu,et al.  Where do developers log? an empirical study on logging practices in industry , 2014, ICSE Companion.