Does bug prediction support human developers? Findings from a Google case study

While many bug prediction algorithms have been developed by academia, they're often only tested and verified in the lab using automated means. We do not have a strong idea about whether such algorithms are useful to guide human developers. We deployed a bug prediction algorithm across Google, and found no identifiable change in developer behavior. Using our experience, we provide several characteristics that bug prediction algorithms need to meet in order to be accepted by human developers and truly change how developers evaluate their code.

[1]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[2]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[3]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[4]  Victor R. Basili,et al.  A Validation of Object-Oriented Design Metrics as Quality Indicators , 1996, IEEE Trans. Software Eng..

[5]  David Hovemeyer,et al.  Finding bugs is easy , 2004, SIGP.

[6]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[7]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[8]  Richard C. Holt,et al.  The top ten list: dynamic fault prediction , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[9]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[10]  Lucas Layman,et al.  Toward Reducing Fault Fix Time: Understanding Developer Behavior for the Design of Automated Fault Detection Tools , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[11]  S. V. Subrahmanya,et al.  A Survey of Enterprise Software Development Risks in a Flat World , 2007, ESEM 2007.

[12]  J. David Morgenthaler,et al.  Evaluating static analysis defect warnings on production software , 2007, PASTE '07.

[13]  Andreas Zeller,et al.  Predicting faults from cached history , 2008, ISEC '08.

[14]  William Pugh,et al.  A report on a survey and study of static analysis users , 2008, DEFECTS '08.

[15]  David Hovemeyer,et al.  Using Static Analysis to Find Bugs , 2008, IEEE Software.

[16]  Robert Feldt,et al.  Dynamic Regression Test Selection Based on a File Cache An Industrial Evaluation , 2009, 2009 International Conference on Software Testing Verification and Validation.

[17]  Premkumar T. Devanbu,et al.  Fair and balanced?: bias in bug-fix datasets , 2009, ESEC/FSE '09.

[18]  Per Runeson,et al.  An Empirical Evaluation of Regression Testing Based on Fix-Cache Recommendations , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[19]  André van der Hoek,et al.  The code orb: supporting contextualized coding via at-a-glance views (NIER track) , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[20]  Premkumar T. Devanbu,et al.  BugCache for inspections: hit or miss? , 2011, ESEC/FSE '11.