Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle

Many practitioners and academics believe in a delayed issue effect (DIE); i.e. the longer an issue lingers in the system, the more effort it requires to resolve. This belief is often used to justify major investments in new development processes that promise to retire more issues sooner. This paper tests for the delayed issue effect in 171 software projects conducted around the world in the period from 2006–2014. To the best of our knowledge, this is the largest study yet published on this effect. We found no evidence for the delayed issue effect; i.e. the effort to resolve issues in a later phase was not consistently or substantially greater than when issues were resolved soon after their introduction. This paper documents the above study and explores reasons for this mismatch between this common rule of thumb and empirical data. In summary, DIE is not some constant across all projects. Rather, DIE might be an historical relic that occurs intermittently only in certain kinds of projects. This is a significant result since it predicts that new development processes that promise to faster retire more issues will not have a guaranteed return on investment (depending on the context where applied), and that a long-held truth in software engineering should not be considered a global truism.

[1]  Premkumar T. Devanbu,et al.  A large scale study of programming languages and code quality in github , 2014, SIGSOFT FSE.

[2]  D. Madigan,et al.  A Systematic Statistical Approach to Evaluating Evidence from Observational Studies , 2014 .

[3]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[4]  Donald J. Reifer.  Profiles of Level 5 CMMI Organizations , 2007 .

[5]  Nancy R. Mead,et al.  Software Security Engineering: A Guide for Project Managers , 2004 .

[6]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[7]  Tim Menzies,et al.  Local vs. global models for effort estimation and defect prediction , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[8]  Tim Menzies,et al.  Distributed development considered harmful? , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[9]  Robert C. Martin,et al.  Iterative and Incremental development , 1999 .

[10]  Katrina D. Maxwell,et al.  Applied Statistics for Software Managers , 2002 .

[11]  Lefteris Angelis,et al.  Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm , 2013, IEEE Transactions on Software Engineering.

[12]  Robert L. Glass.  Software Engineering: Facts and Fallacies , 2002 .

[13]  Victor R. Basili,et al.  Iterative and incremental developments. a brief history , 2003, Computer.

[14]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[15]  Tim Menzies,et al.  Learning better IV&V practices , 2008, Innovations in Systems and Software Engineering.

[16]  R. Morrison,et al.  Study design, precision, and validity in observational studies. , 2009, Journal of palliative medicine.

[17]  Barry W. Boehm,et al.  Analyzing and handling local bias for calibrating parametric cost estimation models , 2013, Inf. Softw. Technol..

[18]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..

[19]  Tore Dybå,et al.  Building Theories in Software Engineering , 2008, Guide to Advanced Empirical Software Engineering.

[20]  Felix Bachmann,et al.  Integrate End to End Early and Often , 2013, IEEE Software.

[21]  Bruce McMillin,et al.  Software engineering: What is it? , 2018, 2018 IEEE Aerospace Conference.

[22]  James B. Dabney,et al.  Error Cost Escalation Through the Project Life Cycle , 2004 .

[23]  Qinbao Song,et al.  Data Quality: Some Comments on the NASA Software Defect Datasets , 2013, IEEE Transactions on Software Engineering.

[24]  Peter Kulchyski.  and , 2015 .

[25]  Barry W. Boehm,et al.  Understanding and Controlling Software Costs , 1988, IEEE Trans. Software Eng..

[26]  Walker Royce,et al.  Software Project Management: A Unified Framework , 1998 .

[27]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[28]  Michael E. Fagan.  Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[29]  Amr Elssamadisy,et al.  Recognizing and responding to "bad smells" in extreme programming , 2002, ICSE '02.

[30]  Edmund B. Daly.  Management of Software Development , 1977, IEEE Transactions on Software Engineering.

[31]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[32]  Stephen G. MacDonell,et al.  Evaluating prediction systems in software project estimation , 2012, Inf. Softw. Technol..

[33]  Watts S. Humphrey,et al.  Introduction to the Team Software Process , 1999 .

[34]  Norman E. Fenton,et al.  Software metrics: roadmap , 2000, ICSE '00.

[35]  Forrest Shull,et al.  Building Theories from Multiple Evidence Sources , 2008, Guide to Advanced Empirical Software Engineering.

[36]  Tore Dybå,et al.  A systematic review of effect size in software engineering experiments , 2007, Inf. Softw. Technol..

[37]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[38]  Michael Grottke,et al.  A systematic literature review of software quality cost research , 2011, J. Syst. Softw..

[39]  Orit Hazzan,et al.  The Agile Manifesto , 2014 .

[40]  Michael Fagan.  Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[41]  Capers Jones,et al.  The Economics of Software Quality , 2011 .

[42]  Nanette Brown,et al.  Agile Methods: Selected DoD Management and Acquisition Concerns , 2011 .

[43]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[44]  Ahmed E. Hassan,et al.  Think locally, act globally: Improving defect and effort prediction models , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[45]  W. E. Stephenson.  An analysis of the resources used in the SAFEGUARD system software development , 1976, ICSE '76.

[46]  T. Capers Jones,et al.  Estimating software costs , 1998 .

[47]  Daniela Cruzes,et al.  Analyzing the Impact of Beliefs in Software Project Practices , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[48]  Magne Jørgensen,et al.  The Impact of Lessons-Learned Sessions on Effort Estimation and Uncertainty Assessments , 2009, IEEE Transactions on Software Engineering.

[49]  Jason Rho,et al.  A decade of reversal: an analysis of 146 contradicted medical practices. , 2013, Mayo Clinic proceedings.

[50]  Steve McConnell,et al.  The Nine Deadly sins of Project Planning , 2001, IEEE Softw..

[51]  James B. Dabney,et al.  8.4.2 Error Cost Escalation Through the Project Life Cycle , 2004 .

[52]  Chris F. Kemerer,et al.  Ieee Transactions on Software Engineering, Manuscript Id 1 Does Software Process Improvement Reduce the Severity of Defects? a Longitudinal Field Study , 2022 .

[53]  Albert Endres,et al.  A handbook of software and systems engineering - empirical observations, laws and theories , 2003, The Fraunhofer IESE series on software engineering.

[54]  Ivar Jacobson,et al.  The Unified Software Development Process , 1999 .

[55]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[56]  Barry W. Boehm,et al.  Developing Small-Scale Application Software Products: Some Experiment Results , 1980, IFIP Congress.

[57]  Barry W. Boehm,et al.  Architecting: How Much and When? , 2011, Making Software.

[58]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[59]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[60]  J. Christopher Westland,et al.  The cost of errors in software development: evidence from industry , 2002, J. Syst. Softw..

[61]  Premkumar T. Devanbu,et al.  Belief & Evidence in Empirical Software Engineering , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[62]  Steve McConnell.  From the Editor - An Ounce of Prevention , 2001, IEEE Softw..

[63]  Tero Päivärinta,et al.  Theorizing about software development practices , 2015, Sci. Comput. Program..

[64]  Barry W. Boehm,et al.  Local bias and its impacts on the performance of parametric estimation models , 2011, Promise '11.

[65]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[66]  J. T. Lochner.  The Journal of Defense Software Engineering , 1999 .

[67]  Xin Yao,et al.  journal homepage: www.elsevier.com/locate/infsof Ensembles and locality: Insight on improving software effort estimation , 2022 .

[68]  Shane McIntosh,et al.  Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[69]  Walter Baziuk,et al.  BNR/NORTEL: path to improve product quality, reliability and customer satisfaction , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[70]  Ellis Horowitz,et al.  Software Cost Estimation with COCOMO II , 2000 .

[71]  Watts S. Humphrey,et al.  A discipline for software engineering , 2012, Series in software engineering.

[72]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[73]  Barry W. Boehm,et al.  What we have learned about fighting defects , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[74]  Watts S. Humphrey,et al.  TSP(SM) Leading a Development Team (SEI Series in Software Engineering) , 2005 .

[75]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[76]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[77]  Brian Fitzgerald,et al.  Theory-oriented software engineering , 2015, Sci. Comput. Program..

[78]  Shari Lawrence Pfleeger,et al.  Software metrics (2nd ed.): a rigorous and practical approach , 1997 .

[79]  Premkumar T. Devanbu,et al.  Ecological inference in empirical software engineering , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[80]  William R. Nichols,et al.  Initial evaluation of data quality in a TSP software engineering project data repository , 2014, ICSSP 2014.

[81]  R. R. Willis,et al.  Hughes Aircraft's Widespread Deployment of a Continuously Improving Software Process , 1998 .

[82]  Barry W. Boehm,et al.  Software Defect Reduction Top 10 List , 2001, Computer.

[83]  Evidence Supports Some Truisms, Belies Others , 1988, IEEE Softw..

[84]  Ahmed E. Hassan,et al.  Towards improving statistical modeling of software engineering data: think locally, act globally! , 2015, Empirical Software Engineering.

[85]  Gregory Tassey,et al.  Prepared for what , 2007 .

[86]  MenziesTim,et al.  Local versus Global Lessons for Defect Prediction and Effort Estimation , 2013 .

[87]  Robert L. Glass,et al.  Facts and fallacies of software engineering , 2002 .

[88]  BoehmBarry,et al.  Analyzing and handling local bias for calibrating parametric cost estimation models , 2013 .

[89]  Steve McConnell.  Software quality at top speed , 1996 .