The impact of software process consistency on residual defects

Residual defects at the time of delivery are an important concern for safety critical software systems. Suppliers and customers are urged to get evidence for what they can do to reduce residual defects. Thus, it is meaningful to learn from historical data concerning the kinds of defects that have escaped from the existing quality assurance approaches and the factors that lead to the residual defects. A total of 3747 defects from 70 software systems developed by 29 Chinese aviation organizations were collected from acceptance tests during the last 5 years. For all these organizations, 38 domain experts from the industry assessed the process consistency to the standard built in the framework of Capability Maturity Model (CMM). Results demonstrate that the process improvement in the range of high consistency is effective in reducing total defects, as well as the minor and severe defects. The high consistency adoption of the practices in CMM Level 1 to Level 3 is more effective in reducing minor defects than severe defects. Causal analysis was performed to investigate the underlying mechanisms. Results reveal that individual cognitive failures cause 87% of severe defects. More approaches to help software developers manage their interior cognitive process are needed for improving software quality in the future. Copyright © 2015 John Wiley & Sons, Ltd.

[1]  Manish Agrawal,et al.  Software Effort, Quality, and Cycle Time: A Study of CMM Level 5 Projects , 2007, IEEE Transactions on Software Engineering.

[2]  Noopur Davis,et al.  The Team Software ProcessSM (TSPSM) in Practice: A Summary of Recent Results , 2003 .

[3]  A. Fuqun Huang,et al.  A Taxonomy System to Identify Human Error Causes for Software Defects , 2012 .

[4]  Tony Gorschek,et al.  Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review , 2012, IEEE Transactions on Software Engineering.

[5]  David P. Baker,et al.  Development and Validation of Aviation Causal Contributors for Error Reporting Systems (ACCERS) , 2007, Hum. Factors.

[6]  Dennis R. Goldenson,et al.  A systematic survey of CMM experience and results , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[7]  Mark C. Paulk,et al.  Capability Maturity Model for Software , 2001 .

[8]  Chris F. Kemerer,et al.  Ieee Transactions on Software Engineering, Manuscript Id 1 Does Software Process Improvement Reduce the Severity of Defects? a Longitudinal Field Study , 2022 .

[9]  Pierre N. Robillard,et al.  The impacts of software process improvement on developers: A systematic review , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[10]  Guilherme Horta Travassos,et al.  Towards a Defect Prevention Based Process Improvement Approach , 2008, 2008 34th Euromicro Conference Software Engineering and Advanced Applications.

[11]  K. Gwet.  Computing inter-rater reliability and its variance in the presence of high agreement. , 2008, The British journal of mathematical and statistical psychology.

[12]  Françoise Détienne,et al.  Software Design — Cognitive Aspects , 2001, Practitioner Series.

[13]  Fuqun Huang,et al.  Study on the Correlations Between Program Metrics and Defect Rate by a Controlled Experiment , 2013 .

[14]  AgrawalManish,et al.  Software Effort, Quality, and Cycle Time , 2007 .

[15]  Robert G. Mays,et al.  Experiences with Defect Prevention , 1990, IBM Syst. J..

[16]  Ayse Basar Bener,et al.  Implementation of a Software Quality Improvement Project in an SME: A Before and After Comparison , 2009, 2009 35th Euromicro Conference on Software Engineering and Advanced Applications.

[17]  Dewayne E. Perry,et al.  A case study in root cause defect analysis , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[18]  Jason Bennett Thatcher,et al.  IS Employee Attitudes and Perceptions at Varying Levels of Software Process Maturity , 2012, MIS Q..

[19]  Frank E. McGarry,et al.  Measuring the impacts individual process maturity attributes have on software products , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[20]  Mark C. Paulk,et al.  Capability Maturity Model for Software, Version 1.1 , 1993 .

[21]  Wang Yichen.  Review of Software Psychology , 2013 .

[22]  J. Shaoul.  Human Error , 1973, Nature.

[23]  S. B. Kiselev,et al.  The capability maturity model: guidelines for improving the software process , 1995 .

[24]  David N. Card,et al.  Myths and Strategies of Defect Causal Analysis , 2006 .

[25]  Fuqun Huang,et al.  Systematically Improving Software Reliability: Considering Human Errors of Software Practitioners , 2011 .

[26]  Rajesh Krishna Balan,et al.  The impact of process choice in high maturity environments: An empirical analysis , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[27]  Mayuram S. Krishnan,et al.  Measuring Process Consistency: Implications for Reducing Software Defects , 1999, IEEE Trans. Software Eng..

[28]  Michael Diaz,et al.  How Software Process Improvement Helped Motorola , 1997, IEEE Softw..

[29]  Mayuram S. Krishnan,et al.  Effects of Process Maturity on Quality, Cycle Time, and Effort in Software Product Development , 2000 .

[30]  Gerald M. Weinberg,et al.  Psychology of computer programming , 1971 .

[31]  B. Boehm,et al.  Modeling Software Defect Introduction and Removal : COQUALMO ( COnstructive QUALity MOdel ) , 1999 .

[32]  Bin Liu,et al.  The links between human error diversity and software diversity: Implications for fault diversity seeking , 2014, Sci. Comput. Program..

[33]  Norman E. Fenton,et al.  A Critique of Software Defect Prediction Models , 1999, IEEE Trans. Software Eng..

[34]  Tomás San Feliu Gilabert,et al.  High‐maturity levels: achieving CMMI ML‐5 in a consultancy company , 2014, J. Softw. Evol. Process..

[35]  Michael Diaz,et al.  How CMM Impacts Quality, Productivity, Rework, and the Bottom Line , 2002 .

[36]  Ana Regina Cavalcanti da Rocha,et al.  ROI of software process improvement at BL informática: SPIdex is really worth it , 2008, Softw. Process. Improv. Pract..

[37]  Luk N. Van Wassenhove,et al.  Total quality in software development: An empirical study of quality drivers and benefits in Indian software projects , 2010, Inf. Manag..

[38]  YOSHIHIRO TOHMA,et al.  Structural Approach to the Estimation of the Number of Residual Software Faults Based on the Hyper-Geometric Distribution , 1989, IEEE Trans. Software Eng..

[39]  Mary Ann Schroeder,et al.  Diagnosing and Dealing with Multicollinearity , 1990, Western journal of nursing research.

[40]  Daniel Galin,et al.  Are CMM Program Investments Beneficial? Analyzing Past Studies , 2006, IEEE Software.

[41]  Herb Krasner,et al.  Accumulating the Body of Evidence for The Payoff of Software Process Improvement , 1997 .

[42]  Ram Chillarege,et al.  Orthogonal defect classification , 1996 .

[43]  William Marsh,et al.  Predicting software defects in varying development lifecycles using Bayesian nets , 2007, Inf. Softw. Technol..

[44]  Barry Boehm,et al.  The effects of software process maturity on software development effort , 1997 .

[45]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[46]  Capers Jones,et al.  Measuring Defect Potentials and Defect Removal Efficiency , 2008 .

[47]  Gleison Santos,et al.  ROI of software process improvement at BL informática: SPIdex is really worth it , 2008 .

[48]  Gerard J. Holzmann,et al.  The logic of bugs , 2002, SIGSOFT '02/FSE-10.

[49]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..