Factors Explaining External Quality in 54 Case Studies of Software Development Projects

Background: Confounding factors can easily make research hard to interpret and generalise. But there is currently no standard list of factors that should always be measured when conducting empirical investigations. Objective: To measure the explanatory power of eight simple metrics (two different pretests, number of members, total working time reported, development method used, test method used, formal specification method used, and programming language used) to explain external software project quality as measured by the project client. Method: We collected data on 54 software development teams over a five year period. A univariate analysis was used to calculate the explanatory power of the metrics and check for interaction effects between the categorical data. Results: Two of the proposed metrics (a pre-test based on a development project and the total time spent per team) led to significant explanation of the quality measurement. It was also noted that the differences between the Java and PHP programming languages did not explain the variation in quality, but some limited data available for JSP indicated this may not be the case for all languages. Conclusion: We recommend that any empirical investigations into external quality at least records the total time spent in man hours and an assessment of the competence of the developers. In addition future work is needed to determine if other programming languages explain variance in external quality.

[1]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[2]  Carolyn Craggs,et al.  Statistics in Research: Basic Concepts and Techniques for Research Workers. , 1989 .

[3]  Bernard Ostle,et al.  Statistics in Research: Basic Concepts and Techniques for Research Workers , 1990 .

[4]  Boyce Rensberger.  The Nature of Evidence , 2000, Science.

[5]  H. E. Dunsmore,et al.  Software engineering metrics and models , 1986 .

[6]  M. Holcombe,et al.  Using a formal method to model software design in XP projects , 2005 .

[7]  Allen S. Lee.  A Scientific Methodology for MIS Case Studies , 1989, MIS Q..

[8]  Kent Beck,et al.  Extreme Programming Explained: Embrace Change (2nd Edition) , 2004 .

[9]  Sharifah Lailee Syed-Abdullah,et al.  Empirical study on extreme programming , 2005 .

[10]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[11]  Mike Holcombe,et al.  Running an Agile Software Development Project , 2008 .

[12]  Walter F. Tichy,et al.  Hints for Reviewing Empirical Work in Software Engineering , 2000, Empirical Software Engineering.

[13]  Walter F. Tichy,et al.  Status of Empirical Research in Software Engineering , 2006, Empirical Software Engineering Issues.

[14]  Judith Segal.  The nature of evidence in empirical software engineering , 2003, Eleventh Annual International Workshop on Software Technology and Engineering Practice.

[15]  Francisco Javier Macias,et al.  Empirical assessment of extreme programming , 2004 .