Measuring where it matters: Determining starting points for metrics collection

Defining useful metrics to measure the goals of a software organisation is difficult. Defining useful metrics to measure the causes of the (failure) to fulfil those organisational goals is even more difficult, as the diversity of potential causes makes their measurement illusive. In this article, we describe a method to select useful software metrics based on findings from qualitative research. In a case study, we apply this method to a previously conducted study of project post-mortem reviews to assess the validity of our prior claims. For this we collected data on 109 new software projects in the organisation in which we conducted the previous case study.

[1]  H. D. Rombach,et al.  THE EXPERIENCE FACTORY , 1999 .

[2]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[3]  Michael Daskalantonakis,et al.  A Practical View of Software Measurement and Implementation Experiences Within Motorola , 1992, IEEE Trans. Software Eng..

[4]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[5]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[6]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[7]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[8]  Sjaak Brinkkemper,et al.  A method to draw lessons from project postmortem databases , 2006, Softw. Process. Improv. Pract..

[9]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[10]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[11]  Alain Abran,et al.  Guide to the Software Engineering Body of Knowledge : 2004 Version , 2005 .

[12]  A Straw,et al.  Guide to the Software Engineering Body of Knowledge , 1998 .

[13]  Sjaak Brinkkemper,et al.  Discovering the Relation Between Project Factors and Project Success in Post-mortem Evaluations , 2004, EuroSPI.

[14]  Janice Singer,et al.  Towards an ontology of software maintenance , 1999, J. Softw. Maintenance Res. Pract..

[15]  Tom DeMarco,et al.  Controlling Software Projects: Management, Measurement, and Estimates , 1986 .

[16]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[17]  Walter F. Tichy,et al.  Proceedings 25th International Conference on Software Engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[18]  Norman E. Fenton,et al.  Implementing Effective Software Metrics Programs , 1997, IEEE Softw..

[19]  Egon Berghout,et al.  The Goal/Question/Metric method: a practical guide for quality improvement of software development , 1999 .

[20]  Ray Welland,et al.  Collecting Feedback during Software Engineering Experiments , 2005, Empirical Software Engineering.

[21]  C. Brodsky The Discovery of Grounded Theory: Strategies for Qualitative Research , 1968 .

[22]  Hans van Vliet,et al.  Measurements Should Generate Value, Rather than Data , 1999 .

[23]  T.C. Lethbridge,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK) and the Software Engineering Education Knowledge (SEEK) - a preliminary mapping , 2001, 10th International Workshop on Software Technology and Engineering Practice.

[24]  Esperanza Marcos,et al.  An Approach to the Integration of Qualitative and Quantitative Research Methods in Software Engineering Research , 2006, PhiSE.

[25]  Mark C. Paulk,et al.  The Capability Maturity Model: Guidelines for Improving the Software Process , 1994 .

[26]  L. Delbeke Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[27]  R. F.,et al.  Statistical Method from the Viewpoint of Quality Control , 1940, Nature.

[28]  Lionel C. Briand,et al.  Practical guidelines for measurement-based process improvement , 1996, Softw. Process. Improv. Pract..

[29]  Mayuram S. Krishnan,et al.  Measurement Programs in Software Development: Determinants of Success , 2002, IEEE Trans. Software Eng..

[30]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[31]  Claes Wohlin,et al.  Assessing Project Success Using Subjective Evaluation Factors , 2004, Software Quality Journal.