A framework for developing measurement systems and its industrial evaluation

As in every engineering discipline, metrics play an important role in software development, with the difference that almost all software projects need the customization of metrics used. In other engineering disciplines, the notion of a measurement system (i.e. a tool used to collect, calculate, and report quantitative data) is well known and defined, whereas it is not as widely used in software engineering. In this paper we present a framework for developing custom measurement systems and its industrial evaluation in a software development unit within Ericsson. The results include the framework for designing measurement systems and its evaluation in real life projects at the company. The results show that with the help of ISO/IEC standards, measurement systems can be effectively used in software industry and that the presented framework improves the way of working with metrics. This paper contributes with the presentation of how automation of metrics collection and processing can be successfully introduced into a large organization and shows the benefits of it: increased efficiency of metrics collection, increased adoption of metrics in the organization, independence from individuals and standardized nomenclature for metrics in the organization.

[1]  N. P. Kolev,et al.  Development of measurement systems with robustness feedback , 1996, Quality Measurement: The Indispensable Bridge between Theory and Reality (No Measurements? No Science! Joint Conference - 1996: IEEE Instrumentation and Measurement Technology Conference and IMEKO Tec.

[2]  N. B. Jones,et al.  Knowledge-based systems for neuroelectric signal processing , 1994 .

[3]  John H. Mahaffy,et al.  On the automated assessment of nuclear reactor systems code accuracy , 2002 .

[4]  A. N. Zaborovsky,et al.  Software and hardware for measurements systems , 2002, IEEE-Siberian Conference on Electron Devices and Materials. SIBEDEM - 2002. Proceedings (Cat. No.02EX529).

[5]  Mario Piattini,et al.  Managing software process measurement: A metamodel-based approach , 2007, Inf. Sci..

[6]  Horst Zuse,et al.  A Framework of Software Measurement , 1998 .

[7]  D. Ross Jeffery,et al.  A framework for evaluation and prediction of metrics program success , 1993, [1993] Proceedings First International Software Metrics Symposium.

[8]  Hans van Vliet,et al.  Measuring where it matters: Determining starting points for metrics collection , 2008, J. Syst. Softw..

[9]  Paul Goodman.  Practical Implementation of Software Metrics , 1993 .

[10]  Henry H. Emurian,et al.  Acceptance issues in metrics program implementation , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[11]  Tapani Kilpi,et al.  Implementing a Software Metrics Program at Nokia , 2001, IEEE Softw..

[12]  Guo Kai.  Virtual Measurement System for Muzzle Velocity and Firing Frequency , 2007, 2007 8th International Conference on Electronic Measurement and Instruments.

[13]  Haiqing Li,et al.  An intelligent measurement system for powder flowrate measurement in pneumatic conveying system , 2002, IEEE Trans. Instrum. Meas..

[14]  Barbara A. Kitchenham,et al.  Experiences introducing a measurement program , 1997, Inf. Softw. Technol..

[15]  Hans van Vliet,et al.  Measurements should generate value, rather than data [software metrics] , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[16]  Mayuram S. Krishnan,et al.  The impact of institutional forces on software metrics programs , 2005, IEEE Transactions on Software Engineering.

[17]  Miroslaw Staron,et al.  Predicting weekly defect inflow in large software projects based on project planning and test status , 2008, Inf. Softw. Technol..

[18]  Kenneth L. Atkins,et al.  STARDUST: implementing a new manage-to-budget paradigm , 2003 .

[19]  Hans van Vliet,et al.  Measurement program success factors revisited , 2001, Inf. Softw. Technol..

[20]  K. Pahlavan,et al.  Measurement of characteristics of voice over IP in a wireless LAN environment , 1999, 1999 IEEE International Workshop on Mobile Multimedia Communications (MoMuC'99) (Cat. No.99EX384).

[21]  Hans van Vliet,et al.  Measurements Should Generate Value, Rather than Data , 1999 .

[22]  Philippe Kruchten,et al.  The Rational Unified Process: An Introduction , 1998 .

[23]  B. Kitchenham,et al.  Measurement Modeling Technology , 2003, IEEE Softw..

[24]  Norman E. Fenton,et al.  Implementing Effective Software Metrics Programs , 1997, IEEE Softw..

[25]  M. Foote,et al.  Video measurement of swash zone hydrodynamics , 1999 .

[26]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[27]  Nikolay Petkov Kolev,et al.  Computerized investigation of robust measurement systems , 2002, IEEE Trans. Instrum. Meas..

[28]  Karl-Heinrich Moller.  Software Metrics: A Practitioner's Guide to Improved Product Development , 1992 .

[29]  Philippe Kruchten,et al.  What Is the Rational Unified Process ? , 2001 .

[30]  William M. K. Trochim,et al.  Research methods knowledge base , 2001 .

[31]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[32]  Mayuram S. Krishnan,et al.  Measurement Programs in Software Development: Determinants of Success , 2002, IEEE Trans. Software Eng..

[33]  Francisca Losavio,et al.  Characterizing a data model for software measurement , 2005, J. Syst. Softw..

[34]  Cei.  International vocabulary of basic and general terms in metrology = vocabulaire international des termes fondamentaux et généraux de métrologie , 1993 .

[35]  Norman E. Fenton,et al.  Software metrics: successes, failures and new directions , 1999, J. Syst. Softw..

[36]  Shari Lawrence Pfleeger,et al.  Status Report on Software Measurement , 1997, IEEE Softw..

[37]  Robert L. Glass,et al.  Science and substance: a challenge to software engineers , 1994, IEEE Software.

[38]  Horst Zuse,et al.  Foundations of object-oriented software measures , 1996, Proceedings of the 3rd International Software Metrics Symposium.

[39]  D. Wisell,et al.  Considerations when Designing and Using Virtual Instruments as Building Blocks in Flexible Measurement System Solutions , 2007, 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007.

[40]  Timothy G. Trucano,et al.  Verification and Validation in Computational Fluid Dynamics , 2002 .