Systematic Literature Review on Software Measurement Programs

ContextSoftware measurement programs (MPs) are an important means for understanding, evaluating, managing, and improving software processes, products and resources. However, implementing successful MPs still remains a challenge. ObjectivesTo make a comprehensive review of the studies on MPs for bringing into light the existing measurement planning models and tools used for implementing MPs,the accumulated knowledge on the success/failure factors of MPs and mitigation strategies to address their challenges. MethodsA Systematic Literature Review (SLR) was conducted. In total, 65primary studies were reviewed and analyzed. ResultsWe identified 35 measurement planning models and 11 associated tools, most of which either proposed extensions or improvements for goal based approaches. The identified success factors include (a) organizational adoption of MP, (b) integration of MP with SDLC, (c) synchronization of MP with SPI and (d) design of MP. The mostly mentioned mitigation strategies for addressing challenges are effective change management and measurement stakeholder management, automated tool support and incorporation of engineering mechanisms for designing sustainable, effective, scalable and extendible MPs, and measurement expertise and standards development. ConclusionMost of the success factors and mitigation strategies have interdependencies. Therefore, for successful MP implementation, software organizations should consider these factors in combination and make a feasibility study at the very beginning.

[1]  Miroslaw Staron,et al.  Ensuring Reliability of Information Provided by Measurement Systems , 2009, IWSM/Mensura.

[2]  Kai Petersen,et al.  Worldviews, Research Methods, and their Relationship to Validity in Empirical Software Engineering Research , 2013, 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement.

[3]  Miroslaw Staron,et al.  Critical role of measures in decision processes: Managerial and technical measures in the context of large software development organizations , 2012, Inf. Softw. Technol..

[4]  Hans van Vliet,et al.  Towards mature measurement programs , 1998, Proceedings of the Second Euromicro Conference on Software Maintenance and Reengineering.

[5]  Lars Mathiassen,et al.  Cultivation and engineering of a software metrics program , 2003, Inf. Syst. J..

[6]  Egon Berghout,et al.  Integrating goal-oriented measurement in industrial software engineering: industrial experiences with and additions to the Goal/Question/Metric method (GQM) , 2001, Proceedings Seventh International Software Metrics Symposium.

[7]  Tereza G. Kirner,et al.  A GQM-based tool to support the development of software quality measurement plans , 1999, SOEN.

[8]  Hans van Vliet,et al.  Measurements should generate value, rather than data [software metrics] , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[9]  Markku Oivo,et al.  SAS: A Tool for the GQM+Strategies Grid Derivation Process , 2010, PROFES.

[10]  Jari Soini,et al.  A Survey of Metrics Use in Finnish Software Companies , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[11]  C. Symons,et al.  The COSMIC Functional Size Measurement Method : Measurement Manual : The COSMIC Implementation Guide for ISO/IEC 19761:2011 , 2015 .

[12]  Per Runeson,et al.  V-GQM: a feed-back approach to validation of a GQM study , 2001, Proceedings Seventh International Software Metrics Symposium.

[13]  Francisca Losavio,et al.  Characterizing a data model for software measurement , 2005, J. Syst. Softw..

[14]  Hans van Vliet,et al.  Measuring where it matters: Determining starting points for metrics collection , 2008, J. Syst. Softw..

[15]  Mayuram S. Krishnan,et al.  The impact of institutional forces on software metrics programs , 2005, IEEE Transactions on Software Engineering.

[16]  Alain Abran,et al.  Software measurement body of knowledge - overview of empirical support , 2005 .

[17]  R. Kaplan,et al.  The balanced scorecard--measures that drive performance. , 2015, Harvard business review.

[18]  Paolo Donzelli,et al.  Towards Reusable Measurement Patterns , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[19]  A. Gray,et al.  Worldviews , 2011, International psychiatry : bulletin of the Board of International Affairs of the Royal College of Psychiatrists.

[20]  William A. Florac,et al.  Goal-Driven Software Measurement. A Guidebook. , 1996 .

[21]  Tony Gorschek,et al.  Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review , 2012, IEEE Transactions on Software Engineering.

[22]  Mario Piattini,et al.  MIS-PyME Software Measurement Maturity Model-Supporting the Definition of Software Measurement Programs , 2008, PROFES.

[23]  Maria Teresa Baldassarre,et al.  Comprehensibility and efficiency of multiview framework for measurement plan design , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[24]  Alain Abran,et al.  ICEBERG : a different look at software project management , 2002 .

[25]  Tore Dybå,et al.  An empirical investigation of the key factors for success in software process improvement , 2005, IEEE Transactions on Software Engineering.

[26]  Barbara Kitchenham,et al.  What's up with software metrics? - A preliminary mapping study , 2010, J. Syst. Softw..

[27]  Félix García,et al.  MIS-PyME software measurement capability maturity model - Supporting the definition of software measurement programs and capability determination , 2010, Adv. Eng. Softw..

[28]  Rini van Solingen,et al.  Business impact, benefit, and cost of applying GQM in industry: an in-depth, long-term investigation at Schlumberger RPS , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[29]  Alain Abran,et al.  A framework for the design and verification of software measurement methods , 2008, J. Syst. Softw..

[30]  GencelCigdem,et al.  A decision support framework for metrics selection in goal-based measurement programs , 2013 .

[31]  Bernhard Daubner.  Empowering software development environments by automatic software measurement , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[32]  Victor R. Basili,et al.  An Approach to Improving Existing Measurement Frameworks , 1998, IBM Syst. J..

[33]  Tridas Mukhopadhyay,et al.  Determinants of success in software measurement programs: initial results , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[34]  Sandro Morasca,et al.  Applying GQM in an industrial software factory , 1998, TSEM.

[35]  Seija Komi-Sirviö,et al.  Measurement automation: methodological background and practical solutions a multiple case study , 2001, Proceedings Seventh International Software Metrics Symposium.

[36]  L. Lavazza,et al.  Automated support for process-aware definition and execution of measurement plans , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[37]  Iker Gondra,et al.  Applying machine learning to software fault-proneness prediction , 2008, J. Syst. Softw..

[38]  Patrik Berander,et al.  A goal question metric based approach for efficient measurement framework definition , 2006, ISESE '06.

[39]  Tom DeMarco,et al.  Controlling Software Projects: Management, Measurement, and Estimates , 1986 .

[40]  Henry H. Emurian,et al.  Acceptance issues in metrics program implementation , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[41]  Touseef Tahir,et al.  A Systematic Review on Software Measurement Programs , 2011, 2011 Frontiers of Information Technology.

[42]  C. A. Dekkers,et al.  The dangers of using software metrics to (mis)manage , 2002 .

[43]  Michael Berry,et al.  A targeted assessment of the software measurement process , 2001, Proceedings Seventh International Software Metrics Symposium.

[44]  Hans van Vliet,et al.  Measurement program success factors revisited , 2001, Inf. Softw. Technol..

[45]  Luigi Lavazza.  Providing Automated Support for the GQM Measurement Process , 2000, IEEE Softw..

[46]  Tapani Kilpi,et al.  Implementing a Software Metrics Program at Nokia , 2001, IEEE Softw..

[47]  Peter Kulchyski.  and , 2015 .

[48]  Touseef Tahir,et al.  A structured goal based measurement framework enabling traceability and prioritization , 2010, 2010 6th International Conference on Emerging Technologies (ICET).

[49]  Tore Dybå,et al.  The Future of Empirical Methods in Software Engineering Research , 2007, Future of Software Engineering (FOSE '07).

[50]  Warren Harrison.  A flexible method for maintaining software metrics data: a universal metrics repository , 2004, J. Syst. Softw..

[51]  Barry W. Boehm,et al.  Some future trends and implications for systems and software engineering processes , 2006, Syst. Eng..

[52]  Mario Piattini,et al.  Implementing Software Measurement Programs in Non Mature Small Settings , 2007, IWSM/Mensura.

[53]  Kai Petersen,et al.  A decision support framework for metrics selection in goal-based measurement programs: GQM-DSFMS , 2013, J. Syst. Softw..

[54]  Kai Petersen,et al.  An elicitation instrument for operationalising GQM+Strategies (GQM+S-EI) , 2015, Empirical Software Engineering.

[55]  Miroslaw Staron,et al.  Using Models to Develop Measurement Systems: A Method and Its Industrial Use , 2009, IWSM/Mensura.

[56]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[57]  Sandro Morasca,et al.  An Operational Process for Goal-Driven Definition of Measures , 2002, IEEE Trans. Software Eng..

[58]  Paul J. Layzell,et al.  Extending the ami approach to encompass the Foundation activity , 1997, Proceedings Eighth IEEE International Workshop on Software Technology and Engineering Practice incorporating Computer Aided Software Engineering.

[59]  Laurie A. Williams,et al.  Validating software metrics: A spectrum of philosophies , 2012, TSEM.

[60]  Khaled El Emam,et al.  Empirical Studies of Software Process Assessment Methods * † , 1999 .

[61]  P. Parviainen,et al.  Practical experiences of tool support in a GQM-based measurement programme , 2004, Software Quality Journal.

[62]  Victor R. Basili,et al.  Validation on an Approach for Improving Existing Measurement Frameworks , 2000, IEEE Trans. Software Eng..

[63]  Marco Torchiano,et al.  Empirical studies in reverse engineering: state of the art and future trends , 2007, Empirical Software Engineering.

[64]  Frank Niessink,et al.  Perspectives on improving software maintenance , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[65]  Mario Piattini,et al.  Integrated Measurement for the Evaluation and Improvement of Software Processes , 2003, EWSPT.

[66]  Cmmi Product Team.  Capability Maturity Model® Integration (CMMI), Version 1.1--Continuous Representation , 2002 .

[67]  Cem Kaner,et al.  Software Engineering Metrics: What Do They Measure and How Do We Know? , 2004 .

[68]  Marcus Ciolkowski,et al.  Empirical results from using custom-made software project control centers in industrial environments , 2008, ESEM '08.

[69]  Jürgen Münch,et al.  Goal-Oriented Setup and Usage of Custom-Tailored Software Cockpits , 2008, PROFES.

[70]  Mario Rivas,et al.  What do software practitioners really think about project success: A cross-cultural comparison , 2008, J. Syst. Softw..

[71]  Claes Wohlin,et al.  Software process improvement through the Lean Measurement (SPI-LEAM) method , 2010, J. Syst. Softw..

[72]  S. Biffl,et al.  A survey on the fitness of commercial software metric tools for service in heterogeneous environments: common pitfalls , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[73]  Félix García,et al.  Implementing a software measurement program in small and medium enterprises: a suitable framework , 2008, IET Softw..

[74]  Ieee Standard,et al.  Adoption of ISO/IEC 15939:2007— Systems and Software Engineering— Measurement Process , 2009 .

[75]  Maria Teresa Baldassarre,et al.  Multiview Framework for Goal Oriented Measurement Plan Design , 2004, PROFES.

[76]  Rini van Solingen,et al.  Using FAME Assessments to Define Measurement Goals , 2000, IWSM.

[77]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[78]  Sebastian Schunk.  Software Measurement @ Siemens - A Practical Approach Allows Best Practice Sharing of Various Organizations , 2008, IWSM/Metrikon/Mensura.

[79]  Lionel C. Briand,et al.  Practical guidelines for measurement-based process improvement , 1996, Softw. Process. Improv. Pract..

[80]  Richard Torkar,et al.  Software fault prediction metrics: A systematic literature review , 2013, Inf. Softw. Technol..

[81]  Alain Abran,et al.  Facts and Perceptions Regarding Software Measurement in Education and in Practice: Preliminary Results , 2011, J. Softw. Eng. Appl..

[82]  R. Leisten,et al.  A framework for evaluating the criticality of software metrics: an analytic hierarchy process (AHP) approach , 2006 .

[83]  Lukasz Cyra,et al.  Extending GQM by Argument Structures , 2007, CEE-SET.

[84]  Hans van Vliet,et al.  Measurements Should Generate Value, Rather than Data , 1999 .

[85]  Victor R. Basili,et al.  Linking Software Development and Business Strategy Through Measurement , 2010, Computer.

[86]  Miroslaw Staron,et al.  Developing measurement systems: an industrial case study , 2011, J. Softw. Maintenance Res. Pract..

[87]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[88]  Cmmi Product Team.  Capability Maturity Model® Integration (CMMI), Version 1.1--Staged Representation , 2002 .

[89]  Mario Piattini,et al.  A systematic review measurement in software engineering: State-of-the-art in Measures , 2006, ICSOFT.

[90]  Wolfhart B. Goethert,et al.  Experiences in Implementing Measurement Programs , 2001 .

[91]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[92]  Alain Abran,et al.  Improvement Opportunities and Suggestions for Benchmarking , 2009, IWSM/Mensura.

[93]  Ray Offen,et al.  Establishing Software Measurement Programs , 1997, IEEE Softw..

[94]  Andrew Boyd.  The goals, questions, indicators, measures (GQIM) approach to the measurement of customer satisfaction with e-commerce Web sites , 2002, Aslib Proc..

[95]  Tore Dybå,et al.  Evidence-based software engineering , 2016, Perspectives on Data Science for Software Engineering.

[96]  Claes Wohlin,et al.  Prioritizing and Assessing Software Project Success Factors and Project Characteristics using Subjective Data , 2004, Empirical Software Engineering.

[97]  David Alan Grier.  The Value of a Good Name , 2010, Computer.

[98]  A. J. Alejandro.  Management indicators model to evaluate performance of IT organizations , 2001, PICMET 2001.

[99]  Aybüke Aurum,et al.  Assessment of software measurement: an information quality study , 2004 .

[100]  Mario Piattini,et al.  Managing software process measurement: A metamodel-based approach , 2007, Inf. Sci..

[101]  D. Morris,et al.  PSM Insight: power and flexibility for issue-driven measurement , 2001, Proceedings 26th Annual NASA Goddard Software Engineering Workshop.

[102]  D. Ross Jeffery,et al.  An Instrument for Assessing Software Measurement Programs , 2000, Empirical Software Engineering.

[103]  Mario Piattini,et al.  Towards a consistent terminology for software measurement , 2006, Inf. Softw. Technol..

[104]  Kathryn Cormican,et al.  An analysis of non-observance of best practice in a software measurement program , 2012 .

[105]  Çigdem Gencel,et al.  A Model for Selecting an Optimum Set of Measures in Software Organizations , 2009, EuroSPI.

[106]  Markku Oivo,et al.  Adopting GQM-Based Measurement in an Industrial Environment , 1998, IEEE Softw..

[107]  Mark Kasunic.  The State of Software Measurement Practice: Results of 2006 Survey , 2006 .

[108]  Watts S. Humphrey.  Introduction to Software Process Improvement. , 1993 .

[109]  Brian Henderson-Sellers.  OO software process improvement with metrics , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[110]  Miroslaw Staron,et al.  Factors Determining Long-term Success of a Measurement Program: An Industrial Case Study , 2011, e Informatica Softw. Eng. J..

[111]  V. Basili.  Software modeling and measurement: the Goal/Question/Metric paradigm , 1992 .

[112]  Victor R. Basili,et al.  A Methodology for Collecting Valid Software Engineering Data , 1984, IEEE Transactions on Software Engineering.